Posted by AaronClausen
Hi All,
We've been busy over the break in amongst a bit of beach time making a number of critically important architectural changes to the platform.
We've basically just come out of 11 years of saying "sure no problem" trying different things to make a platform that enables high quality citizen science so that all these records can get through to decision makers. In short, we've been too generous and the result is the platform has sprawled out of control and had become difficult to maintain.
It's one thing to build a new piece of functionality, but it's another thing to maintain it sustainably over the following decade.
Our infrastructure has increasingly become overloaded with requests, serving up thousands and thousands of requests for legacy and complex functionality that sometimes isn't even used.
And our team burn hundreds of hours trying to keep up with it all late at night and on weekends.
Much of this functionality was implemented in a bespoke manner to meet a particular requirement under pressure or get someone out of trouble at the time.
One tiny example is that we've ended up with 10 different bespoke ways to display a list of sightings for the various types of objects that have sightings hanging off them (category pages, species pages, user profile pages, location pages, priority species list pages, collection pages, latest / featured / unidentified / inconclusive sightings etc etc). They all had varying levels of filtering, varying levels of sorting the results and varying levels of pagination and sensitive data handling.
We also had a plethora of bespoke code that provided complex regional based statistics and only to a select few individuals = not sustainable.
We also allowed anybody to create a priority species list - many of which were created with gusto but then neglected over the long term and never updated again.
Priority Species Lists now only support formal government priority species lists - like the Threatened Species Action Plan 2022 - 2032 Priority Species.
There are countless other examples where the complexity of the platform had become far too high and it was becoming impossible to maintain.
Google was smashing some of these more complex pages with it's indexing robot and also causing us more invisible pain and financial burden.
The way we displayed google maps on every sighting page was costing us thousands of dollars per month with no benefit.
Nearby sightings have been moved to a separate page so we don't have to clobber the database and render them on every single sighting page load.
We've implemented much more aggressive performance caching so that our servers remember larger and larger amounts of content that changes infrequently, so that the same content can be served up to many thousands of users while putting less strain on our infrastructure.
Over the break, we've given the platform a massive haircut.
Thousands of lines of code have been removed.
The way we integrate google maps has been redeveloped.
Complex functionality has been merged and consolidated into common components.
About 20 database tables have been binned.
Entire half-baked features that weren't really utilised properly have also been binned.
This is also why we are unable to provide a solution for formal surveys any more.
We now have a single way to display a list of sightings right across the platform and everything leverages and benefits from this common functionality.
If we enhance it, all collections now benefit platform wide.
This is because the entire platform has now been migrated to a collections based architecture.
This means NatureMapr itself is now just one big huge collection.
The Canberra/CNM region is just another collection.
And every single region E.g. Sydney, Hunter, Pilbara, are all just their own collections too.
The NSW BCT Land Libraries program is just another collection.
The sightings in your own user profile are now also underpinned by your own private collection and of course the sightings for Mount Majura or Black Mountain are also a collection.
Latest sightings, Featured sightings, Unidentified and Inconclusive sightings are also.. you guessed it.. just other collections.
All collections get a consistent sightings list, field guide and dashboard/stats functionality.
We build/maintain it in one place and every collection benefits platform wide.
We have turned off and thrown several other areas of complexity in the bin that either nobody used, or didn't provide sufficient value to justify it's own existence and the impact on our team, our families and the huge financial costs in maintaining it.
The man hours that go into maintaining and supporting the platform is by far our largest and most unsustainable line item.
But we're bringing this under control and have to be more ruthless.
Having everything essentially running on a collection means we now have a consistent architecture giving us re-use across the entire platform.
We can now easily render stats for entire NatureMapr or for just the Canberra region and these stats are publicly available for everybody as is the new Species Richness Score now displayed on all regional home pages.
If we make any improvements to these dashboards, they will be rolled out for every single dashboard in one go.
As a general rule, the platform is progressively becoming more and more focused on supporting a diverse range of collections (which support the diverse range of projects and groups that people organise themselves into) rather than just being purely region based (which is how we started out in 2013).
Conway's law says that the structure of an IT system will generally reflect the structure of the people or groups that use it - and this is what we're seeing happen naturally with the rise of collections.
Regions are decreasing in relevance and collections are increasing in relevance.
We're generally deleting more and more legacy region based code and building new, re-usable collection based functionality.
All of this work is to help ensure we can keep up with the growth and keep maintaining (and funding) the platform over the long term.
We are narrowing our focus so that we can provide the best possible experience for getting opportunistic records of all plant and animal species through to decision makers.
That's all we do and we're staying laser focused on it to ensure our survival.