Geoscape Australia (formerly known as PSMA Australia at the time of engagement) specializes in providing national, authoritative location data to businesses and governments. In simple terms, location data is the new champion of analytics. Like time, it provides a framework for analysis and can reveal previously unseen links that drive competitive advantage and generate value for business.
The maturing of remote sensing capabilities and growth in Internet of Things (IoT) means more location data is being produced than ever before.
The sheer amount of location data available means we need to simplify data supply so businesses and governments can focus on solutions.
The problem
Geoscape Australia promises customers fresh, accessible and scalable location data. To deliver on the promise, the client turned to agile software development and engaged Thoughtworks to simplify the generation of their data products and transform their business strategy.
PSMA had previously engaged with the Thoughtworks community and our approach to digital platform transformation at our tech events.
This synergy saw us partner to deliver an end-to-end technical solution to address the platform challenges Geoscape had identified, including standardizing inconsistent supplier data formats.
Our systems, siloed data and the sheer size of the datasets we produced were becoming cumbersome and requiring significant manual intervention by geospatial specialists.
It was critical that data quality be maintained at a high level at all times, as the location data Geoscape delivers is critical to organizations including Australia's emergency services.
Within the original workflow, manual intervention was required to process the volume of geospatial data sourced by Geoscape. Engineers kicked-off processing jobs and waited hours for a result. It also opened the possibility of human error.
With these factors in mind, developing a solution to process geospatial data without significant manual intervention was key to the success of our partnership.
The solution
With data pipelines at the core of Geoscape’s business, we sought to build a custom solution. In just 10 months, the combined Thoughtworks and PSMA team (now Geoscape) went from discovery and inception to launching the streaming data platform for real-time customer consumption, alongside a suite of Quality Assurance (QA) tools. The team focused equally on building the platform and lifting the capability of Geoscape to extend the platform.
Our solution outcomes included:
Streaming Data Platform
We built the streaming data platform to leverage cloud technologies and minimize operational work.
The platform consists of a series of data pipelines that automatically ingest, validate, sanitize and standardize data as soon as it’s made available by suppliers. From here, internal product teams consume the data at various points in the pipelines and use it to power the products they deliver to their customers and value-added resellers.
We used a streaming solution based on Amazon Kinesis to ensure multiple customers could tap into time series data, and AWS Lambda for compute. Amazon Kinesis ensures a near infinite scale up and down based on workload, while AWS Lambda reduces operational effort and provides significant horizontal scale capability.
Data QA
Over and above the technical solution delivered, we developed a process for data quality assurance. The process enables Geoscape to quickly verify an entire dataset on an ongoing basis. There is also a system for converting exploratory analysis (using Jupyter notebooks) into ongoing monitoring and alerting to ensure the quality of the data remains consistent.
Tooling
We implemented tools like centralized logging which allows Geoscape to see the state of the whole pipeline at any moment in time, as well as monitoring and alerting, error handling and repeatable local development environments.
In short, making data processing repeatable and predictable, reduced processing times from days and weeks to hours and minutes, allowing PSMA to meet client real-time needs and expectations.
Capability uplift
It was rewarding to work with a client like Geoscape that is ahead of the curve with respect to shifting their culture. They had already adopted an agile approach and they continue to iterate upon this to ensure they are continually driving collaboration and innovation across the business.
We wanted to ensure the PSMA team was in a strong position to maintain the platform when we rolled off, so we supported them through hands-on implementation of continuous integration, continuous deployment and data version control among other practices.
An ongoing partnership
We’re continuing our partnership with Geoscape as they migrate more of their products to the new continuous data platform.
