PSMA Australia specialises in providing national, authoritative location data to businesses and governments. In simple terms, location data is the new champion of analytics. Like time, it provides a framework for analysis and can reveal previously unseen links that drive competitive advantage and generate value for business.
The maturing of remote sensing capabilities and growth in Internet of Things (IoT) means more location data is being produced than ever before.
The sheer amount of location data available means we need to simplify data supply so businesses and governments can focus on solutions.
Geoscape by PSMA Australia promises customers fresh, accessible and scalable location data. To deliver on the promise, PSMA turned to agile software development and engaged ThoughtWorks to simplify the generation of their data products and transform their business strategy.
PSMA had previously engaged with the ThoughtWorks community and our approach to digital platform transformation at our tech events.
This synergy saw us partner to deliver an end-to-end technical solution to address the platform challenges PSMA had identified, including standardising inconsistent supplier data formats.
Our systems, siloed data and the sheer size of the datasets we produced were becoming cumbersome and requiring significant manual intervention by geospatial specialists.
It was critical that data quality be maintained at a high level at all times, as the location data PSMA delivers is critical to organisations including Australia's emergency services.
Within the original workflow, manual intervention was required to process the volume of geospatial data sourced by PSMA. Engineers kicked-off processing jobs and waited hours for a result. It also opened the possibility of human error.
With these factors in mind, developing a solution to process geospatial data without significant manual intervention was key to the success of our partnership.
With data pipelines at the core of PSMA’s business, we sought to build a custom solution. In just 10 months, the combined ThoughtWorks and PSMA team went from discovery and inception to launching the streaming data platform for real-time customer consumption, alongside a suite of Quality Assurance (QA) tools. The team focussed equally on building the platform and lifting the capability of PSMA to extend the platform.
Our solution outcomes included:
Streaming Data Platform
We built the streaming data platform to leverage cloud technologies and minimise operational work.
The platform consists of a series of data pipelines that automatically ingest, validate, sanitise and standardise data as soon as it’s made available by suppliers. From here, internal product teams consume the data at various points in the pipelines and use it to power the products they deliver to their customers and value-added resellers.
We used a streaming solution based on Amazon Kinesis to ensure multiple customers could tap into time series data, and AWS Lambda for compute. Amazon Kinesis ensures a near infinite scale up and down based on workload, while AWS Lambda reduces operational effort and provides significant horizontal scale capability.
Data QA
Over and above the technical solution delivered, we developed a process for data quality assurance. The process enables PSMA to quickly verify an entire dataset on an ongoing basis. There is also a system for converting exploratory analysis (using Jupyter notebooks) into ongoing monitoring and alerting to ensure the quality of the data remains consistent.
Tooling
We implemented tools like centralised logging which allows PSMA to see the state of the whole pipeline at any moment in time, as well as monitoring and alerting, error handling and repeatable local development environments.
In short, making data processing repeatable and predictable, reduced processing times from days and weeks to hours and minutes, allowing PSMA to meet client real-time needs and expectations.
It was rewarding to work with a client like PSMA that is ahead of the curve with respect to shifting their culture. They had already adopted an agile approach and they continue to iterate upon this to ensure they are continually driving collaboration and innovation across the business.
We wanted to ensure the PSMA team was in a strong position to maintain the platform when we rolled off, so we supported them through hands-on implementation of continuous integration, continuous deployment and data version control among other practices.
We’re continuing our partnership with PSMA as they migrate more of their products to the new continuous data platform.
