How AI helped a client unlock direct access to complex product data in a week

The client

The client is a leading industrial supply company. It supports critical operations by ensuring fast and reliable access to essential products that keep facilities, equipment and people working safely and efficiently.

The challenge

Thoughtworks was tasked with helping the client unlock direct, self-serve access to product data managed within its product information management (PIM) platform, STIBO STEP, by building a data product that provides direct access without needing to go through the platform itself.

The platform, STIBO STEP, will reach end-of-life at the end of 2027. A SaaS version is being made available, but this would increase costs for the client by 70% and remove direct database access.

Added to this, expertise and knowledge of STIBO STEP was expensive for the client. Every new data feed, change or schema question required the client to pay for STIBO’s professional services. In other words, this was becoming a costly and recurring dependency that wasn’t sustainable as the organization moves toward self-serve data.

The turn of the screw

With the data product due to go live only six weeks in the future (owing to commitments and promises made to stakeholders downstream) and the number of SKUs to be extracted totalling more than six million, the closed and opaque platform posed a significant risk to the client and the project. The client was caught between a potentially expensive and long migration away from STEP and risks to its operations.

Thoughtworks was chosen as the partner for this project based on our proven expertise in enterprise data modernization and AI engineering, including prior work building the client's data platform architecture.

The objective and approach

The hardest part of the problem was extracting 6.15 million SKUs with 650+ attributes from STEP's complex and poorly documented scripting environment. More specifically, the STEP proprietary Outbound Integration End Point (OIEP) approach requires JavaScript business rules running inside STEP's proprietary scripting engine. Rather than raising STIBO tickets or procuring a specialist, we used Claude to generate, debug and iterate the extraction code.

This was not a simple code generation exercise. STEP's Scripting API is strict about which Java methods can be called from JavaScript, the documentation is sparse and several approaches that looked correct on paper failed at runtime. Each failure was analyzed and a new approach tried. 

While an iteration cycle would typically have taken weeks, we were able to compress it into days. A fully working, production-grade JSON extraction pipeline was built and validated in under one week, and STIBO professional services were not engaged at any point.

The solution

STIBO STEP's native export formats weren't sufficient for this pipeline. STEPXML requires heavy transformation; STEP's built-in Generic JSON format omits attributes with no value and offers no custom logic. 

Instead, the extraction uses STEP's OIEP configured with the Business Rule Based Message Processor. This is a licensed STEP feature that replaces the standard format engine with two custom JavaScript business actions: a Node Handler, invoked once per product, that collects attributes, cross references, asset references and classification references into a JSON message, and a Joiner, invoked once per batch, that combines those messages into a single JSON file. 

The OIEP reads from the approved workspace and can be triggered manually for a full load or on product approval events for incremental updates. This delivers the output as JSON to a S3-backed Unity Catalog Volume.

The Databricks pipeline supports both a full batch load and an incremental load via Auto Loader, giving the flexibility to run a complete refresh or process ongoing updates as needed. Data lands in an append-only Delta raw layer, then flows into a source-aligned data product table (Delta with Iceberg UniForm). Downstream consumers access it through Snowflake via Delta Direct, an external volume and catalog integration over the Delta log, exposed through a published view.

Importantly, the solution was production-grade from day one, built with full CI/CD, infrastructure as code, secrets management, testing and security scanning in place. This pattern is now the reusable template for future data products sourced from STEP, giving the client a flexible, self-serve path to its own product data.

Outcomes and impact

Despite the time pressure and significant constraints, Thoughtworks delivered the technology to give downstream teams direct, self-serve access to product data.

The outcomes include:

In the short term, the client met a challenging deadline using its own team and AI-assisted engineering, without the need for additional vendor engagements. In the long-term, meanwhile, cost savings and continued operational excellence are unlocked by a pattern that can be reused and will provide a strong foundation as the client's data architecture continues to evolve.

There’s no trade-off between immediate impact and future scale here; this initiative delivers both.