Data products: The gap between technical excellence and business value

Data products are a widely-discussed if often misunderstood idea. And, with AI stealing a huge amount of mindshare in business and technology over the last few years, the effort put into ensuring they're effective, secure and impactful is arguably receding. This needs to be corrected: the continuing gap between execution and real value remains. It can't be fixed by AI and may even hamper effective AI use by making it more challenging for organizations and teams to find trustworthy, and well-organized data that is essential for AI initiatives.

This is why Thoughtworker Amy Raygada is writing a series of books on Data Products. Aimed at helping organizations unlock greater value from data products and to bridge the gap between execution and impact, the series will take in a range of topics — from design and deployment to governance. 

With the first volume of the series published late in 2025, I wanted to speak to Amy to learn more about her perspective on the field and what the industry needs to understand.

Richard Gall: Why a book on data products now?

Amy Raygada: Because we keep building Ferraris nobody drives! Organizations are spending six- and seven-figure sums on data platforms with adoption rates as low as eight percent. 

The technology works. Data teams are talented. And yet business users are still pulling reports from Excel. That gap between technical excellence and actual business value is the problem this book exists to close.

There's no shortage of books on data architecture or data engineering, but the question of how you turn data into something people actually use — that delivers repeatable business value, that has real users and real ownership — that conversation has been missing. This book is that conversation.

What makes data products unique as a kind of product

RG: Why do we struggle to treat data as products? What’s unique and distinct about managing data products compared to products in general?

AR: There are a few things that make data uniquely hard. First, users often can't articulate what they need. They'll tell you they want everything and then only ever use two charts. Second, data has this invisible quality problem: bad data rarely breaks the system, but it will quietly erode trust until users stop showing up. Third, data sits at the intersection of multiple organizational boundaries: IT owns it, business uses it, compliance governs it and nobody quite agrees on who's accountable.

In traditional product management, you're designing an experience; in data product management, you're designing the raw material for decisions. The user wants to make better decisions faster. That's a different design challenge entirely. 

A marketing manager I worked with put it perfectly: "I don't need to know our customer lifetime value down to the penny. I need to know if it's getting better or worse, and I need to know in 30 seconds." That single insight reshaped an entire analytics product. It's that kind of user understanding that most data teams never develop, because they're too focused on the data itself.

Mindset and organizational culture

RG: A lot of the framing around data products is based on mindset and skills: but isn’t the problem really one of organizational culture and politics?

AR: Absolutely — I'd be doing readers a disservice if I pretended otherwise. I've seen technically perfect implementations fail because no one with power actually cared about adoption. I've seen governance frameworks ignored because they were designed without business buy-in. The book is honest about this: plan for 18 to 24 months of cultural evolution. You can change an org chart in a quarter, but you cannot change how people think and work in a quarter.

However, here's where I push back slightly on framing it purely as culture and politics: culture follows structure. When you organize teams around user value instead of technical functions, when you create a data product manager role that bridges business and engineering, when you tie success metrics to adoption and ROI rather than pipeline elegance, you start shifting the incentives. 

Culture changes when the conditions that shape behavior change. So yes, it's deeply political and cultural, but the product thinking framework is precisely how you start moving those conditions.

Balancing federated governance and centralization

RG: Federated governance is clearly an important facet of data products. Why is it challenging? And surely it needs some degree of centralization to be successful?

AR: You're right, and this is one of the most misunderstood points in the whole data mesh conversation. Pure federation creates chaos: inconsistent standards, duplicated effort and integration nightmares. I've seen organizations attempt federated governance because it sounds modern, only to end up with teams that can't talk to each other and executives getting conflicting numbers in board meetings.

The answer is what I call the 'four-layer governance stack'. The bottom two layers, infrastructure automation and computational policies, should be centralized and automated. You build governance into the platform itself so teams get compliance benefits without extra overhead. The top layers are where federated decision-making lives: standards that teams can adapt and a governance council that resolves conflicts rather than controlling everything.

The shift-left philosophy is critical here. If you're reviewing data products for compliance after they're built, you've already lost. Embed the governance checks into the development tools, make the right thing the easy thing and reserve human judgment for genuine edge cases. That's when federation becomes sustainable — when the center isn't a bottleneck but an enabler.

Data product principles and an evolving environment

RG: Was it hard to write a book like this when things change so rapidly? Is there anything that you think is relatively timeless? What do you think may need to be updated or revised?

AR: Honestly, yes. There were moments writing the AI chapter where I felt like I was trying to hit a moving target. Regulatory frameworks are evolving, tooling is changing and the speed of AI adoption in enterprise is accelerating in ways that are hard to predict.

But here's what I came to: the principles are timeless, even when the implementation details aren't. Start with user problems, not technical capabilities. Measure success through adoption and business impact. Build governance that enables rather than constrains. Organize around value delivery. Those ideas were true ten years ago and they'll be true when AI systems are making decisions we can't yet imagine.

What will age? The specific regulatory landscape, the EU AI Act details, the California transparency laws, those will evolve. Some of the tooling references. Possibly the specific framing around data mesh, which is still being debated. That's exactly why this is Volume One of a series. The strategic foundations I'm confident will hold. The technical implementation details in Volume Two and the operational patterns in Volume Three will need to be living documents.

Key takeaways and what's next

RG: What do you hope people take from the book?

AR: One thing above everything else: the problem was never the data, and it was never the technology. It was always the gap between what data teams build and what humans actually need.

I want readers to finish this book and immediately think differently about the work sitting on their desk. Not 'how do I improve this pipeline' but 'who uses this, what decision are they trying to make and am I making that easier or harder?' That mindset shift, from data as byproduct to data as product, is what I've seen transform organizations. Not the platforms. Not the frameworks. The way people think about their work.

If a data leader reads this and goes back and conducts their first real user interview before their next build cycle, this book has done its job.

RG: What’s coming in future volumes?

AR: Volume one is the strategic foundation: the why and the what. It answers the question: 'Where should we begin?' Volume two moves into execution. It's the technical guide covering architecture, design principles, platform engineering and the control plane infrastructure that makes self-service at scale actually work. That's the 'how do we build this?' book.

Volume three is about sustainability; what happens after you launch. Versioning, governance at scale, domain onboarding, team enablement and lifecycle automation. The question that will answers is: 'How do we make this work long-term?' Because too many organizations successfully launch data products and then watch them slowly become tomorrow's technical debt.

The series is deliberately sequenced this way. You can't build well without strategic clarity, and you can't sustain what you build without the operational infrastructure to support it. Volume One gives you the keys. The next two volumes show you how to actually drive the car.

Thanks to Amy for taking the time to answer my questions. You can learn more about the book here:

Data Products Vol.1 From Projects to Products