Getting traction with Data Mesh

It’s been almost five years since Zhamak Dehghani announced the concept of Data Mesh in her article How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh. This article resonated with many people due to its clear articulation of pain points felt by modern organizations trying to get value from the data they hold.

Since then, the Data Mesh community has been working on techniques and approaches to fulfill the vision that Zhamak laid out. Vendors have also begun to incorporate support for Data Products into their tooling. The need to refresh how we manage data has only grown more urgent in the meantime.

Thoughtworks Data Engineer Arne Lapõnin and Technology Director Chris Ford recently published a Foundations of Data Mesh video course with O’Reilly media. In it they cover the core principles of Data Mesh, along with advice on how to get started, how to build momentum, and how to take into account human factors. This course was created from their experience helping Thoughtworks clients get started on their Data Mesh journeys.

Chris explains:

“When clients ask me whether Data Mesh is helpful for them, I start with a three question test:

• Do you have sufficient scale? If you have a development organization of 15 people, then perhaps a simpler collaboration model would be sufficient to meet your needs.

• Do you have a strategy that depends on deriving value from data? If data-driven insights or features aren’t a priority, then you might not have sufficient organizational motivation to go on a Data Mesh journey.

• Are you prepared to take advantage of the autonomy Data Mesh will afford to your product teams? Data Mesh is a model that moves responsibility to the teams who have knowledge and context of the data, but this doesn’t work if those teams lack the autonomy to make decisions.

If the person can say yes to all three questions, then we can go deeper and work out whether a Data Mesh approach is useful to them.”

If an organization believes that Data Mesh is something they want to take on, it can be hard to know where to start. Arne describes their recommended approach:

“It can be tempting for organizations to take up one element of Data Mesh first and try to tackle it in isolation. In our experience, this one-at-a-time approach simply doesn’t work.

For example, some organizations start with the Data Platform and try to make it fully featured before starting to build Data Products. They risk building the wrong platform due to lack of feedback from the intended users.

Others go in the opposite direction and focus exclusively on Data Products and neglect foundational capabilities. In this case they create a lot of waste because the process of building Data Products ends up non-standardized due to lack of platform support.

Our advice is to consider all the elements of Data Mesh in tandem. You should develop initial use cases in parallel with the Data Products needed to power the use cases, alongside the Data Platform needed to support the Data Products, with the governance needed to facilitate the consistency you need to tie it all together.

There shouldn’t have to be a large up-front investment. You know you’re doing it right when there’s a natural rhythm between Data Products delivered and improved supporting capabilities making them easier and easier to create.”

Since this approach relies on use cases, it’s important to have a means of choosing the right ones to get started with. Arne explains how he has helped clients to do this:

“Identifying initial use cases is not so hard if you have the right people in the room, and extremely hard if you don’t. Chris and I recommend the Data Mesh Accelerate workshop described by our colleagues Paulo Caroli and Steve Upton.

Essentially, this is a process that works backwards from strategy to data products:

• Describe a desired data state

• Derive measures to know when you’re getting there

• Explore use cases that could influence the measures

• Discover Data Products needed to support the use cases

This keeps you honest and connected to your aspirational outcomes. Also, treating your data as a product with a purpose will have immediate benefits and keep your decision-makers bought-in.”

Above all, Data Mesh is a responsibility model for how data is managed in an organization. That means that for a Data Mesh adoption to be successful, it needs people to change. If you need people to change, you need them to be engaged. Chris cautions leaders to keep people in mind:

“Don’t forget about the human factor. For any kind of change initiative, you can’t just tell people what’s going to happen and expect them to do it. You need to give them tools and even more importantly to give them motivation.

Make sure that as a leader you praise the behaviors you want to see and that you give your colleagues support and space to learn.”