The hard parts of data architecture

Full transcript

Rebecca Parsons: Hello, everyone. Welcome to another edition of the ThoughtWorks Technology podcast. My name is Rebecca Parsons, the Chief Technology Officer for ThoughtWorks. I'm here with one of my co-hosts on the podcast, although she's here today as a guest, Zhamak Dehghani.

Zhamak Dehghani: It's great to be here, Rebecca.

Rebecca: We're also joined by someone who's has also been on this podcast before, Pramod Saladage.

Pramod  Sadalage: Thank you, Rebecca. Nice to be here with you, Zhamak.

Rebecca: Thank you both for joining us. What we'd like to talk about today? We foreshadowed this in the episode when we had Mark Richards and Neal Ford talk about their latest book, Software Architecture: the Hard Parts because both Zhamak and Pramod collaborated with Mark and Neal on this book to add aspects of data architecture. We wanted to get them on this podcast as well to talk about their experiences with that. Let's start with the mechanics almost.

Mark and Neal have been collaborating with each other for a long time. I do know, Pramod, you did collaborate with us on the building evolutionary architectures book, but how did this collaboration come about? How did you end up being co-authors on this book?

Pramod: You want to go first, Zhamak?

Zhamak: Sure. Neal and Mark have been working on this material for a long time. they had a wealth of information, to share off the back of their book that they had written on foundational concepts in architecture. Neal invited me to join because, at the time, I was working on data mesh as a paradigm shift in data architecture. We wanted to really extend the conversation around architecture to data, and particularly, usage of data in analytics and ML. I did not take a moment to even think about it. I said yes, immediately, because I knew it would be an amazing learning experience to work with Neal and Mark to contribute their book.

Pramod: My story is something similar too. I have been, like you said, Rebecca, working with Neal on some of these concepts of like the intersection of software architecture and data architecture, especially around the concept of how you would evolve some of these things. The contributions in the evolutionary architecture book, as well as some speaking engagements and some O'Reilly software conferences, and things like that. Neal asked me if you'd like to contribute to this book, and just like Zhamak, I jumped on the idea, and here we are, probably after a year of banging on the keyboard.

Zhamak: I think there is a little grain of truth that I think is the sub-context of actually this book or this work that, for a long time, our specialties have been divided. A bunch of us were very focused on the computational architecture and operational. It feels like Pramod had great experience on the data aspects of it more than I. I think this was a great opportunity, even from the skill set and experience and a specialty, bring folks like Pramod who had that data focus, and perhaps, Neal with a computational focus, and it was necessary. I think it was necessary for Pramod to join us.

Rebecca: How do you see these ideas around data architecture fitting into these broader concepts of architecture? I know, we've been evolving our thinking about what constitutes architecture for quite some time, but did it feel like, "Okay, we've got all of this computational stuff. Let's pull the data stuff on the side," or did it actually feel like it fit into to the book as the two of you were working with the two of them?

Pramod: Yes. In some ways, I would say it fit very well and the reasons being, like any trade-off analysis, you do, there are probably I would say, 10% to 15% of that that doesn't involve any data, but majority of it is involved. You are talking about like, "Should I take eventual consistency or not?" That's not just a software decision. It's all about data. Similarly, there are other discussions about coupling versus cohesion or versus things like, should I take on distributed transactions or not?

Bunch of these all involve the data, and whenever you're making decisions about the architecture, underlying theme that you are trying to drive at is, how do I make data available? How do I keep the data consistent or how do I keep all these pieces together so that I don't have to take on other accidental complexities because of my decisions? Some of these, ultimately if you keep digging down further and further, they affect data and how the data is available. Either to the applications or to the analytics platforms so that you have to involve the aspect of data into those decisions.

On the surface, it may seem I'm making a software decision, but if you dig deeper, you can see that data is a big driver into what decisions you arrive at.

Zhamak: I think on the analytical side, on the operational data storage and how do I model my data for my services as Pramod said, it's actually a bit closer to the decisions that you make around the design of your applications, is part of that. From the analytical perspective, it's often been a very disconnected set of decisions and trade-ups. One thing about this book that actually brought the analytical decisions closer to the operational decisions, was the narrative that the book explains on behalf of this organization, the hypothetical organization, Sysops Squad that is. It's an organization that provides technical staff to support staff for people that bought electronics for example.

If you ground the decisions that you have to make at an organizational level in a real or even a hypothetical example, you will see that your operational structure, how you design your services and your applications, easily gets extended to, how do I now expose data from these applications so that I can now make intelligent decisions for the Sysops Squad? In fact, we have a narrative that follows through the book that puts the reader in the shoes of decision-makers in this organization. Initially, the decisions around how do we build our applications? How do we allocate the data or organize the data for those applications?

Very quickly as that organization evolves, they start thinking about, how can I optimize my workforce based on the knowledge that I have in terms of the demand? How do I upscale my workforce? How how do I make intelligent decisions around the routing of this? Support requests are coming. The moment you start extending those operational capabilities to intelligent or intelligently augmented decisions, then we have to bring the conversation of analytics as part of a bigger picture of architecture conversation. It can no longer be a separate conversation. I think that was a wonderful thread through the book to bring all of these decisions together. Same as what happens in reality in real organizations.

Rebecca: It is interesting to me how in the past there was that hard separation. This is the transactional data store. This is the operational data store. This is where the work happens, and then we'll toss everything over. That's where the analytics goes on, and there was such a separation. I like the way you phrase that, Zhamak, and thinking about as we want to become more of a data-driven business how we have to start dragging that analytics back into it because those workflows are now requiring access to information and to that analysis of the data. It used to be someone else's problems.

Pramod: I think the other aspect of this is also even operational systems nowadays need analytical data. If recommendation engines, for example, they're right there on the website along with the transactional systems but they need analytical data. You can't rely on analytical data. That's a week old because that's when the data ran or you can't rely on them because of whatever other circumstances. It's like now we have to think more about, how can I get analytical data faster? How can I get analytical data in a shape that something more operational can use it? You have to think a little bit different about where the systems fit.

Rebecca: Let's step a little bit further back in history because for a very long time, well, first, for a very long time, architecture was simple. You had a mainframe. Then oh it's now client-server and we have a much more sophisticated understanding of architecture and the list of the different kinds of architecture; security architectures or network architectures, software architectures, integration architecture, all of those different kinds of architecture. For a very long time, people just talked about data. You never really heard data architecture. What do you think was behind this evolution of thinking as it's not just the data but we're going to elevate our thinking about data to this term of data architecture?

It's not like we didn't have these problems, we had to think about where data was and we had to think about issues of data locality. What do you think has happened that has allowed us to get this more sophisticated way of viewing data?

Pramod: I would say multiple trends, a bunch of trends back like maybe 20, 25 years ago, like you're saying, clients servers, most of the data was in one database, and you just connected to that and just used it. Like Zhamak was saying, you take the data from there and pump it somewhere, ETL or ELT, or whatever, and you have a data warehouse, very distinct set of requirements and distinct needs to be met.

The more cloud adoption or maybe even the whole concept of using data for more purposes, capturing more amounts of data, capturing data from different sources, like the speed at which the data came at you, and the various sources where it came from and all of that stuff, exploded the ways in which data can be used, should be used, and the way value can be captured from that data. All of these forces are now giving more importance on how we can use the data.

That's why I think now data is coming more in the center, of course, like cloud adoption, cloud databases, all the technologies are also helping us solve that problem. That's what is, I think, making data be a more prominent actor in this software architecture landing.

Zhamak: I agree with that. I think there is another element here which is the complexity of our organizations adjacent to our aspirations and all the diversity of the use cases of the data that we have. Increase the scale of organizations, just simple, I guess, physical factors, entropy, we have more and more business units within a lot of organizations. There's a lot more interactions and interconnectivity between these business units. The organizations have a continuous drive for growth, which brings complexity to the picture.

At the same time, now we have the volume of the data that we need and we have the computational power to really think big and aspirationally in terms of how we can use the data in variety of machine learning models and in optimization of our workforce and processes and ML-driven features. If you look at that complexity and then look at what we had built architecturally, there is a mismatch here.

Architecturally, again, we haven't really paid much attention, at least on the ML analytical space. I think on operational space, we have had advances. In the analytical space, it was just good enough to run those ETLs, get the data out with blood and sweat and tears, convert them through the pipelines, and put them under some sort of a schema. It doesn't matter how long it takes, it doesn't matter how different the data gets modeled from the reality of the facts, it's good enough for reports.

Given the complexity and aspirations and proliferation of use cases and proliferation of sources of data, that no longer works. If we step back and think about what is architecture? What's the role of an architect? Neal has this very-- I know there are many definitions of architecture and I love many of them, like the framing that Martin proposes is, decision about hard things that are hard to change.

Neal has also this wonderful description that architecture is about-- lots of architecture decisions are around how we decouple systems so we can get some sort of an optimization around scale, around scale, around speed, and then how do we integrate them together? In fact, the book has been structured around patterns of architecture that allows decomposition and patterns of architecture that allows reintegration. The moment we think about now this big data that we thought it was step problem, side problem and in the side of the organization we will have a warehouse, we'll have a leak. Now we have to think about how do we decompose it so that we can respond to that scale and complexity and multiple demands of the organization.

Then how do we connect this data back so that we can get this high-level intelligence and insights out of reconnecting the data back together? That's exactly the problem that architectural software or architecture at least has tried to solve. It was really interesting to me to see that early 2020, InfoQ put for example data mesh as an architectural trend on their architectural trend publication that they publish once a year. When they did that, they told me this is the first time that on this publication we have included something around data architecture. It's the first time that really the conversation around how do we make architectural decisions, around at least the analytical data, have entered the conversation.

Rebecca: I do wonder, picking up on that idea that you were talking about when you look historically, particularly in the pre NoSQL database, you had data warehouse-style architectures that were handled by the analytics folks over there to the side. You had relational databases that was handled by the database administrators. There was just one thing there. Now, in part because of the advent of NoSQL databases, but I also with this moving in of the analytics and some of the problems with distributed data and scalability and the proliferation of data sources, it isn't any longer any one role's problem to think about this.

There are different concerns and in many of the architectural conversations I've had with Neal and Mark, the word trade-off just comes up all the time. When you've got trade-offs across different problem domains or different technology domains, those trade-offs become even more complicated. That to me is what architecture is about, is, you've got a problem you are trying to solve, you have a series of constraints that exist and you've got to make trade-offs amongst what you're trying to achieve on the basis of these constraints.

I think just that overall complexity of the data landscape where you can't deal with complexity unless you have some framework for thinking about it and that's what architecture is all about. Fascinating, that's a question I've been pondering quite a bit is what does it take for something to be able to be called architecture?

Zhamak: I really like actually this trend that's happening. I feel like when we think about evolution of technology, sometimes a cool new tech comes along and that technology drives architectural behavior. In many cases, an architectural paradigm comes along and it transcends the existing technology and then the technology needs to catch up and adapt to it. I saw this model with the microservices back in the late 2000s like 2011, I think, we started the conversation by 2012, 2013. At least I was knee-deep in projects building microservice.

I remember there was a friction with using the technology we had in this fine-grain decomposed architecture. Over the course of the last decade, we saw containerization, we saw the orchestration tools. We saw all of this technology that had to come along to really make that paradigm feel native to the developers. I really hope by immersing more of the architectural thinking into the data landscape, we can create these meta frameworks that try to solve a real-world problem at that moment in time that perhaps Transcend Technology and then the technology comes along to fit in that paradigm. I'm excited about consequence of talking about data architecture.

Rebecca: Pramod, tell me a little bit about your contribution to the book. How did you decide what scope you want to address and what's the key takeaways are from your perspective?

Pramod: Yes, like Zhamak was saying, the two patterns we try to access, lay out everything under is the patterns for decomposing and the patterns for bringing things back together or integrating. Like, what are the forces that are driving you to decompose things or to bring things back together? We go through a bunch of patterns like that like for example, scaling. If you have to scale certain things, then maybe a monolith architecture may not work for you. Then what are the forces? Is it the amount of connections of the sheer size of throughput or what are those forces that force you to decompose certain things?

The inverse of that is, if you're ready to decompose, maybe there are forces that are wanting to put things back together. One classic example of this is distributed transactions. For example, if it's too hard and if you can probably put things in one service that can take care of functionality that is common, maybe that's a driving force to put things back together, maybe make the domain bigger or the service bigger and that kind of stuff.

Once we get up to this point, like then we said, "Maybe we should give some concrete examples on like, if you are going to break things apart, what are the ways in which you can break things apart? So, we take the reader through a journey of like, first of all, defining what a data domain is. From a domain-driven development concept, we have come to understand some concepts inside the business have this functionality that can be a domain and they do certain functionalities, and how does that relate? How does that domain in the business and domain in software architecture relate to a data domain?

What is the data domain? What are the pieces of a data domain? How do you define adore data domain and things like that, and give you an example of, again, using the Sysops Squad as a practical example that we take throughout the book, it's the same example, same concept, and same-- One funny thing I thought was really good that we did and was the concept of same developer. Even the people, people names like developer name, data architect name, float throughout the book. If you hear Dana, Dana is a data architect, and she's thinking about all these terms. Starting from chapter one all the way to the last chapter, you can hear Dana's voice as a data architect, giving her input and that kind of stuff.

Dana starts thinking about, what is the data domain, how I can separate, and that kind of stuff. Once we take Dana through the thinking of what is a data domain, we have laid out the concept of breaking apart a monolith. There's a lot of talk about how do we break apart a monolith and how it's hard and how it's not easy and that kind of stuff. We lay out a five-step process on which people can go about breaking apart a monolith, like of course applications have to be live when we do this. Bringing in some concepts from the database refactoring book into this concept of how you can break things and then lay out a five-step process for that.

At that point, we said, okay, this is good enough to talk to people about how to break apart monolith but we thought there is a topic that we have not dealt a lot. Like nowadays, the amount of choice you have in terms of what technology to pick for a database or a data storage at a very high level is immense. It is confusing, it is on which all trade-offs are based. Like, do I pick a key-value versus do I pick like a graph or do I pick a cloud-native database like Redshift or Aurora or something, or do I pick snowflake or, do I pick CockroachDB? Those choices are immense.

We broke apart these choices into, I would say, eight or seven distinct buckets. Each bucket will got certain rating and in The Fundamentals of Software Architecture book, Neal and Mark had come up with this rating system for certain things. Mark especially encouraged me to think about rating system for database choice and we ended up doing that for these seven or eight different big buckets of things and giving them ratings around software access. Like, would you use this for consistency purposes? Would you use this for throughput purposes? Would you use this for ease of developer? Sometimes that's a choice. How easily can I use this?

The other choice would be around tool support, if you're using some SQL language, does the database support that particular tool choice and things like that. Giving you a star rating system on those databases so that it helps the reader to make a choice. Now, of course, every reader is in a different context, but given that context and given this star rating, maybe make it easier for them to make a choice on what technology to use and what database storage mechanism to use in their architecture. I think I focused more on the operational side of the data. Let the Zhamak focus on the analytics side of the data.

Rebecca: Okay, so, Zhamak, what would you want readers to come away from in the sections? I'm going to take a wild guess and say it has something to do with data mesh?

Zhamak: It does. It does. In fact, I contributed to this book while I'm deep in the data mesh space on writing a book on data mesh. I think it was a little bit challenging for me to step back and stay away from my emotional state. From the evangelist of data mesh to really sink in with an unbiased view, what are the trade-offs between the different analytical architectures because this book was about that, was about as promotes the different, trade-off, a framework to give users to make decisions for themselves.

I feel my job was a bit easier than promoted, even though that's in the operational space there's so much. Like there's so many complex decisions to make around the caching and the replication and the mode of access. Those exist in the analytical space as well, but from the bigger picture architecture perspective, there are not that many different architectural styles. In my section, I think what I would like readers to take away, first put themselves in the shoes of the Sysops Squad and see why they are actually making decisions around their analytical systems. What's driving that, what business decisions are driving that.

We have that narrative in place around how data can optimize the business of Sysop Squad and the experience of the customers, and then look at the moment there's three macro architectures that exist: data warehousing, data lake, and data mesh. I know there are some technologies in the middle, such as lake house, the lake want to look like a warehouse and warehouse wants to look like the lake, but at the macro level really, those are these three big decisions or big options that you have.

Then look at the advantages and disadvantages of each of those. We have that laid out in the book. If you're starting and you're not a very large complex organization, a very centralized, consolidated approach like your warehouse may be okay, but know that would limit the functionality that you can run on that warehouse. Know that there's going to be extreme partitioning between the actual domain knowledge of data and where the data gets modeled? The domain knowledge is really within the tech and line business domains, but now you're moving the data away from that to a data warehouse and the data warehouse team.

If that's not good enough, have a look at the advantages and disadvantages with Lake, that Lake perhaps it gives you a bit more flexibility because it doesn't require that rigid schema modeling of a warehouse, that really tightly coupled like Snowflake's schema with bring some rigidity and slow you down. It doesn't have, that this is less structured, but it's also more prone to going stale, more prone to have difficulty understanding the domain context within this data that we dumped into the lake.

If these are not good enough, then move to data mesh and look at what data mesh advantages and disadvantages are and what we try to do. I think I've talked about that enough but really it's about getting that agility and the speed that you want and bringing data usage and data consumption as close to those operational teams and as close to the operational databases, but yet realize the respect the differences between the two.

Data mesh really lends itself to modern architecture principles and practices around domain modeling and around loose coupling. It allows you to really have that holistic architectural conversation, not just a data conversation in an isolated fashion. In the book, then we went through a little bit deeper on the architectural aspects of the data mesh. We introduced this concept of data quantum, which I have shamelessly borrowed and stole from you, Rebecca, in fact. Maybe you want to talk to that a little bit.

I love this concept because when I recall the definition that you have in the book, you wrote evolutionary architecture is the smallest piece of your architecture that structurally has all of the elements that it needs to autonomously do its job. In the case of data mesh, data quantum is the competition and the data and the policy, all of the structural components that makes a domain-oriented data autonomously, useful, alive, timely, respecting privacy, all of that.

It's a new concept and I know the word can be a little bit alienating. People go, "What do you mean by quantum?" I love the concept and I took it from evolution architecture. We go a little bit deeper on what is this and how does this fit this data quantum into the bigger architecture.

Rebecca: Given the fact that you both are on this call and we've talked a lot about how notions of data have evolved over time and such, and yet, it still does feel in particular, Zhamak, towards the end of what you were talking about, that there still is a distinction between the domain of analytics at using domain in a different way, but that you really should think about analytic computing and operational computing.

That they are still different even though as Pramod mentioned earlier, many of our operational systems now require analytical data. Do either of you foresee a time when this hard distinction between operational systems and analytical systems are just going to go away, or do you feel there's something fundamental about that divide?

Zhamak: I can have a go at this. It goes back to that question of what is architecture. How do we decouple responsibilities? We still have that delineation of responsibility and then how do we integrate these pieces so there is this continuous feedback? I think maybe there will be a point in future that our technologies are just so universal that it doesn't matter that I'm running a transactional system that requires a certain response time, at certain workload, a lot of reads and writes at the same time, and the teams that are maintaining that, really caring about the current states of that system.

Maybe there is a time that operational systems, we can come up with technology that universally satisfy those. At this moment in time, I know your question is really about future, but if I look at just what I can see ahead of us for a little bit longer, those still have certain characteristics that are quite unique to them. As an example, the relationship between compute and storage, the relationship between code and data in the operational world, is that in a way that you write the code, you write your imperative code. As part of running that code, you’re generating states that then you persist. That the state is really here to be supportive of that computation.

In the analytical world, when that data gets changed, modified, and transformed to be used, the relationship between competition and code changes. In the analytical for in machine learning model, in fact, is the data that is feeding the computation and making decisions. We are finding patterns and retraining our machine learning models. The relationship is different. The nature of the access is different. Why they need to be integrated, tightly integrated, feedback into each other, operational data feeding into analytics, feeding into this temporal view and aggregate view of the data and the intelligent feeding back to the intelligent API's within the operations.

As far as I can see right now, I think they have a delineation of responsibility. That requires us to think about them as slightly different but yet integrated.

Pramod: Yes. I would agree with that. I also add a little bit around even within the analytical side or the operational side, I would say 20% to 30% is in that other circle, If you draw a Venn diagram of operational systems and analytical systems, there is some commonality. That commonality is where the integration happens tighter. The other things that don't intersect at all, that's where we tend to think of them as separate things like this deep analysis, our data scientists digging through the analytical platform to find patterns and things like that are on the operational side trying to optimize the system to serve, give more throughput or serve more transactions and things like that. Those two don't have anything to do with each other in some ways.

At the same time, the integration batch of data from analytical side or from operational batch to like, how can I get operational data faster to the analytical platform is a question that people are asking now more than they used to probably five to 10 years back. People would just say, how fast can I run this batch? Two times a day, three times a day? Now, people are asking, "How real-time can I get this data?" They are no longer talking about batch and that's a change in thinking on like how much integration that is needed into these two sides of the organization or of the architecture as such.

Zhamak: I think one way to think about how do I separate and how do I integrate is really think about this domain concept of domain-oriented design. Organizationally, we have decoupled ourselves over the last decade around the business and business domains and aligned our technology and services, and applications with the business. What we can do is to say, okay, so we have all of these wonderful different domains. We have the order management. If you're in retail, we have e-commerce, we have customer management and they have a set of services and technologies to serve the business outcomes.

Let's extend that to now analytical data-sharing and analytical data-capture and analytical use cases within that domain so that integration and that feedback loop becomes quite tight and rapid because we have now this domain alignment, and of course, we have then interdependency with contracts across the domains but we move away from this functional siloing and decomposition, analytical and operational to a world of decomposition around, really, domain outcomes and domain capabilities. Within the domain if zoom in, yes, there will be some analytical pieces and there will be some operational pieces.

Rebecca: One last question for each of you, we now have this thing called data architecture. What would I have to do to be a data architect? What advice would you give to people? How should they pursue thinking about some of these questions of data architecture?

Pramod: I think data architecture in some ways is like having empathy for the users of data, thinking about how this would be used. Don't think of it as a thing to control or a thing to design for. It's more about how is your data going to be used and where it is going to be used and then how well can you provide the data back into the organization or into the other use cases, whatever that may be. Then go down from there in terms of tools, technologies, people, process, all of those stuff come into play.

How can you organize your team so that the data architecture is working good for them, is being more productive for the developers, and things like that? How can we organize the data in such a way that it is productive, again, for the business users and things like that. I generally focus on that side so that the architecture is providing for the end-users in such a way that it is useful. It is making them productive. The end-users of an architecture are of course the business people, but they're developers. There are other stakeholders, there are many different kinds of stakeholders.

I generally tend to also work with this software. We are not necessarily fighting with the software people. Data architecture is part of software architecture. We just care more about the data and they care more about the software, so how do you come together to care about the architecture? If you look at it that way, then you can work within the team of software architecture to deliver whatever architecture the company needs or the organization needs.

Zhamak: I really want to emphasize that last point. I think data architecture or data architecture decisions should not be isolated decisions. They have to be part of a bigger picture architecture and that bigger picture architecture is both around applications, integration of their data into ML and analytics, and integration of that back into the application. Application system architecture, analytics architecture is all just one big architecture decision.

Of course, when you go deeper, you might need to make specialized decisions but a lot of the decisions are about the integration, the harmonious kind of integration of these independent pieces that need to work together. Maybe we're all talking about architecture and then yes, there are some specialized fills in the middle, but at the end of the day, we're making arch decisions.

Rebecca: Excellent. Thank you, again, Pramod, we've been cycling you through a few of these podcasts lately, and thank you, Zhamak, for joining as a guest, as opposed to a co-host on the ThoughtWorks Technology podcast. I hope people found this discussion useful. I personally found it fascinating, so thank you both Zhamak and Pramod.

Pramod: Thank you, Rebecca.

Zhamak: Thank you, Pramod and Rebecca. It was wonderful.