Tackling system complexity with domain-driven design

Episode transcript

[Music]

Alexey Boas: Hello and welcome to the Thoughtworks Technology podcast. My name is Alexey. I'm speaking from Santiago in Chile. I will be one of your host this time, together with Ashok. Hello, Ashok.

Ashok Subramanian: Hello, Alexey. Hello, everyone. I'm Ashok, one of your regular co-hosts. I'm speaking to you from London. Looking forward to recording this podcast along with two of our guests today.

Alexey: Yes, exactly. This time we're delighted to have Prem and Karthik Krishnan here with us. Hello, Prem. Would you mind briefly introducing yourself?

Premanand Chandrasekaran: Yes, absolutely. Thanks, Alexey and Ashok. Glad to be here. As you know, my name is Prem Chandrasekaran. Been a long-term Thoughtworker, and played a variety of technology-centric roles. These days, I play the role of head of tech for our rest of Western Canada markets in North America. My passions are in building large scale distributed systems, and very glad to be here.

Alexey: Thanks a lot, Prem, we're delighted to have you with us. Karthik, could you tell us a little bit about you?

Karthik Krishnan: Absolutely. Firstly, thanks for organizing this, Alexey and Ashok. I'm Karthik Krishnan. Been a long-term Thoughtworker too. In fact, coming up for my 10-year completion in April of next year. Most of my time at Thoughtworks and before has been on the technology side of things, being a developer, played various roles from tech leads and architects and stuff like that. Currently, I play a role of technology principal to one of our retail accounts at Thoughtworks.

Alexey: Amazing. Thanks. The topic this time is Domain-Driven Design. Prem and Karthik have recently published a new book on the topic. It's called Domain-Driven Design with Java: A Practitioner's Guide. We have all heard of Domain-Driven Design, it's one of those powerful and deep topics with many, many different applications. No wonder it's been around for 20 years or so. Eric Evans published his blue book back in, what, 2002, 2003, something like that.

Maybe just to kick off the conversation, maybe we can start with the definition. Prem and Karthik, what's your view of what DDD is?

Prem: Sure, I can take that. At the end of the day, we are all building software so that we can deal with complexity more effectively, arguably. Every problem has some amount of inherent complexity associated with it. Then if you've got a hundred requirements, you have to build for those a hundred requirements. There is no getting away with it. However, there is no need to deal with all of that complexity in one goal. DDD provides us with a set of principles to help us deal with that complexity. It does that by doing a few things. One, it helps us break down complex problems into more manageable ones. Then you can deal with each of these independently, more or less. Finally, also allow these parts to interoperate very seamlessly. It does that through a variety of things that it evangelizes, like the notion of a subdomain and bounded context in ubiquitous language and a whole bunch of vocabulary that it introduces. The work is all based on Eric Evans' original seminal book on this topic, way back in 2003, as you said.

At least, that's how I would characterize Domain-Driven Design. I'm sure there are much better definition than probably Karthik can allude to some of those as well.

Karthik: Oh, absolutely. In fact, the way we started looking at Domain-Driven Design as a discipline is where you try to grasp what the domain is, the real problem domain itself. One of the key challenges that we see is not having a common language between the technologists and the business folks. Aligning on a shared language is one of the key tenaet of Domain-Domain Design. Talking about, I think Prem touched upon it, embracing the complexity. Try to see whether you can avoid adding more accidental complexity to the already complex business problem.

One of the other things that DDD talks about is keeping separate domain models for a particular context. That's how I look at defining DDDs.

Alexey: When we look at the technology landscape today, and how complex it is to building modern systems, tackling complexity is something that's still very, very relevant. Is that how we feel about the concept and DDD and the role of DDD in modern software development?

Prem: Yes, look, so we get that question quite a bit. Why is DDD relevant right now? The book on the subject was written two decades ago. The way that I see it is a few things have happened. One, Moore's Law obviously has slowed down, which means that now we are not building more and more powerful processors, which means we can't cram more and more logic into one thing. The second is the advent of the cloud. Previously, we could buy a big, large mainframe and hope to put all of our business functionality centered around that piece of hardware. Whereas with the cloud now, it seems to embrace this model of working with very small commodity hardware which may not be as powerful individually but collectively become infinitely powerful.

DDD started talking about this, I guess a bit ahead of its time. We were still very much in the age of building monolithic applications. With the cloud becoming such a ubiquitous thing, now there is a need for us to start looking to embrace distributed architectures. Somehow that has coincided with the whole principles of DDD also maturing in the last two decades. That's why it feels like DDD and all of the concepts surrounding that topic is a lot more relevant right now because of the kinds of ecosystems that we have and the kind of problems as well. We're processing the most data that we ever have, and that is only expected to increase exponentially, and continue to do so in the coming years. That's why it feels like the principles are very, very relevant right now.

Karthik: Absolutely. Just to add on, I think Prem, you touched upon it, the problems existed, and we had certain ways of approaching this from a solution perspective. Today, the amount of technology advancements that we have, and the fact that I can do things like function cell service, for example, now, how am I going to look at distributing my application? How am I going to break that problem in a way that it's still not going to-- like what Martin Fowler calls a distributed monolith? How am I going to be able to bring all those concepts together and still be able to leverage all the advanced technologies? I think it's still very, very relevant to talk about DDD.

Ashok: When you talk like the description and people look at designing or building new systems, you obviously have the good fortune. I suppose that some people have a good fortune of starting from scratch and being able to define these bounded contexts very well.

Most systems, I think like most freshman's technologists, will, again, live in a world which is a bit messier than that. First of all, would it still be relevant in those situations, circumstances, and then if it is, what is your advice or guidance? If someone was to read your book and look at some of the concepts and advice, how do we take them into a world that is maybe a bit more messier than what you would describe it?

Karthik: I can take a shot at it, Prem. Basically, we cannot ignore the current organizational structure and the constraints that we are playing with. Like you said, most of the times we end up in a brownfield scenario where we are existing in a system that's already built over the years, and now we are trying to see how am I going to apply DDD here.

One of the key things about when Eric talks about bounded context is, not just going into this an ideal world scenario and start breaking down the problem, but look at your problem in the current context. Basically also involves including the way you have structured your teams, the way these systems are already connected in the current ecosystem. It is not about trying to go and start from a fresh whiteboard and start from there. It's about looking at your current constraints, looking at your existing solutions, and start iteratively breaking them down. In fact, one of the key things that we try to stress is, don't try to get this bonded context perfect. Don't try to get these boundaries perfectly on the first round. You are going to evolve over the period of time.

Prem: One thing I'll add there, also there is this concept of some portions of your ecosystem being more important than others, the ones that actually give you your business differentiation. The idea, folks call it the core subdomain, which is basically the one part that, or parts that give you the biggest bang for your buck. Now, the first order of business is to identify what those portions are that actually only you do very well, or you do the best. Then now focusing on those parts, first and foremost, isolating those parts from the rest so that now when you have to pivot, you are able to do this at a pace that you decide as opposed to being at the mercy of somebody else.

In a Brownfield application then, the way that one would look to apply DDD is to try and isolate those core portions of your system from the rest. This is, again, not easy. You may be in this situation where you are this dreaded big ball of mud where everything is so cobbled together where it's really, really hard to work with, but you have to start somewhere. The whole idea is to use the principles of evolutionary architecture, which our CTO [Rebecca Parsons] and Neal Ford have talked about in their book. A lot of the principles that are described there very much apply. Then you use those techniques to now start firstly identifying the core and then isolating it, and then building from there, as opposed to trying to solve world hunger.

Alexey: One thing I particularly find interesting in applying DDD concepts in those scenarios, is that it also helps connect technology to business conversations and conversation about functionality and those kinds of things. You don't have to start just from a technology perspective. You can really take a bird's eye view of where you're going to start, and divide and separate the components and context around that. I find that quite powerful.

Ashok: Premanand, I think when you were describing this earlier on as to why now, you also referenced we are generating way more data, and I think there's also a lot of interest among organizations in terms of trying to tackle their data ecosystems. Would you say similar concepts apply over there as well? Could you use some of the techniques that you described around DDD around software into the data space as well?

Prem: Yes, absolutely. That's a great question. In fact, one of our colleagues, Zhamak Dehghani, has actually talked about this fairly eloquently. She's also written a book on the subject called Data Mesh.

At least when DDD was originally described, it was described in the context of transactional systems. Or at least a lot of the examples seem to be in the context of transactional systems. For analytical systems, they still continue to remain mostly untouched, where we still have these monolithic data warehouses or data lakes and so on and so forth. Zhamak came about and talked about this in her series of articles on martinfowler.com, and then subsequently the book where she describes how the principles of DDD can be applied in the analytical space as well.

Data mesh in the analytical space, microservices in the transactional or OLTP space, I think those are manifestations of DDD done well. If you've got a great set of microservices ecosystems, then you've probably applied DDD most likely very well. Similarly with data mesh again applied DDD well, and you get a series of well factored data products.

Ashok: Given that you both have actually written a book on this topic, is there something on the book that you can give an example or something that would give a flavor of how someone, if someone was looking to adopted, is there a real world problem or something that you've taken and describing the book that our listeners can maybe look at and explore?

Karthik: Yes, absolutely. In fact, when we started writing this book, we said, Hey, we don't want to make this again, it's another conceptual abstract thing. We were like, how do we take our own existing experience, and in fact, take a real world problem, which one of us have actually involved in our past life, and try to actually take that through all the way from explaining the problem to the technology folks, and try to get this shared understanding and break that down from a problem perspective and taking it all the way to the solution. We said, let's actually start with something like that and play that through.

In fact, what we picked was more from an international trade business, pretty much what, there's a concept called letters of credit, which is about companies or people buying stuff from another country and trying to pay for it. That's a complex business, and I had worked on that in my previous life. What we tried to do, in fact, is pre playing that the technology person, and I played a subject matter expert, if you will, from a domain perspective, and engage in those conversations, trying to apply some of those techniques to arrive with those common understanding and so on.

Absolutely. You should check it out from that perspective. Looking at the problem, breaking them down all the way up.

Prem: We get this a lot as well. I mean, why would you write a book on DDD now? There are already so many books on the same topic. There is the blue book, there is a red book, and then there is a distilled book. I mean, there's so many of them. Here is the thing, and Neil Ford says this well. He says that the blue book is the one that everybody wishes they would've read, but never actually have read. It's a tough book to read. At least it was for me. Looks like a lot of others share that sentiment.

The point is that although the concepts are really, really powerful, they are also expressed in a bit of an abstract way. This is not a criticism. Arguably because those ideas were expressed abstractly, that's why they have endured for so long, and will probably continue to endure as well. What we wanted to do was to take almost a practice, a layman's outlook to this, and that's where we took the approach of how do we translate these abstract concepts into something that we can apply in our daily lives? That's what we have tried to do starting from requirements. We have also hypothesized two ways where you're starting from scratch in a greenfield scenario. Also if you were in a brownfield scenario, for that same letters of credit problem, that's what we have tried to tackle in the book. Hopefully, it's a reasonably interesting read.

Ashok: It's almost like the way you were describing in the red felt like. Actually it's not just the way you seem to have expressed in the book. It's not just a technology problem alone, but it's that collaboration that we often hear is really necessary as we are trying to get both the business domain experts together along with technologies to be able to get the right system boundaries in place.

Prem: Yes. In fact, our thing is that without the domain experts, you just cannot do it. You do need to have a very clear understanding of what you're trying to do before you delve into how you are trying to solve it. Again, yes you have the domain experts, but there doesn't need to be a very clear distinction between the domain expert is the one that does all of the thinking, whereas the technology expert is the one that actually does the translating. Idea generation is not something that needs to be monopolized by just one set of people.

The whole idea of DDD is to make sure that the collective understanding of the entire team or teams is elevated to a point where ideas can come from pretty much everyone. That is based on the premise that we are all speaking the same language, which means that we all have the same understanding of the problem, and thereby are able to contribute to solving the problem much more effectively.

Yes, domain experts, obviously you do need that visionary, at least one of them, who is able to think of what the future can look like and how we can change the world, sort of, but that visionary can then now make their ideas or express their ideas in a manner that others can also start contributing just as well. Now as a collective, we are much better off.

Karthik: In fact, a lot of those innovations come from the solution world sometimes. The domain experts would not perhaps even anticipate a potential possibility that the solution world starts revealing. It is certainly a two-way traffic, absolutely.

Alexey: Yes, it's interesting because it goes back to communication among people with different backgrounds and expertises and leveraging creativity. That feels like a very, very modern topic everywhere. It's quite interesting to see those techniques leveraging those kinds of things and making them available, and making them possible at the end of the day.

That's something I'm curious about. In the book, you've been looking at 20-year old concepts and thinking about how to apply them from a practitioner's perspective. We know that the technology landscape has changed a lot over this time. What have you seen that has remained stable over time about DDD, and what has changed? What are some new things or some things that are not valued anymore? We talked about some things that are stable, like communication and creating ubiquitous language, but what's your view on that? What remains stable? What changed?

Karthik: If you look at the specific tenants that DDD talks about, there is a set of tenets that comes under tactical design. Again, Eric Evans doesn't actually segregate that as two different aspects, but when you look at it, things that you actually apply on a tactical basis on a day-to-day development basis, those concepts are not new at all. In fact, a lot of those tenets, things like value objects, things that we talk about, contract testing, those things existed, but now when you think about DDD and look at the value object, I want to be able to think if I'm looking at a financial domain, I want to think things like money, things like currency, things like forex trade, which I would've just done that based on my oops learnings.

Today when you actually look at it from a DDD lens, they actually made a lot more sense, but they were not anything new that DDD brought into the world. When you look at the strategic side of things, I think that's where, like Prem mentioned when the book came out by Eric Evans to today, some of those techniques that were a lot more harder to implement are getting better. Things like even storming, things like domain storytelling, those techniques are evolving a lot more where people are able to use them effectively in getting that shared understanding. I think that's how I see that certain things do not change and some of those are made better easier today.

Prem: What's changed really, to assess on that, is the availability of a lot more tactics. Like Karthik said, even storming, domain storytelling, contract testing methods. We've got frameworks like PACT and Mountebank and so on and so forth. All of these arguably drew inspiration from the literature that DDD popularized and then made them available, which then made practicing DDD a lot more easier. Still pretty challenging, but definitely easier than what it was when Eric originally wrote his book.

Alexey: Yes, I think it makes sense. I think some of the techniques, especially things like even storming and wardley mapping, we see a lot of teams actively using that, and because of its widespread usage over time, they also start coming more into common usage and common capital, things that you can dip into your toolbox that then help you identifying assist. That makes sense.

Prem: Yes, one more thing that I want to add there, which I did not say earlier, is this whole idea of bounded context. Everybody talks about sub domains, everybody talks about bounded context, but then the question really comes is, how do you identify the boundaries of these things called bounded context? Turns out that there is no right or wrong answer here. Karthik originally talked about it, where it's-- you can draw inspiration from existing things, like your existing organizational structures, the things that your domain experts do and talk about and so on and so forth, and draw some inspiration in terms of where those boundaries lie. There's no specific way to say this is right or that is wrong.

With these new techniques now, it has become a bit easier to apply them, especially things like event storming, and you mentioned wardley mapping as well. I know what the purpose of what I'm trying to do is, now I'm trying to actually going to break it down. I identify a process flow, and that's where domain storytelling, even storming, and all of those techniques actually really help you. That's definitely a huge evolution in that space.

Karthik: Good that you brought wardley mapping here. One of the key decisions that you make as part of figuring out treating those boundaries differently, treating those bounded contexts differently, I think Prem touched upon things around core domain versus supporting versus generate, wardley mapping certainly provides a good amount of inputs to those decisions as to what is really your core today which could become a supporting or generic or period of time as things evolve. wardley mapping certainly gives you that view into looking at your sub domains differently.

Alexey: Yes, that's interesting. Going a little bit to the process of writing the book, we all know it's a major undertaking, and having a look at something as deep as DDD and bringing a fresh view on that. Do you have any words of wisdom for people thinking about writing a book and or any fun stories about your process of writing this book that you wanted to share?

Prem: Yes. When we were originally approached to write this book, we talked to Neal Ford, given that he's done this so many times, and his first piece of advice was to not do it. We chuckled and he was like, "Nobody reads books. Prem and Karthik, don't do it. If you are going to do it, then understand that it's a massive undertaking." True to its word, we were like, "No, we want to see how it feels." We did undertake the thing. It is quite grueling. I can speak for myself, without Karthik nudging and pushing me along, I think I would've given up very early in the process.

I'm so thankful that it was him actually doing this in a non-aggressive way, in a very endearing way, I might add, which really kept us going. We could see the finish line, and this was in June or so, and then just pushed through it, and it was great to do it.

Karthik: Absolutely. In fact, few things that came to my mind when I even tried to sign up for this project was, can I actually clarify whatever I'm thinking in a better fashion in my own mind. A lot of those concepts I think I knew for the last 20 years, but when I'm trying to replay that in my own mind, things were actually going all over the place. I'm not able to define this a lot more succinctly, so how do I get better at that? That was one goal, and I think hopefully some of them were achieved, I think.

The other aspect is articulating to the audience. How do you play it down? Just because you think and you have read about some of these things, practiced it for some time, how can you bring that down in a way it is conceivable for the audience at different levels. I'm able to actually help articulate some of those thoughts better.

Those are the two things that I kept in mind as part of bringing this whole book thing. I think Prem claimed a huge part in terms of, "Hey, we should talk about this. We should talk about this," and bringing a lot of things from outside in, and, like you said, and we had complimented each other with our own strengths to get this book out. Very happy about this at the end of the day.

Prem: Yes, and we did this all in source code, so that's one thing. We probably will share how we did that. All of this is written in AsciiDoc files, and then we have a build which produces PDFs and Word docs and uploads it. We have a pretty interesting process as well to actually create the artifact. Of course, the publisher has their own as well, but we have diffs and all of that which Karthik built. Lots of really good things, and I think we'll probably share that in some other forum. Yes, there's lots of good stuff that we can share with others while we do this. Yes.

Karthik: I wanted to add one more thing. The entire book was done with remote pairing. We literally did this all through the pandemic, starting somewhere at 2020-ish, now, all the way till 2022, pretty much through the pandemic. In fact, I met him just last week in the last three years for the first time in person. That journey of actually sharing both of our screens and actually exchanging ideas and each of us taking turns in typing the content, yes, it was certainly an experience as well.

Prem: Yes, we truly paired on it. We truly paired on it, and remote paired on it. Yes, initially, we just couldn't believe that we could actually pull this off. It was like an Alice in Wonderland kind of journey. It's great to still be on the other side. Yes, great experience. Others should go through this as well, but it is still grueling.

Alexey: Amazing. Thanks for sharing. Thanks for sharing.

Ashok: It seemed there was, that might be a topic for another podcasts entirely, just talking about how do you actually go about writing a book using the techniques that you both seem to have shared and evolved over this. I think the thing that seemed to stand out for me was, it's like you'd recommend someone goes through this journey, definitely, if they want to really internalize some of the concepts and be able to clarify the best way to learn is to try and teach the topic, seems the one that stood out for me over there. Now definitely you can point to a book that people can read if they want to learn more. That looks like you've at least achieved the goal that you started off with. That's good.

Prem: Yes, it's definitely one thing off the bucket list, for sure. Whether anybody reads it or not, we don't know how valuable it is. We don't know. It's definitely something that was a big learning journey for us. Lots of things got cleared as we did it as well. Definitely recommend others doing something similar, maybe on the same or different topics.

Alexey: Some final thoughts. Any final thoughts to share, or maybe where can people learn more about the DDD besides the book, of course? Any final recommendations on that?

Prem: Yes. There are a couple of things. One, we do internal boot camps here at Thoughtworks. We have worked with close to a hundred people, probably even more than that by now. We started in 2022, and it was almost us going through the material that we were writing as part of the book. We've got some good feedback. We recently did a full day workshop at QCon San Francisco where we did a DDD bootcamp and also wardley mapping and team topologies, which are very related topics. That's that.

Then there is DDD-Crew on GitHub, DDD-Crew. They have some amazing material that people can use to actually apply. Nick Tune is another person that has a lot of really, really good and valuable material in terms of how to work practically with some of these principles and get pretty far. Those are things that come to mind. Of course, there is the Blue Book and the Red Book. Those obviously are always going to remain two seminal pieces of work.

Karthik: I think, in addition, I would also say people should try and apply some of these techniques. Don't try to do this big bag. Start somewhere. Try to apply these techniques in your own world. There's so much context that actually plays into this whole aspect of applying DDD, so start somewhere. All these other materials that Prem talked about can certainly compliment and help you in the journey.

Alexey: That brings us to the end of the episode. It was a great conversation and great to have you with us. Thank you so much for joining.

Prem: Thank you very much for having us.

Karthik: Thank you.

[Music]

 [END OF AUDIO]