Delivering strategic architectural transformation

Podcast Transcript

Mike Mason:

Hello and welcome to the Thought Works Podcast. My name is Mike Mason and I am one of your hosts, and I'm here with two guests today, Zhamak and Ryan. I'll ask them to introduce themselves.

Zhamak Dehghani:

Hi, I'm Zhamak Dehghani a tech consultant from our San Francisco office.

Ryan Murray:

And hi, I'm Ryan Murray. I'm the director of our digital platform strategy team and I'd be also based out of our San Francisco office.

Mike Mason:

And so today we're going to be talking about strategic architectural transformation and all of the stuff that goes with that. The thing that I really love when talking about architecture is that everybody has a different viewpoint on what architecture even means. So if we're talking about strategic architectural transformation, Ryan, can you just give us 30 seconds on what you think that is? What is architectural transformation? Why do we need to do it?

Ryan Murray:

Sure Mike. I think there's a lot of ways to interpret architecture that are all valid. We do a lot of work with customers around architecture and platforms, and those words have meanings at lots of levels. At the level of strategic architecture I think what we're really talking about is how broad systems interconnect within the enterprise to allow the macro value proposition to be delivered and that's across many teams in many different parts of the technology stack. So less about sort of technology details and more about how information and business capabilities are exposed and flow within the organization.

Zhamak Dehghani:

I almost wish you hadn't asked this question cause there we can spend half an hour talking about what architecture is, but if one lens to look at it is decomposition of the systems or the components at the application level or at the system level or the organizational level. So I think under the umbrella of architectural transformation, we're talking about the whole ecosystem within an organization or within sizable part of the organization and how we decompose the capabilities that organization need to provides and implement that with different systems and how those systems need to share information, share context, and utilize or leverage each other so that the composition of big pieces that are hard to change and expensive to change. I think that's the lens I would take for this conversation.

Mike Mason:

I think we wanted to talk a bit about the drivers for doing this and Zhamak you already mentioned one which was difficulty in changing things. So I think something that has always been an issue in software systems is being able to respond fast enough to a business change. And certainly when we have kind of older, bigger systems, they tend to be slower to change. So I mean change is a driver, is that the only one?

Zhamak Dehghani:

Yeah, I think change is definitely a big driver. And speed of response to the changes is the driver that we see with a lot of organizations, especially the ones that already grown to a sizeable size, a big size and then they need, they've over the time with that growth they've become slower and slower. So how can they now maintain that growth while become faster? So speed of change, but also the nature of change is also changing. Like some of the time in the past maybe we see organizations that they experienced the growth in the number of users or the number, the market share that they have and they need to respond to that growth. We quickly in some cases, you know new parts of the organization being born for instance in the analytics space that the, a lot of the practices in the organization or units are new, so within the organization they're growing. So I guess the growth of the organization and the speed of change are the two ends of the spectrum.

Mike Mason:

And Ryan, you, when we were talking earlier you talked a little bit about how those drivers are changing and impacting the way that we do architectural decomposition.

Ryan Murray:

Yeah. I mean I think to build on what [Zhomik] said first as a way to lead into that, I think there's other factors that I'm seeing a lot in large organizations different from sort of the digital darlings, if you will, are the importance of business correctness. One of the things that, which I think a lot of organizations are struggling with, is being able to have a capability, whatever that capability might be, be kind of factored into the right view of the different dimensions of the capability. So if we talk about for example, like customer information as a domain, it's a very broad domain and I think a lot of companies struggle from sometimes both of these problems of one is they've tried to be too monolithic in thinking about this thing called customer. When in reality a customer has many different points in which it interacts with the organization. You know, and from kind of establishing an account with an organization all the way through to transacting.

Ryan Murray:

And those are very different problem spaces. And when you do a kind of master data management approach, you can try to lump too many things together, which then really affects speed of change. But there's also the other side of that, which is when similarly customer may become too distributed across the organization and then customers interacting with your organization across, what [Zhomik] was saying, a lot of these different business lines that people are getting into, they don't have a cohesive experience of their relationship with the organization because they're actually the architecture underlying it is fractured. And so I think those are two interesting, that's sort of the scope of the way we look at business concepts like customers or products or some of the other foundational things is a really big part of what we need to look at when we're thinking about strategic architectural change.

Ryan Murray:

Because again, there's this constant tension between expediency versus correctness and flexibility. But I don't feel like we hear people talk enough about correctness in the sense of people. Customers don't like when they're suddenly their history with an organization suddenly through some channel is invisible. Or when they're, the way they transact with the organization is different and inconsistent. Right? Or they're not able to find things about their relationship to the organization. And so it's, there's a really interesting tension between expediency of build and autonomy that we talk a lot about in the agile world. And then also having this kind of idea of correctness and then being able to flexibly build things that are correct but also build them quickly, I think is the big tension that architectural transformation or architectural design is really needing to focus on.

Zhamak Dehghani:

Yeah, I think what Ryan says is really important. Like we always talk about the main forces of change, speed and scale, and we forget the counter forces that need to kind of deal with that, which is things like integrity and correctness and security. So we compromise security, we compromise integrity when we go fast and when we go bigger.

Ryan Murray:

Yeah. And organizations have very different, or let me say that customers have very different expectations of organizations. And one of the things I've been facing with customers recently more frequently is this idea of the mental model change that you need to make when you start to think about moving fast, the classic moving fast and breaking things. And of course we don't want to actually break things. But on any given day that I use a Google product or I use some of the other large scale web products, I may or may not find a different user experience on that day. But most of those products I either don't pay for, I pay very little, and so I just fundamentally accept that as part of the, seeing that evolution and experimentation happening to me as a consumer is something I accept.

Ryan Murray:

But there are plenty of organizational situations in a supply chain or in, kind of a business to business data exchange or commerce where that sort of unexpected change, it leads to significant business continuity problems. And so those are very different environments and they need a different approach to architectural validation and the meaning of speed and flexibility is actually quite different.

Ryan Murray:

So knowing, one of the things that we talk a lot about in our digital platform strategy practices is really looking at the architectural layers and realizing that on a podcast, a diagram would be helpful, but we don't have one. There's, what you can do at the edge of your architecture where you're touching the customer through various channels like mobile or web or conversational and the experiences that you're driving, there's a lot more room for experimentation and architectural kind of flexibility and even duplication than there might be in the level of the core systems like core customer management, core transaction order management. And so each of those levels requires very different thinking about what transformation means and what scale it's needing.

Mike Mason:

I would suggest that another difficulty is all of the headline interesting buzzwordy sexy stuff is coming from the digital darlings and the interaction points around the edges because people can see them and they're sexy to work with. Whereas a lot of what we need to do is actually more difficult things than just building interesting UIs.

Ryan Murray:

Well a great example Mike of that that I've been bumping into quite a bit now is sort of API first is obviously been a movement now for some years and I think it's extremely important as a way of thinking about how to find those boundaries that [Zhomik] was mentioning earlier, and sort of API's, for me I think of them broadly as an interface meeting, transactional interfaces, whether they be rest or RPC style. Also coupled with kind of event streams and how you, a domain advertises things that are happening within it. I think that's all part of a boundary, a domains boundary API.

Ryan Murray:

But one of the things that I see happening a lot now is as the API revolution is taken hold in the enterprise as well is what I think might be a repeat of our kind of soap whizzed old days of just instead of really thinking about business transactions as the fundamental exchange documents, the way soap started, I think we're, we're heading quickly towards the automation that was created around code generation led to just exposing kind of methods off of objects, very willy nilly and making them networked objects and obviously all the dangers and problems that come from that.

Ryan Murray:

I feel like with API's we're starting to see something similar where a lot of the tooling is improving but now organizations are getting kind of reports that say hey you should have an API strategy and you should build these 1200 API's and looking at them they all kind of look like RPC hidden unrest and you know that's a great, it's going to be the soap problem all over again where you're barfed out a lot of low level functional detail and you end up with a platform of API's that actually doesn't get you any of the value that you were looking for in terms of flexibility, correctness, interconnectivity, you end up with, again, just a new technological stack in the same tangle.

Zhamak Dehghani:

I think that's a really valid observation. I see that as well. But I think it's because defining API's, finding the boundaries in the first place and then defining meaningful interfaces that can extend and there are developer friendly, consumer friendly is really hard. It's the art part of the architecture, software design and there's not so much science around it. So the tools will drive the change and the tools have, it's easy to just play with, with the tools and play by the rules of the tools without really thinking about what API should be exposing. How it should be decomposing the landscape of the capabilities and how can I evolve it. Because we're not good fortune tellers, however we decompose it, it would change. So how to build evolvability into it. And I think even us as a practice, we continuously try to come up with new ways of, principles and practices underneath the composition of the landscape and creating meaningful API's or streams or boundaries.

Mike Mason:

Yeah. And I think, I was just going to say, I think one of the things that exacerbates that is the fact that there are a lot of uses of package software in the world and in general, some of those things do not lend themselves to being modular and having well-defined boundaries around them. And in many cases, a vendor is economically incented to try to expand into as many pieces of your organization as they can. And so you end up with actually quite a lot of forces making it more difficult for you to draw boundaries and create decent API's between things.

Ryan Murray:

You know, there's a couple of interesting patterns I think to call out specifically there that I've seen over the over recent last five years or so. And one of them is the semantic problem that occurs when a vendor does try to expand into lots of adjacent areas. Right. And that inevitably a vendor's also incented to make a product that says widely applicable as possible. And so there's a really, that leads to that kind of semantic diffusion, if you will, the kind of dumbing down of the product concepts to be flexible, if you will. And that flexibility doesn't lend itself well to understanding at a technology level. You know, I've seen a number of cases where you end up with sort of data models that are highly generalized with even things like user defined fields built into core, kind of entity models and obviously the expediency of that.

Ryan Murray:

I understand, that unfortunately though when it comes to people seeing payloads on the wire that start to expose those things when it comes to an organization trying to bring new people on and start to develop new capability, that kind of poor clarity about how a product actually works and kind of how it supports the domain really that's one of the aspects of not being able to move quickly that I think we don't think a lot about is not just the inflexibility of code but the incomprehensibility of the way capabilities function in the way they're wired together and packaged software definitely creates a lot of risk in that space.

Zhamak Dehghani:

One of the clients I was working with recently, they were looking to expand their business model to directly facing the customer. And not using the channel distributors and so on. And so they were looking at different packages. And when you look at just the simple commerce space or marketplace space, there are so many players and they almost none of them give you everything you need. You always have obviously custom software that you have to build, but they all overlap with each other. They all leaking into, into each other's boundary. So that's one problem. And then the other problem is they have very strong sense of data modeling that works for them as a package software. So a lot of the organizations end up adopting the internal models of those package software and leaking the identifiers, the internal data structures that they have into all other parts of the business.

Zhamak Dehghani:

So for me, usually one way of validating, if the way we've decomposed our systems is the right way, is through coupling for building self-contained features. So if I need to build a new feature or a new capability and I need to change five systems, which pretty much communicate the same set of reference data, an identifier as across the communication. I've created a distributed monolith and a tightly coupled and a leaky system. For me, one of the very simple tools to test the boundary or the suitability of the decompensation boundary is this idea of access of change, where change is happening, how like looking at the portfolio of the projects and see what capabilities we're trying to build and how many systems do we need to touch for each of these capabilities to be implemented.

Ryan Murray:

Oh this is an interesting pet peeve of mine. Hopefully I won't digress here. The one of the things I think is a solution path to what you're talking about when we talk about what does good modeling actually look like and it's really easy to talk about things that we see not working. I've always found it valuable to try to, I have this terrible term of it called business truthful modeling, but this idea of a box if you're doing logistics, you know boxes have real physical characteristics and representing those in software faithfully is really important. Even if the, in an agile sense I don't really care about all the dimensions of the box or its weight or its whatever right now, taking the time to model the box as it really is, is I find really valuable to that kind of future state.

Ryan Murray:

When you try to do something new, in the end, the business typically changes along the lines of what the real physical value creating things are and if you've taken the time to model them a bit faithfully as opposed to the least kind of minimal work that you can do from an agile point of view. I find that you often get less of that problem of having to change across multiple systems because one of the ways you can, maybe I should have said this first, was one of the ways that you recreate change across multiple systems is simply because the systems aren't complete. And that's different from the what you're talking about, I think, in the boundary sense, right, of I have a distributed monolith because concepts are spread across multiple parts of the system. There's also the systems are just incomplete and so every time I need to do something new, I have to go back and expose a thing that the system might have exposed a long time ago. But agilely speaking, it wasn't needed.

Ryan Murray:

So I'm seeing that a lot also where we're touching multiple systems, not because they're confused, but simply because they all need some small amount of change to bring together the new value proposition. I don't know it's completely avoidable, but I do think taking the time to model things faithfully to the business and exposing things that maybe don't have customers right now but are intrinsically valuable to the capability or to that asset is actually worth doing because while people talk about you're not going to need it there, I was just having this conversation with a colleague today about there was a field that represented a membership type in and when they exposed membership that wasn't a needed thing but they expose membership as an asset and it would have been trivially easy to expose the membership type the first time they did the development.

Ryan Murray:

However it required a number of teams to be coordinated to add that field later. And membership type is just this intrinsic concept. And so there's an interesting balance there of, I know agilely speaking you didn't need it, but at the same time it was virtually free to have just exposed it. It is part of membership. It's critical to somebody. And so you know, you don't get, I talk about serendipity, like great architectures I think enables serendipity because people can see what the core capabilities are and what the assets are and they can imagine new things they can do with them. If you don't expose those things effectively, then that kind of serendipity doesn't occur. So I think that there's another interesting dimension of, kind of how agile architecture and how agile practices also affect future change.

Zhamak Dehghani:

I'm definitely going to take this down a rabbit hole. Now you said something that I've been really thinking about recently because I came from a client that was a post startup. So the hacker culture was very dominant and I started thinking about this aspect from a local optimization versus global optimization. So a lot of the kind of hack it, get it out there, say faith works or maybe some of the agile methodologies falling to the end of the spectrum that is optimizing locally, optimizing for now, optimizing for what I'm building. You know this local feature that I'm building and a lot of the, I guess water folly and traditional way are leaning towards the global optimization.

Zhamak Dehghani:

I'm going to optimize for the next 10 years. Nothing's going to change and I'm going to optimize for all of the organization and all the architectural patterns that I'm going to see and I'm hoping we can double up a set of pragmatic kind of guidelines to see what is the right, how do you find that the right optimization point. Like what you mentioned as an example was designed for the business truthful messes, is that a term that they use, design for the thing that it is that we know it is, that physically exists. Maybe this is one of those guidelines. So having a few of those guidelines to find the right place and the suspect term of local optimization versus global optimization.

Ryan Murray:

Yeah, a shameless plug. We have a good insights article on the Thought Works website that talks a little bit about some of these concepts. But yeah, I think we can do a lot more to do that. And I don't know if it's a gear change, you know that it's the wrong time, but I think that one of the other challenges that I'm seeing in a lot of established organizations that aren't digitally native is the separation between the technology organization and the business, really gets in, makes it difficult sometimes to have these conversations and to make the right investments in the way that you're talking about between global and local optimization. Again, there's a lot of disillusionment if you will on the part of the business with the speed of IT in many organizations, but that unfortunately compounds this kind of expediency, local optimization thinking.

Ryan Murray:

You know, and I know the version you're talking about I think is more in the kind of the developer's heads to some degree, but there's also that project pressure that a lot of people face as well to just, you take too long get things out the door and doing strategic architecture transformation. It really is a partnership between the business and IT or it needs to be because the business architecture and the technical architecture obviously need to really mirror each other to create that kind of flexibility and serendipity that we've been talking about and getting to that is a business, it's a business facing business value investment that you need to make. You know, you can't hide strategic architectural transformation effectively in other work. Everyone, people often try to do it when the business doesn't understand the value of it, but it rarely succeeds.

Ryan Murray:

In fact, you've talked about this a lot Zhamak like sometimes you make the world worse in the process of architectural transformation or in fact you always make it worse for a short period of time. Getting, making the world better again is the part that gets left out a lot. And so I do think that there's a lot to be said for technologists being much more conversant in the language of the business and being able to explain the importance of and explain how you will visibly see the results of architectural transformation so that it becomes a shared concern. And then, now I'm probably rabbit holing so I'll cut myself off.

Mike Mason:

Well I don't think it's entirely a rabbit hole because it is, it's a common problem that people run into when they're talking about doing software. I feel like we as technologists have not always covered ourselves in glory when it comes to asking the business to spend money on something that is, mostly a technical thing. And then being surprised when we don't particularly produce outcomes a couple of times and then the next time they're much less willing to spend that money. I think we are starting to see organizations understand a bit more that you need to do a kind of a constant investment in the technical health of your systems and architecture.

Zhamak Dehghani:

I think the language that Ryan mentioned, developers understand the business domain and design around that. So we had the, kind of the movement of domain driven design that introduced that concept in a more systematic way is how developers can collaborate with business design, the technology around the domains. I feel while we've moved a little bit forward in that space in the operational landscape but in analytical world or in areas where we are still, technology is very new, a lot of the technologies, so technologies are hyper specializing around the tool and a technology that's, the language of business gets diluted and forgotten. As an example, like in operational world we had the hyper specialization around database. You know we had the database has a layer, DBA's, they only talk about database language and then we have the business domain layer or the business process layer and the UI layer and techies were just hyper specializing around the technology.

Zhamak Dehghani:

And then the business concepts were just cross cutting across different layers and it was just so hard to change anything because the data, I think the data engineering is kind of catching up with analytics. Our world is now in the early stages that we don't have a lot of cross functional teams or cross functional individuals. We have hyper specialized people. I see the same patterns repeating there. Whenever I talk to our kind of data engineering clients on projects, they always talk about we have some sort of ingestion services and cleansing services and this kind of lead layering architecture that's existed now, it exists 90 degrees flipped in a pipeline in a data pipeline. And then when you talk about the, okay, tell me about the data. Tell me about the business concepts. The vocabulary is very poor. Like you exhausted vocabulary very quickly just because people don't think about them so much with thinking about the functional aspects of data processing pipelines. And I'm hoping that notion will kind of come to the analytical world and the data engineering world as well.

Ryan Murray:

No, it's interesting Zhamak, I think what you're pointing out is really important. Like whenever the technology problem is hard, that we focus a lot on solving the technology problem. And I would argue that databases for sure, and maybe some of those other layers were like that, right? Like data was, it's sacred. We all know that. And so we had people really specialized to make sure that it was, in the end that we were being safe and transactionally correct and all those other things. And so I wonder if we have an opportunity to recognize that and make sure that we keep the semantic importance of the data present as we evolve our technology ability to deal with the incredibly large volumes, cause I definitely see the same type of problem, but honestly, I actually don't think it's, this is a new problem.

Ryan Murray:

I've seen this with business intelligence groups for years, which is they get the data out of the transactional systems and then they remodel it for business reporting. But what's always fascinated me is how often the modeling that goes on mirrors the upstream business. Like you're reporting on the same business that is being transacted. And that may sound silly, but I can't count the number of organizations that have two independent groups that are managing the transactional systems and the analytical systems. And they're both modeling and they're modeling the same thing fundamentally, but they're modeling it in different ways because of those subtle local optimizations you talked about for their needs.

Ryan Murray:

And I've had some success in the past with really trying to unify that understanding of the business models so that the, having those teams work together so that the upstream teams model in a way that's more, not global but broader so that it can, the data that actually does transit down to the analytical systems is already in a shape that needs less restructuring so that the ownership and understanding is both shared but also kind of more single sourced if you will.

Ryan Murray:

I feel like there's, with kind of modern event streams, we're seeing another opportunity to think that way and you know, making sure that our events are true rich business events that fully reflect what's happening in the upstream, but they're also adequately able to be used by the downstream for analytical and kind of predictive purposes. There's a lot to be said and I think we're seeing some good stuff out of conflict and stuff around kind of the registry, event registries and these things are bringing, quickly bringing the adventure of an architectural world along with the kind of the API world of having good documentation and thinking about those things as first-class interfaces.

Ryan Murray:

But again, to your point, we're still very focused on emerging the tech for that. I don't know that we're having enough conversations about what does it mean to build event streams of things that are business meaningful. And that's especially challenging because some of the event streams when you're talking about like IOT devices, incredibly high velocity telemetry, sometimes that may, that isn't possible. So there's also different worlds that you optimize for. Right. And I just, I get a kick out of how complex the world is becoming and there really are no principles that are kind of universally valuable anymore or they're becoming fewer.

Zhamak Dehghani:

Yeah, I completely agree. Like the, even in the data world and even stream world, the services closer to the consumption are easier to model around the kind of business concepts and domain concepts, the sources, the data that the services closer to ingestion, whether they are devices, there are now mobile applications or sending a lot of metrics about all the different types of applications. So all the different types of interactions and because of the volume and steel technology catching up with the scale of the data that it needs to deal with. The opportunistically mix and match domains to kind of enrich the data or cleans data. So the boundaries that are kind of more clear on the consumption side as in the business consumption side becomes leakier on the source ingestion side because of the volume of data that you, when you get it opportunistically, you want to do all those big joins across different data sets and do enrichment and that has its own problems and challenges.

Ryan Murray:

Yeah. One of the things that I've interested to experiment more with is having more well-defined ownership of the enriched data because more oftentimes that data's being enriched because fundamentally, again, there's a business concept that's either somehow the upstream systems are fractured in a way that maybe isn't actually even ideal or they're simply macro observations that you can make across kind of core domain entities, core capabilities. But either way it feels like a lot of what we talk about with the upstream systems around longstanding ownership of a business capability. I feel like longstanding ownership of aggregate data sets is also really important and I've seen a couple of organizations starting to do this well of really defining clear business data ownership and stewardship so that we don't get as much of the bending of things to the technical need, but much more keeping that, seeing that aggregate as itself a genuine business concept.

Ryan Murray:

I think there's a lot there that is, when you have, because the technology is so challenging, you have like that tendency to have the data group and with the data experts who kind of end up owning the data lake. And it seems very clear to me that some of the larger organizations I've worked with, they're doing this well, have gotten it right when they make sure that there's still business ownership of every aggregate concept in the lake and that it's not a technical enterprise, but it's a fundamentally business focus enterprise.

Ryan Murray:

And I think a lot of the things we talk about in our digital platform strategy team is really about technology capabilities and how they support business capabilities. And so keeping that separation and saying how you move data at scale, whether, you know with the streaming technology or how you think about data lakes and data lake design, our technology enablers to something else, which is fundamentally a business concept, right? And if it's not a transactional capability but analytical one, that's fine, it still needs to be driven from that business lens. And so setting up teams with product ownership that's genuinely business focused is something that feels like it's bringing a lot of people success and keeping them out of that data swamp. But it's early days.

Zhamak Dehghani:

It could now go down the rabbit hole of a product ownership for data.

Ryan Murray:

Well you know, another interesting thing that I've been noticing in a couple of our clients recently is because like with our platform work, we've kind of standardized enough things that we're able to get in and start moving technology like this architecture transformation and some of the delivery infrastructure transformation faster than we have in years past. And what that's done is like technology has kind of quickly unlocked capability or been on the verge of unlocking capability and business owners are just not ready to move at that speed. And I know we've all seen this, but now it's starting to become this repetitive pattern where the business actually needs to now learn to interact with the technology organization in a much quicker and much more governmental way than they're used to.

Ryan Murray:

And that's been interesting as like, again, looking at some of the larger digital organizations whose product owners fundamentally recognize that they're wielding technology to create the business vision and they either have strong partnerships with technologists to do that or they themselves have enough background. And there are a lot of more traditional organizations, they, it seems like the crisis of business leaders not being tech savvy is becoming very acute.

Mike Mason:

Well, I'd like to thank both of our guests today, Ryan and Zhamak. Thank you both for being on the podcast. This has been very interesting to me and I hope everybody listening enjoyed it. If you did, please leave us a review cause that helps us expand the audience and have other people find the podcast. Thanks very much. We'll see you on the next one.