Beyond the AI facade

[00:00:00] Shayan Mohanty: Welcome to "Pragmatism in Practice," the Thoughtworks podcast where we explore practical approaches to becoming a modern digital business. I'm Shayan, Thoughtworks' Chief Data and AI Officer, and I'm joined by Joe.
 

[00:00:13] Joe Murray: Thanks, Shayan. I'm leader of our service line that we call Customer Experience and Product Innovation.

[00:00:18] Shayan: In today's episode, we're going to be talking quite a lot about C-suite AI anxiety, some of the operational realities behind bringing AI into the fold, things like tokenomics. We'll also be talking about why product-led engineering is suddenly beating consulting decks and what's leading to that change. Joe, I'm really keen to hear from you about some of the client signals that we're currently seeing from the front line.

[00:00:44] Joe: Absolutely. I think we're seeing a lot of sensitivity with consultants showing up, having everything being AI. There's a lot of AI lingo getting dropped right and left, which is creating quite a bit of skepticism among our clients for consultants that show up talking about AI as the centerpiece versus business results as the centerpiece.

At the core of product thinking is focusing on outcomes that you want to achieve by providing a feature or capability to a target audience in a way that that audience engages with that capability in a manner that generates a measurable change in outcomes. That could be employee experience and our employees engaging with internal systems to net the productivity gains that you're expecting. Or it could be engaging with customers in the sense that customers are engaging with products in a way that drive conversion rate, higher profit margins, longer lifetime values.

All of those metrics still matter. The consulting advice and solutions become valuable is when you can connect the dots between those business outcomes and how those get enabled by AI-specific technologies differently than they've been enabled in the past.

[00:01:56] Shayan: An area that I find really interesting is aligning to business value. I think business value is really a hard thing to pin down. In our world, for instance, we do a lot of data platform modernization. Frequently, there are just so many different stakeholders that get affected by a data platform that just nailing one type of business value becomes impossible. You really do have to blend it all together.

I'm curious, from your perspective, how do you align to that? What are you seeing work? What's not working? How do we wrap our arms around that particular problem?

[00:02:31] Joe: Business value is a really big topic, and it depends on the kind of business you are. At the end of the day, it's outcomes that matter, really. We've done a lot of work with the VA, for instance, and so a really meaningful outcome for the VA is decreasing suicide rate among veterans. That's not a monetary value, that's not a saving money value, that's a save the world value.

You can understand causality between how difficult it is for veterans to engage with the services they need and the suicide rate that ensues as a result of that. By lowering the difficulty of veterans being able to find, discover, and engage with the services that they're entitled to has a direct causal relationship to a decrease in the suicide rate. That's a powerful way to think about outcomes.

Then, of course, I've already mentioned a few of the classic business outcomes, things like reducing cost, increasing sales, profitability. Because you have world-class customer-focused organizations like USAA, for instance, they actually command higher premiums than your more budget insurance companies, yet they have the most loyal customers on the planet because of the high degree of service that they deliver.

There's a direct relationship between building better products that engage with true needs humans have and the profit that you can command as a result of doing that.

Lastly, from a data perspective, we see really complex problems like geoscientists trying to digest terabytes or petabytes of information to decide where to dig for that next gold mine or where to set up the next natural gas well. There's a really interesting and powerful application of using AI to help separate the signal from the noise and help those types of users really see the meaningful and actionable data that help them make better business decisions. Those better business decisions result in meaningful outcomes like a higher hit rate on mines that actually become profitable versus those that don't.

[00:04:34] Shayan: That makes good sense. I think another thing that's top of mind, at least for me, is how do you bridge between aligning to business outcomes and this idea of AI-fying everything? To be more specific, if we're not talking to the right people at the right level, or folks who aren't thinking in the right way, there's this tendency to just throw AI at things, and the assumed impact is going to be some efficiency gain. Just by replacing humans or by doing something less interesting.

To your point, the right way to think about this is a holistic transformation, thinking about things in terms of outcomes and outcomes that we'd like to drive and re-deriving the right implementation from that. I guess in practice, do you see these two things as being in tension with one another, or actually is everyone very much aligned to the idea of aligning to business outcomes, and there isn't really that other thing of let's just throw AI at everything?

[00:05:30] Joe: I think there are a lot of dynamics in play. One is we're very clearly in the middle of a hype cycle. We've seen it many, many times throughout history. I've been doing this for 30 years. I started my career when people were understanding what is this thing called a website, and why should we have one? There was this proliferation of businesses throwing up websites because everybody has one. There was a proliferation of really bad websites that actually hurt people's brand more than help.

That just repeated itself when mobile technology came on the scene. It repeated itself again when machine learning came on the scene. You can see probably half a dozen key pivot points in technology over the years where something new comes on, everybody rushes to do it, and half the people just do it because everybody's doing it, and they end up not getting the results that they thought.

I believe the hype cycle's a very real thing. We're seeing that with AI today, where people are trying to deploy AI because they want to be seen in the market as an AI-modern company, but they really don't understand the connection between how AI can actually deliver value. That's one. The hype cycle, I think, is one reason.

Another reason, and it's kind of attached to hype cycle, is because companies that are powered by AI are actually seen as more valuable. We see board pressure. Our partners Teneo, mentioned to us quite recently that they have hundreds of mid-cap companies that are coming to them for advice on their AI strategy because their board is coming down hard on them for not having one.

You see executives just reacting to a demand from the board without really thinking about why they're doing this, or why are companies that are powered by AI more valuable, and what does AI mean for my industry and my company in making me more valuable. I think those are a lot of the dynamics, and there's probably many others, but I think those are the two big ones.

[00:07:18] Shayan: I think you just touched on something that's interesting, which is there's this very clear C-suite/board pressure that everyone at every level is contending with, which is we need to be doing AI. What is our AI strategy? How do we get more AI into everything? What I'm curious of is earlier we were almost talking at the level of a particular implementation. It's much easier to articulate the value a use case is able to bring, the business value that is captured, and therefore we can derive AI's impact in it.

It's much harder to look at an entire business as a whole and say whether the AI transformation program, whether a full-on AI strategy, is inherently valuable or not. How do we go from a single project to something that is essentially a program? It's a transformation program. How do we still thread that business value through from one thing to another?

[00:08:16] Joe: There's a whole bunch of dimensions there. Where we can start with that is understanding that AI is a technology that allows you to think about different tasks and operations in your business across the board in different ways. Not just the way you engage your marketing, your customers, like I've been emphasizing from a product perspective, but how are your employees more productive across all the business functions, from HR to finance to operations, et cetera.

Thinking about AI as a platform or part of your business technology platform that's enabling the business is a really valid and powerful way to think about it. You've seen some things as well around companies working with you on "how do I actually become an AI-powered enterprise"? You've seen some good patterns and some not-so-good patterns. Maybe you could expand on that.

[00:09:08] Shayan: I think the patterns that I've seen are relatively straightforward in that, number one, you really do have to work backwards. There needs to be someone at the client who has at least a reasonable end state in mind. It doesn't have to be super high resolution or anything like that. They have to have some degree of ambition to be able to imagine what becomes true about their organization because of the advent of AI.

Frequently, that's either they're able to do more with the same capacity, meaning efficiency gain, some combination of that, with there were some things that used to be in the too-hard bucket that are now no longer in the too-hard bucket. With those two things combined, what becomes true about your business?

In order to navigate that appropriately, you have to have a stakeholder that deeply understands the business and also has a good enough grasp of the technology to understand the art of the possible. They don't need to be technical experts or anything like that. They just need to know what the direction of travel actually is. Obviously, we can help with that, but it requires someone with ambition, and that's not always something that you get for free. I think that's like level setting.

[00:10:17] Joe: We talk about the courageous executive all the time, and we've always talked about that. Something that goes with innovation and has always come with innovation is courage to explore, and experiment, and validate these hypotheses that you're talking about. When you're envisioning what must be true, that starts as a hypothesis. You need to explore that and be willing to be wrong, but in being wrong, you learn what's right.

[00:10:41] Shayan: That's exactly right. I think there's another piece of this, which is what we frequently see is an organization will start with, "Let's just give everyone an enterprise Claude license or ChatGPT license or something like that." They open up access to the API to everyone. That's a good place to start, but that's not an AI transformation program; that's not an AI strategy.

That is scratching the very tip of the iceberg in getting people familiar with tools. AI is a lot more than just a tool. AI is a full-on technology and a capability and something that you can really wrap your arms around and integrate in various clever ways. The point is that frequently, if we don't have that courageous executive, someone with vision and ambition and a reasonable finger on the pulse, both in terms of business and the technology, then frequently that's where an AI strategy starts and ends.

We have to come in and say, "Well, actually, we're happy to help you with a bunch of these other things, like getting your estate AI ready, if you will, or getting a governance platform in place. That's not helpful if the organization's not in a place to metabolize it. Then it's just technology for technology's sake, and the organization is not in a place to be able to actually use it.

I think part of this is very much leaning into those courageous executives. The people who have that vision, have an understanding, and who can guide the organization from the very, very early innings of "Let's just get everyone exposed to tooling" to later parts of the game where it's like, "Now we actually have to figure out how to connect. Like we said earlier, business value to this core capability. How precisely are we mapping that and the organization?" That is a non-trivial problem. The moment you expand it past the single-use case.

[00:12:28] Joe: 100%. I'm curious what your thoughts are. When we talk about having a courageous executive that has a vision of the art of the possible, I think we're in an unprecedented moment where what's possible is changing at an alarming rate. Think about in the last six months. We've gone from the beginnings of models that are broadly available that can support discrete AI applications to now autonomous agents that can work together as agentic workflows.

It seems like just when you wrap your head around one of these concepts, the next concept comes. I'm curious what your thoughts are on the velocity of change and how we need to rethink the size and pace of the iterations that we're pursuing in order to be able to keep up with the pace of change?

[00:13:16] Shayan: Having smaller, tighter loops that we can execute on faster allows us to stay sensitive to winds of change. Being able to apply AI practices to accelerate delivery, yes, but also keep teams smaller, faster, nimbler. Be able to leverage the latest and greatest in the technology to bring operational efficiencies where we can. That's the only way.

At the moment, what we're seeing is, to your point, capabilities are shifting every single day. Part of that is understanding the technology, but part of that is also just understanding the direction of travel. Being able to anticipate, even if you don't know exactly when it's going to happen. Being able to have said last year, even, we are inevitably going to end up in a world, sooner rather than later, where we have a whole bunch of agents that are running around an organization doing stuff.

We didn't have to have evidence of a well-designed agent that can do a job end-to-end before we could make that prediction. We could see from the direction of travel where we were headed as an industry. The industry came together and said, "This is probably going to be a thing," and lo and behold, to your point, we're here now. We have agents, and those agents are actually useful.

I really do think it's just about having a finger on the pulse, being sensitive to the winds of change, and being able ot work in a framework where you're able to iterate extremely quickly and your'e not bounded by multi-year long programs but instead, you're able to chunck it up into smaller pieces so that as the world shifts around us, we can incorporate those new changes into our workflows.

[00:14:50] Joe: 100%, and that's a great segue into the other big question that's emerging. Kind of goes back to, I think you've mentioned in the introduction, tokenomics is something everybody's concerned about now. I know you've said that becoming an agentic enterprise is an operational decision, not a technology decision. With that, new ways of governance and also understanding your costs, because again, when we think about outcomes, we think about ROI.

It's very easy to look at a dramatic increase in revenues, for instance, based on an agentic workflow, but not realize you're costing yourself double and because you've got agents gone wild, driving token costs out the roof. Maybe you can expand on a little bit about what does that op model look like? What should organizations be thinking about as governance best practices as we start to deploy these autonomous agents?

[00:15:43] Shayan: It's a little bit of like a working-backwards type of thing, but we'll try and attack this problem from both ends. Imagine you're an organization that is just starting out on their AI transformation journey. A whole bunch of people have access to Claude and ChatGPT and insert tools here. The very next thing that you're going to want to do, make those tools accretive to your business.

The only real way you can do that is by making your business, your estate legible to those tools. You want ChatGPT to be able to reason about your data, your operations, contextualized to your business. I'm hand-waving a lot of not just technical complexities, but some legal complexities as well about the BAAs and stuff that you have to sign, but assume that all of that operational stuff is done.

The first thing you really should be focused on is, "I need to make the first big step in my organization and operationalizing this then means that my estate needs to be AI-ready." That means AI-ready data. That means AI-ready services, your CRM, your ERP, these various systems that you use to run your business now need to be legible and consumable by AI in some way.

If you then assume that is true and you've made that change, you've implemented that, the very next thing that you're going to start asking about is, "How do I make sure that the right people with the right agents are accessing only the right data at the right time?" You're going to put a whole bunch of controls around the thing, which then implies that you have an entire governance apparatus around this.

Your preexisting data governance implementation might only cover your data platform, but these agents are oftentimes stitching together capabilities across not only data but systems and services and so on. You could talk about your data that's sitting in Databricks or Snowflake, but you also have to think about the agent's access to things like your ERP and CRM and all these other services that you're using.

Now, you have to think about lateral access, and it's not just data anymore. It's everything. Now, you have to have this overlay over everything, and that's just around compliance and governance. Assume you have that, that implies that now at this point, you have a number of agents that are running perpetually. Running various parts of your business. You've got a number of employees that are very agentically enabled. They're running a bunch of their own agents and frequently spinning up many sub-agents and doing all sorts of things.

At some point, you transformed from a business where your primary cost was, let's say, people to a business where you're now saying that you have two primary costs, you have payroll and you have tokens. This is interesting because if you've made it this far, then what you're also saying is that you still have more work to do. There's still going to be more agents.

It's not like you're going to suddenly stop building agents or agent's are going to be just steady state. You're going to keep consuming tokens. And because you're being metered on tokens, if you're going to Anthropic or OpenAI or any of these other folks, they are charging you at some level per token. Which means that as the total number of tokens you're consuming increases, the total amount of money you're being charged increases, which means that your costs are only ever going up and to the right.

The punchline here is that some point you are going to start caring, not just about the unit cost per token, but you're going to want to change the relationship between cost and tokens as much as you possibly can. The only way to do that at that level of scale is to start owning more of the token generation supply chain.

You're going to want to start renting GPUs, running open-weight models, perhaps even tuning some of those models to be a little bit more specific to your business. At this point, if you've done all of these things, then you now have the DNA of being an AI-first company. At that point, you can keep scaling your AI consumption.

The costs associated with that are not linearly related to the token consumption. That is what we're seeing in terms of patterns, and it's this very clear spectrum of maturity. The cool thing is that every company is at a different stage. It's not like everyone is starting at exactly the same place.

We work with AI natives that are obviously very far along this curve. We also work with enterprises that are just starting out. Now that we've seen this movie a couple of times, we can offer very clear, direct help no matter where someone happens to be on that maturity curve. That's awesome because then we get to basically replay the same movie over and over and over again.

[00:20:30] Joe: A big thing you introduced is if you're truly an AI-first enterprise, you need to have that control plane that enables the governance and observability of these autonomous agents, as well as the AI that your employees are using. Can you talk a little bit about how we try to solve that with AgentWorks?

[00:20:49] Shayan: Yes. Like I said before, we've seen this movie a couple of times, and for those who are listening who might not be familiar with AgentWorks, AgentWorks is actually a platform that we built to address this whole governance, and compliance, and token economics question. It's a fully governed agentic runtime that runs in any environment across clouds, on-prem. Wherever you need it to run, it will run.

The reason why we built this is because we've now seen this movie a bunch of times. Along the way, we recognized as people were crossing that threshold from my estate is AI ready to some extent, and now I need to gate access to certain things. We realize we're building the same apparatus over and over and over again.

What I wanted to do is make sure that we're focusing more of our time into the earlier part of our conversation, focusing on business value for our clients. Ultimately, yes, making sure the agent accesses the right thing at the right time is part of driving business value, but really the business value is held in that use case where we're driving some operational efficiency, making them more money, costing them less, whatever it ends up being, that's the business value that we need to be oriented around.

Every moment that we're spending not working on that problem, frankly, is a moment wasted, especially if we are building stuff that we've already built before. It's not a good use of anyone's time. The whole point here is we spent the time and energy to build this platform so that we could just walk into a client and be like, "Hey, we can take care of 90% of the engineering plumbing we would ordinarily have to build in order to give you that safety and those levels of guarantees. We just have this platform. We'll just drop it in so that we can focus all of our time and energy in building the agents that will drive business value for you."

The mechanisms for compliance and governance and all this other stuff. I'll throw a bunch of buzzwords out there, but there's a lot of policy-is-code in the platform. Being able to define arbitrarily complex policies around what an agent can and cannot do. All of this is governed deterministically. The systems that govern the agents are themselves non-agentic on purpose because you need to be able to trust the apparatus itself. Otherwise, where does trust start and end in a system where an AI is able to judge itself?

We made a bunch of decisions along the way. We had to contend with what does agentic identity mean. How do we think about monotonically decreasing scope? How do we think about audits? How do we think about caching? How do we think about all of these things? We had to go through this process because we've again seen this movie many times, and we wanted to just preemptively build this thing so we didn't have to waste time in the future.

[00:23:37] Joe: I'm going to pick up on that and riff a little bit about going back to our business value capture and how product thinking helps insulate your technology initiatives in a matter that allows you to enjoy successful outcomes more often than not. That's why we came up with our 3-3-3 approach, which is about a gated way to first create that hypothesis that you iterate on that starts with a very rapid concept that you explore and initially validate that you have a reasonable hypothesis in as fast as three days.

Nominating that, the hypothesis at the end of that concept exploration is good enough to take it to a proof of concept. Then your proof of concept in the next three weeks or so is what helps you navigate the four product risks, which we talk about all the time, which have always been true and remain true.

There's a risk of is it desirable by your target audience? Is it usable? Is it viable financially? Is it feasible technically? If you go about that POC phase successfully, you've validated your hypothesis against those four risks, and you can move into deploying your first production MVP with some confidence. You're going to get the outcomes that you expect because that's what those four product risks are meant to help you navigate. If you can navigate those risks, your odds of delivering meaningful outcomes is actually quite high. The interesting thing around how AgentWorks and 3/3/3 work together is you can actually start incrementally exploring new agentic solutions and validating those over time. You don't have to boil the ocean like you were saying before. You're literally working in one-quarter cycles to evaluate and roll out new capabilities.

[00:25:20] Shayan: I think that the speed between iterations is really the key at the moment. Like we talked about earlier, staying sensitive to the winds of change is really only possible if you're working in really tight timelines with a small, focus team. That's what is expected these days, which means that average team sizes are shifting. The average timeline for engagements is shifting. I think 3/3/3 is very clearly our most successful delivery approach at the moment. It comes back to just being able to move at pace, not only with our clients but also with the rest of the world.
 

This has been great, man. We talked about AgentWorks. We talked about 3/3/3. We talked about what it means to rotate an organization towards being AI-first. The shift from AI hype to real business value clearly comes down to things like intentional design, cost, observability, governance. Just throwing Claude and OpenAI like ChatGPT at everything won't build your competitive moat, but product-led engineering will.
 

[00:26:27] Joe: If you're willing and ready to bring structure to your agentic workflows, you can head over to Thoughtworks.com to explore our 3/3/3 methodology and see how Agentworks helps leaders scale AI sustainably. Don't miss our upcoming episode with Teneo, where we'll unpack the financial reality of tokenomics and enterprise scale. Shayan, this has been amazing as usual. Love always chatting with you about this stuff. Thanks everybody, for joining us, and we'll see you next time.
 

[00:26:57] Shayan: Thanks so much.