Themes in Technology Radar Vol.35

Lilly Ryan: Hello, and welcome to the Thoughtworks Technology Podcast. I'm your host for this episode, Lilly Ryan. Today, we are going to be talking about the Technology Radar Volume 35. Joining me to discuss this are Doppler members Chris Ford and Effy Elden. Chris is a mendicant Thoughtworker originally from Australia and currently residing in Barcelona. Folks in the functional programming community might know him for music-as-code. He's now Thoughtworks' chief architect.

Effy joined Thoughtworks as a consultant specializing in "DevOps and duct tape" before ending up in the Global Information Security team. She lives in Melbourne with her greyhound, Noodle, as well as a lot of tech junk that she swears she will do something with one day. Maybe today's that day, I don't know. Welcome, Effy. Welcome, Chris. How are you both?

Chris Ford: Yes, really good, Lilly. Nice to be here today.

Effy Elden: Yes, nice to be here today.

Lilly: We're talking about the Tech Radar. Volume 35 is going to be out pretty much, I think, when people are hearing this. As the both of you were in the room when this was being discussed and debated, I wanted to get into some of the detail about what's in this edition in terms of the themes, and what themes are also in relation to the Tech Radar, because that's a whole thing in itself. To get us started, though, Chris, would you mind taking us a bit through how the Tech Radar is put together and what that looks like?

Chris: Sure. You could see the Tech Radar is a direct democracy of technology within Thoughtworks. We have more than 10,000 people globally who are working on different projects with different clients, seeing different things. The way we run the Tech Radar is that we put out a call for all those people to nominate what we call blips or items for the Tech Radar that they've been observing and that are of interest.

Then what we do is we take all that information, and different members of the editorial committee, which we call Doppler, are responsible for gathering and curating and getting details and following up to get information for the blips. Then the Doppler all meet in one physical location over a week and hammer out what we think the 100 or so most relevant blips are. The Tech Radar serves as a sensing mechanism for Thoughtworks originally, and now we share it with everyone else.

Lilly: Effy, what's the difference between a blip and a theme?

Effy: The blips are very small, discrete items. They're submitted by the Thoughtworkers who are on the ground working for our clients across the world, and they represent a specific tool, technique, platform, language, or framework. On the other hand, the themes are a bit more emerging over the course of that week when we are looking at all these proposed blips.

Often we notice trends, we notice groupings of things that belong together, or specific motivations behind a lot of the new tooling that we're seeing. Part of what we do during that week together is note down when a potential theme appears to be emerging. Then at the end, we look through and look at what's the most prominent theme, what we think is interesting to talk about, what best encapsulates that vibe that we felt over that week.

Lilly: That vibe is-- I want to talk a bit about the vibe as well, because one of the things that we get the benefit of when we have folks from Doppler here is about like what it was like in the room and how it went discussing that stuff over several days, because I understand that the opinions can get pretty heated.

Chris: I don't think we're as argumentative as all that, but there's certainly a lot to discuss, because certainly with Tech, the way it is now, it's very common for me to hear some other wise and experienced person at Doppler nominate a tool to solve a problem that I don't know I have yet. There's certainly a lot of rapid adjustment of mental models that I get out of being part of Doppler in response to collaborating with all my colleagues. Very much a mind-opening experience because that room represents, in aggregate, the sensing mechanism of all of Thoughtworks. It's a pretty powerful fire hose.

Effy: Yes, as Chris said, I wouldn't say there's much in terms of arguments, but certainly some debate sometime is about what we think we can actually encapsulate into a blip and what feels a bit too complicated to blip, as we put it. Obviously, it can be easy to think something is simple and can be put into a paragraph until you try and write it, which is why sometimes things initially get up and later don't quite make the cut.

I think certainly with the themes, no one in that room would really disagree with what we've ended up with for the themes here because they were so prominent and so obvious, I think, at the end of the week exactly where those clusterings were, where those vibes were taking us.

Lilly: It's probably not a surprise to anyone listening either that all of the themes that we have here are related in some way to LLMs and working with LLMs and how that works. In fact, other types of models and whatever we can be doing with them in practice. It's been interesting to watch this evolve over time. I am really interested to see how you folks have perceived that as well. The first thing that we've got is putting more agents to work and talking about evolving over time. This theme follows on from the Volume 34's theme of putting coding agents on a leash. In this case, it's putting more agents to work. What's more agents in this case?

Chris: I think really what we're seeing here is the impact of an industry trying something out and then encountering new problems and all at the same time. One of the most obvious and notable cross-cutting concerns of the blips was that there were a lot of them that were about having some surface where a developer can be running more than one coding agent at the same time.

What I guess originally you might have is a CLI or an IDE where you've typed in some instruction and the agent's going away doing stuff in response to your harness and what you told it. Once you've been doing that for a while, I think everyone who's got that experience finds that the wait time in between instructions and results, and also the potential of getting more stuff done, draws you into running more sessions at once.

I think there's a lot of people who started out by just having multiple command line tabs who are looking for user interface that makes it a little bit easier to know when a particular agent needs their attention to get to the next stage or to make it a little bit easier to manage the different workspaces or Git subtrees. That's what we're seeing. We got a lot of blips that were around Claude Squad, T3 Code, Orca, and actually many others that I can't even name now. They were all solving this one problem that we think a lot of people are hitting at exactly the same time.

Effy: I think it's also about the evolution of the tooling and, in particular, the models. How much they're capable of without a human directly providing oversight and feedback turn by turn? A few years ago it was, I'm in my IDE, I have an agent as an assistant on the side, and I'm prompting it to do a small bit of code. I look at that, or maybe I'm typing, and I'm using tab autocomplete to fill out these chunks.

As the models get more capable, it's now like I can just throw a card to it and come back and the card's done. With less need to have a human intimately involved as the code is being written and tested and that feedback loop is progressing, we see humans go, "I've got Claude on my job. What am I doing now?" The answer tends to be, "What if I've got another Claude on another job?"

Whether that's part of the same repo, splitting off into multiple tasks, using things like Git worktrees so you can have each agent in an isolated space, or whether that's, "Oh, there's three different projects I'm working on, I've got a Claude agent on each of those. Now all I'm having to do is wait for one of them to pop up and need something from me."

Lilly: How should people be approaching this? Because it seems to me that there are a lot of different ways that this could go. There are a lot of things that people could try. Maybe the question of should is, it is definitely opinionated. Is there a better way than other ways to do this? My experience with this has been that if I try and do that, my own attention gets pretty split across things.

I will often start something off and then totally forget that I had that thing going or that it was waiting for me, or I'll be on some completely different part of my work in some other place and come back and realize, "Oh, actually, that's finished a task I don't even need anymore."

Chris: That's quite fair enough to say be careful of mistaking activity for progress. We know that using coding agents can really suck people in. It's a really fascinating way to create software now. I don't think there's anything wrong necessarily with doing multiple things at once, but I would say that Effy's point about agents being capable of working on tasks for longer is probably the part of it you should lean into.

If you're able to improve your harness or use a model or create a feedback loop so that you can kick an agent off and need to intervene less often, that's probably superior to being able to spin one or two more plates at the same time and probably also better for your cognitive load and well-being.

Effy: I think it also comes down to following good engineering practices around properly scoping, shaping, and sizing work. Having a well-defined task that you can hand to an agent that's going to be able to do through to completion without lots of unknowns that it's going to have to guess or come back to you with, without it being open-ended enough that you might find the agent still going hours later.

By breaking down work properly, scoping it properly, having clear acceptance criteria, proper automated feedback loops so the agent can actually know when it's done and when it has met the requirements, that helps you be able to be a bit more hands-off. I think the other part of it is that kind of ability to know what needs your attention, where things are at, that observability and visibility into what's going on with your agents.

I think that's where we're starting to see this new generation of tooling, which is like meta-harnesses, meta-orchestrators, operating one level above things like OpenCode, Claude Code, Codex, that provide interfaces where, for example, an agent that needs your attention is going to pop up to the top as a waiting agent as opposed to an agent that's working, which will be lower down, interfaces that at a glance, can tell you, "Here's all your agents in their current status," which ones are waiting on approval for a command, which ones are finished, which ones are working.

That gives you a bit more visibility compared to a challenge I know I faced early on, or still face to this day, which is I've got 20 different terminal windows open and I'm having to mouse through to check, "Is that agent done? Is that agent done?" and an agent is sitting there for two hours waiting for me to approve a copy command so it can continue.

Lilly: One thing that worries me a lot when I'm working in this kind of sense, and I assume that, in particular, Effy, it also worries you too, given that we're in a similar line of work, is how to know when something has gone wrong. There's certainly like when an agent is asking for your attention because it has hit a point where it has to for whatever reason, because you've put a hard rule in place or because there's some kind of point at which it has stopped, but that doesn't always mean that that's the right point for it to happen or that something hasn't broken earlier on.

How does one set something up so that you can run a situation like this confidently with multiple things long running in parallel, even get up and walk away from it, maybe, whatever you need to do, and feel okay about it, not breaking stuff?

Effy: I think it's a good question. I think there's a few different approaches that are somewhat rapidly evolving at the moment. This time around on the Radar, we've blipped agent sandboxing in Adopt because it's just something that you need to be doing, whether that's coding agents running locally on your laptop or business agents running in the cloud. It's the capabilities of models to do great work, but also great harm if they go off on the wrong track, if they get manipulated or prompt-injected, or even just fundamentally misunderstand something, is also growing. We need actual technical constraints in place so that a model that does go off on the wrong path can't do too much harm. I think sandboxing is a big component of that.

I also think that what we're seeing from the frontier labs around built-in safeguards like Claude Code's auto mode and Codex's approve-for-me is another piece of the puzzle there, because it means you've at least got a little bit of safety. It's not something you should rely on primarily, but at least means you can be a little more confident walking away for a few minutes and you won't come back to the entire root directory of your hard drive being deleted, which I saw someone on Twitter complaining about recently.

Lilly: Chris, Effy mentioned earlier tooling that would help a lot with observability, monitoring, and so on, but when it's operating at this kind of capacity with lots of agents spinning up their own sub-agents running for a really long time and going in parallel, how does observability become meaningful in that situation? How do you actually get real oversight of that stuff working and not just become kind of like human-in-the-loop, see it as some kind of meat proxy really for whatever else is going on, and just push button, let's go?

Chris: One of the principles I would suggest people follow is that if they know stuff that the agent doesn't, then that will probably explain a lot of the times why the agent does things that they think shouldn't happen. For example, one basic thing might be that if your agent can't tell whether the build is red or green, then it's not reasonable to expect that the agent, when it's going away and coding on a big epic or something like that, that it could actually respect build discipline.

You mentioned observability, that would be another example of where a good developer should be able to keep an eye on production to know the impact of their changes, and hopefully if you're developing in a way that Thoughtworks would recommend your code is going to production very often, so that production performance of the application should be meaningful feedback in what you do next.

One of the techniques that we would suggest is that, where developers get information to improve their work, also expose that to agents. Both the status of the build and production, if an agent can be keeping an eye on that while it is iterating and while it is running the tests or whatever other harnesses you have, then it's more likely that it will be kept under control because it will be getting feedback that tell it if it's out of control.

Effy: I also think part of keeping an effective human-in-the-loop is not allowing human-in-the-loop fatigue. As a security professional, for a long time, when I was running my agents, so I had it strict. Every time it tries to run a command, I have to approve that, but obviously, when they're running hundreds of commands every minute, that's just not feasible.

When I'm spending all my time reviewing long lines of Bash and Python, I quickly ended up getting into this headspace of skim it for a second, hit enter, not really understanding what that code is or what it does. I think that's, again, when using things like sandboxing, least privilege, ensuring that the worst-case outcome is not that bad, suddenly, you don't need to have a human-in-the-loop there because you've actually got hard controls rather than relying on human judgment constantly to prevent something going wrong. I think human-in-the-loop where a human is needed, not necessarily in all the possible places you could insert one.

Lilly: How do you know when a human is needed?

Chris: I think one of the trends that we're seeing is that the way that we assure quality is becoming more diverse, and actually, as a developer, what I'm quite excited about, it's allowing us to try techniques that were perhaps a bit more considered avant-garde or impractical to implement before. I know on this Radar we have, for example, TLA+, which is a venerable model checking tool on the Radar. We also have things like mutation testing.

I think that in answer to your question about when a human is needed, I would say the answer is whenever there isn't an effective materialization of a check that the agent can use to check its own homework. I would expect that most of the time, the human activity is higher leverage if reinvested in improving the application space harness, and less in inspecting individual pieces of code that go to production. That latter one is, as Effy hinted, quite exhausting, and after you've inspected one diff, that doesn't put you in any better position the next diff that comes along, whereas if you have implemented a new testing or some other review process, then it's a virtuous cycle.

Effy: The other important element here is just about accountability. An agent fundamentally cannot be held responsible, cannot be held accountable for things, and so there are cases where you need a human in the loop because you need a human to be accountable for the consequences of a decision, for the consequences of a change. Exactly how often you need that and when you need that is going to depend a lot on your context. For example, the regulatory burdens you're under, the sensitivity of the systems and data involved, but fundamentally we should be having a human hitting the button whenever there's a need for accountability because we can make agents very smart, but we cannot make them accountable.

Chris: To build on that, Effy, though, I think you already mentioned this, but human-in-the-loop theatre is a problem that we're seeing. When you do have that need for human accountability, that human has to have the opportunity to absorb enough context to meaningfully make a decision. If the human is just there to press yes, then it's not genuinely human feedback, it's just a opportunity to shift blame onto that poor person who approved the unsafe change.

Lilly: Something that you said earlier about making sure that you have well-defined tasks and you're splitting things up well, I think also comes back into play here, because you can, and there are certainly different libraries of skills and other kinds of tooling that you can use that will help you to explode your ideas out, work out what you want, figure out what the shape of certain things should be, getting down into tiny details, and set some direction.

There are also agents that will happily go off and write their own down to an infinite level of granularity as well. If you're allowing, I think too much control to go to that in terms of how much decision-making gets taken away from you in that sense of what you're trying to define, I think that too is also an important place for there to be a person who's actually setting some direction and making sure that that direction is still matching.

Chris: I think there are some developers who don't really like to engage in analysis or value, and so it can be really tempting, if that's the case for you, that you say, "Well, here's a bunch of documents, I don't really want to worry about what the backlog looks like or what the story breakdown looks like. I'll just chuck it into the agent. The agent will digest it, and then off we go."

I would definitely encourage people to not give in to that temptation, because, to your point, Lilly, I think a human being understanding what a meaningful slice of value looks like is still really important, and if you can organize your work that way, you're going to be having much more meaningful review points that you, as a human, can give feedback on and that maybe you can demonstrate to your colleagues or to your users. That discipline of working out how to turn a bunch of stuff into a set of valuable chunks is not obsolete at all. In fact, maybe it's more important than it's been in a while.

Effy: I think also something we've cautioned against in this Radar is surrendering too much of your own cognitive decision-making to agents, particularly around heavyweight frameworks, for methodologies for software development with agents. I think it's very important that you make the workflows that your agent is doing work for you rather than the other way around. You shouldn't be adapting how you do development to match an opinionated framework that is not built for your context, for your team, for the way you're working.

It's much better to pick up things that are a bit more lightweight and actually evolve a workflow that makes sense for you and your team and the space you're working in. That way you actually have full knowledge of exactly what those steps are, what those feedback loops are, what those decision points are, rather than importing thousands of lines of markdown that tell your agents to do things that you probably haven't even read.

Chris: I think that's consistent with what our advice has been for a long time in frameworks in general. I think sometimes it can be tempting to use a really heavyweight web framework without feeling like you have to understand how the pieces work. Our advice has been in the long term, understanding the constituent parts, which is probably easier if you pick a simpler option, is likely to make it easier for you to maintain the quality of your output over time.

Lilly: Speaking of heavyweights, I wanted to move us on to the next theme that we've come out of this edition with, which is Beyond Frontier Models, I think we're calling it. This is the theme that came out of a whole lot of blips that, as I understand it, were pulled together because folks were seeing a lot of people falling into the trap of thinking I must use Fable for everything because it is the most powerful and the most important and the most cutting-edge, and therefore it will be the best tool for every job. This theme was something that you folks put together because that is a real issue in quite a few different ways. Chris, could you kick us off in talking about where that goes wrong?

Chris: Sure. I think the blip, to start with, is one that we've called blanket frontier model usage. To your point, that's just, oh, okay, if powerful is good, more powerful is better, so I should be using Fable for everything or Astra for everything. This can be nice because in a way it puts less burden on the programmer to work out what their needs actually are, because if they just hit it with a big enough hammer, it'll probably work. That's not always the case, because there can be advantages to other models that are not simply how powerful they are.

Also, it's not very prudent in a time when organizations are worrying about how much their inference costs them, because the big expensive toy might be the one that is the most fun to play with, but if everyone does every task, however simple or whatever its nature, with the most expensive model, you're unlikely to control your costs over time. Switching between different models and different contexts is probably one of the big levers you have to meaningfully keep costs under control while still getting the job done.

Effy: I think also part of the reason we've had this explosion of LLMs and AI-based tooling is because the frontier labs and the big labs are starting to build very powerful general-purpose models that can do anything without the need for context-specific training, task-specific training. That's really powerful because it means you don't have to do any work. You can just pick something up and use it. That doesn't mean that there aren't benefits, particularly for larger companies and larger users to look at models that are a bit more fine-tuned, that are a bit more domain or task-specific.

Often we find that they can get similar or even better results for a fraction of the cost. Things like GLM recently releasing GLM Legal, which is a version of the GLM open-source model that's been fine-tuned for legal tasks and now outperforms all the frontier models on legal-related tasks because of that layer of fine-tuning. I think we've all been very happy to see how far general-purpose models can take us, but certainly using a general-purpose model for everything all the time is not always going to get the best results and certainly not going to give you the cheapest results.

Chris: To build on that, I think GLM in particular was one of the model families that we discussed as getting creditable coding performance without having to go to the big-name frontier labs. There's certainly some distance between open-weight models like GLM or Kimi K3, et cetera, and what you get from the frontier labs, but it's not such a big difference. Certainly, I think that in my personal coding experience recently using GLM 5.4 to get things done, I didn't feel like I was operating in a second-class way. It was actually something that was really not quite at parity, but certainly in the same ballpark as the frontier models.

You shouldn't assume that you've got to either pay up big or stick with something second-class. There are now options, and probably the larger and the more canny organizations are looking to balance their inference mix between what they get from frontier models, or the smaller models from frontier models, or locally hosting things themselves, or even using the neoclouds that are springing up to offer open-weight inference at a more reasonable price than you might get from the big labs.

Lilly: How are things going with local coding models? I'm talking about things that run on your own host these days. What tasks make sense for that kind of thing? Because that's definitely getting a look in as well.

Chris: Yes, we've got that on the Radar. I would say that we shouldn't-- Local models are more popular and more feasible, but they're still large and not necessarily practical unless you have some beefy hardware in all terms. I think we should encourage people to separate local models, which are interesting, and you should have a look at, and maybe for simpler tasks that can be broken down with less context or less processing of a large code base, that's something you could consider.

To separate that class from open-weight in general, it's not simply running my laptop or running on the servers of the big labs. I would say, the dominant modality of open-weight is either self-hosted in your own data center or through neoclouds. I've definitely seen this misconception where people think that, because I guess open-source often means running stuff on your own machine, that open-weight needs to mean that as well, and it's just one modality.

Lilly: Effy, I know you've been experimenting with a mixture of all of these different things with your own work. What kind of tasks have you seen split up in a way that can make the best use of the tools that are at hand?

Effy: I certainly think we're starting to see cases where local models are capable of doing tasks where the data or information involved is highly sensitive and you just don't want to take the risk of sending that off your laptop at all. I've started seeing some AI-based workflow orchestrators, where you've got an LLM call here, an LLM call here, it then sends the result here, where you can say this particular step should run on my laptop with this model because it deals with looking at a large amount of sensitive data and the results will get sent off. As a result, you're keeping hold of that data store and stopping that ever leaving your laptop. I think that's one thing we're going to see more and more, is local AI is good enough to handle all of your sensitive private data.

Lilly: While we're talking about different models and proliferation, we need to talk about decision models like Jev. I think that's the elephant in the room here. Chris, your face did stuff when I said that. What are your takes?

Chris: Well, yes, so a very good point. I think the launch of Jev just before our Doppler meeting was certainly something that was causing a buzz in the room because it does maybe shift the calculus of what you should use for what. If folks don't know, Jev is a so-called system-one model from the idea of thinking fast and slow. It's able to make decisions at much less expense and with much less time than if you attempted to use a full LLM to make a quick decision.

We are seeing people even composing decision models with other models, where the decision model can make maybe a decision on what grade of model is appropriate to do the real work, for example. Also, a lot of applications in business contexts of LLMs are really misusing the generality of LLMs a lot of the time. Actually, what you need is a decision between a few predictable discrete options so that you can take the next step in your application.

If you're using an expensive LLM, you're not really making the best use of the technology because there's a lot of power there that isn't really being brought to bear. If anything, the extra power of a full LLM might allow it to hallucinate and come up with answers outside the scope of what you're asking it. Certainly, in those user-facing applications, decision models like Jev are getting folks very excited.

Effy: I think where we're seeing cases, either where general-purpose, it's often frontier models have been used for very simple tasks like sorting your email into different folders. It makes so much sense to use something a bit more specialized, a lot more cheaper to use, like Jev and other decision models, which there's been an explosion of, including ones that run quite well locally. It's just so much more practical.

Because of how much cheaper and faster these models are, we're also seeing, in cases where people love to shove an LLM but couldn't do so, now there's a whole new world opening up to them. Things like having a decision model review every line of a 400,000-line log file to highlight ones that are suspicious or that show a certain pattern or that have some other criteria which you can't necessarily build a regex for, but it's simple enough that any human eyeballing it could say, "Yes, that one, that one, that one." We're starting to see these new possibilities open up thanks to Jev and the fact that we have these general-purpose classifier models now for that kind of more obvious classification.

Chris: I should mention that we decided to blip decision models rather than Jev because, with the speed of the space moving, there are already other competitors that are built on the same design. I guess Jev is the one that sparked the initial excitement, and it's got maybe one of the better names, being named after Jevons Paradox, which really encapsulates the kind of thing that Effy was talking about of being able to use models in new contexts, but it's already not the only option. We didn't want to hone in just on one tech when it's really a space that's opened up.

Lilly: Really has. I saw a blog post last week where someone had compared a whole variety of different decision models and gave them all the trolley problem, which was definitely an interesting way to benchmark that kind of thing, I suppose. Would not recommend using the trolley problem as a thing you get a machine to make a decision about, but in case you needed to know where they sat, that is something you can look up.

Chris: This really points to maybe a challenge to the way we build the Radar these days, which is that developments come out, evolve, terms shift in a time span that's shorter than the six months that we normally do the Radar. We're maybe used to working in a tech industry that moves fast, but that a sampling rate of once every six months gives us a good look. Certainly, I think we found even on this Radar, encountering blips where the thing that blip would be about wasn't out the time of the last Radar, and we're already wondering whether it's too old to bother with. It's an exciting time to be doing something like a Tech Radar.

Lilly: Another interesting thing about that, I suppose, is semantic diffusion, because you start to talk about something and you have to stop before you can even get into the conversation because you need to define the term because it has shifted so much, even within the space of three or four months. The next theme is AI factory, which is one such term. There's a lot of stuff that goes underneath the heading of AI factory. Effy, I'll throw this to you first off. What is an AI factory, and according to whom?

Effy: That's a great question. As you said, there's a lot of semantic diffusion, but I think it's important to distinguish an AI factory from something like a software factory because in one case the output is AI and in the other case the output is software that's built using AI. I think that's a key distinction to make because there's a lot of overlap between those things and a lot of people talking about both of these spaces. Fundamentally, one is about how you build and run things with AI, and the other is how you build things using AI and get software from AI.

Lilly: Chris, have you got your own definition of AI factory, or what have you seen it shift from and to?

Chris: Well, I guess of all the terms, AI is the most overloaded and generic and has been so for many decades. The two semantic clusters I would say that AI factory tends to refer to, one is this idea of a inference mix, just to call back to our earlier comments about open-weight models, between being able to run open-weight models in your own data center, being able to mix in neoclouds, or being able to pick between one or more frontier providers. That idea of almost likea token grid is one of the things that people mean when they say AI factory.

I guess the factory metaphor is attractive because people imagine that means predictable and optimized. I dare say probably most of the people who are using the term haven't actually visited a real factory, but I also assume that must be how they work. Otherwise, modern civilization wouldn't continue. The other main path of the term AI factory that I see is effectively running LLMs and AI stuff on Kubernetes, if I could be really frank about it.

In that second sense, it's less about the kind of the grid of where you're getting your inference from and more about the way that your platform engineering is having to evolve to support the massive demands that both AI coding and customer-facing AI apps are going. We've seen a lot of blips in this area, but we've got KServe, KAITO, for example, on model serving and everything in that family, because it's a whole new dominant use case that needs a bunch of tools to be invented very quickly to make it convenient for the people in your platform engineering organization to keep up.

Lilly: There's also a lot to tie in here with the token economics issue that everybody's been dealing with as well. It's not just about how you get to-- The theme that we were just talking about using the right tool for the right job, but also how to do it more effectively, how to keep track of that cost over a variety of different providers, and all of that kind of thing. That gets bundled in here too, right?

Chris: Yes. For example, LiteLLM, maybe despite some visibility of security issues that it might've had over time, I guess that's the hazard of being a popular new product. It provides routing and abstraction across different models. That gives you a way when you do want to, external to your organization, pull in inference or connect, you need these kind of abstraction layers to keep you that control or also things like visibility or support the token version of FinOps, et cetera.

If you don't have those layers of abstraction in place and you're just connecting applications directly to frontier providers or any other API, then it's really hard for your organization to manage what is happening and when that resource of inference is rapidly becoming a significant item on your spend. It's really something you need to have. It's not just about expenditure, it's also just really about control and management.

Lilly: Just trying to think of anything else I wanted to ask about the AI factory, but honestly, that's it.

Chris: Well, maybe I guess we might have a little bit less to say about the AI factory, but that's because the whole point is it's trying to make a space that is exciting boring. That's why we're using the metaphor of factory. It's not supposed to be exciting. Really, the exciting thing is supposed to be, I don't know, the kinds of coding that we can enable that we couldn't before, or the kinds of applications that we can make customer-facing. You might say that the whole AI factory theme is about supplying the kind of infrastructure you need to make inference boring as possible.

It's good work, and I almost hope that we don't need to have this as a theme in upcoming Radars because we see a kind of standardization and stabilization in this part of the space so that maybe we can get on with doing the more creative stuff.

Lilly: By more creative stuff, I want to take us to our last theme, which is harness engineering at scale. That's the one that I doubt we're going to see any particular standstill on anytime soon. That's my view anyway. Another thing where we've had a little bit of semantic diffusion as well, since we have landed on this term, what is a harness these days? Effy, do you have a take?

Effy: Oh, I wouldn't want to try and give a comprehensive definition, but one thing that's always seemed interesting to me is that a lot of people at least used to use the term "agent" when I was thinking of a harness because they would, for example, refer to Claude Code as an agent. To me, the thing that runs inside Claude Code is an agent but the tool that's running the agent is the harness, and the harness is comprised of everything around the agent that orchestrates the agent and allows it to do what it needs to do. It's the actual tool that calls the inference. It's the skills and plugins and MCP servers and capabilities that are made at the disposal of the agent. It's effectively all the interfaces and knowledge that the agent has at its disposal to enable it to do work, I guess, would be my elevator pitch.

Chris: I think I'm yet to read an article about harness engineering that I feel has consistently used the same definition throughout. I think it's quite common for there to be contradiction even literally within one opinion set of one person. Where I think maybe my take on this would be that there's a meaningful difference between who is maintaining and creating the harness that should be part of our analysis. It is not the same thing when there is code that I rely on to get my job done written by someone who works for OpenAI or Anthropic compared to code that I and my team work on, or Markdown or whatever else, or Markdown code context that my organization contributes to together.

I think you could talk about them all collectively as being the harness, because there was one particularly well-known definition of harness as everything that's not the model. I think if you can't distinguish between stuff you're relying on the provider of the coding agent doing, stuff you have the control to do yourself, and stuff that you might have to coordinate with your colleagues to come to agreement on, that's really ignoring the main material distinction that will affect programmers day to day.

Lilly: The one thing I've seen most people agree on is, and this is going to sound trite, but it's true, that harnesses are important. Earlier in the same conversation, we were talking about how the harness is the thing that really needs to hold a lot of the guardrails and the context and keep things on the straight and narrow when you're delegating to long-running agent sessions.

All of these things have to come in somewhere, yet the harness is also going to shift context and maintainership and all kinds of things depending on which one you're using for which project, in which context, at which point in time. There's also a whole lot of things being built on top of and around harnesses, regardless of what you think they are. I know, Effy, you proposed, I think, Agent Plugins as a blip for this as well. We're talking Agent Plugins here, but this is also a harness thing. Is this a harness agent diffusion thing? How did that go? What's the plugin ecosystem like?

Effy: Agent Plugins is a new open-source standard that a few of the major frontier labs and similar have adopted. Version 1, at least, is effectively a standard for defining and sharing MCP servers and agent skills. A while ago, we had the agent skills standard come out, and that said, you've got a Markdown file. It has some other Markdown files as references. It might have some scripts that can be run. An agent should read it by looking in this directory. We never had a way to encapsulate those and be able to share those around.

Agent Plugins doesn't solve all of that. You still need something like a marketplace, which is another thing you can read up about on the Radar. Agent Plugins is at least a step forward to say, "Here's the 10 skills that my team have developed for everyone in our part of the world," or "Here is the eight skills that are relevant to this CLI that I'm downloading from this open source project," or "Here is the definition of the three MCP servers you'll need to get your agent up and running with all of the ecosystem that we're using here at the company," and define that in a standard way so that I can install it just as easily in OpenCode as I can in Codex. Before this, there has been effectively no standard in each-- Each of the different coding agent harnesses have effectively defined their own definition of what a plugin looks like, how you install them, what they should define, et cetera.

Lilly: I think in the write-up for this one, you've drawn a parallel with platform engineering from a couple of years ago that as soon as something comes onto the scene. People are trying to figure out how to build it into their workflows, how to build with it, and how to get the best out of it. They're trying lots of different kinds of things. We get multiple versions of the same sorts of ideas. Some stick, some don't. Is it the same thing here? Is it worse because people are vibe coding their own now and there's quite a lot more of it going on? Is it better because we're getting better clusters or more variety? What do you think?

Chris: I think when we talk about platform engineering, sometimes people think that means server infrastructure, because that's conventionally what's happened in a lot of organizations. Created a platform organization by renaming their infrastructure organization. I think here what we're seeing through enterprise harness shared assets across the organization is something that behaves like a platform, in that it is something that is created once and then benefits a whole bunch of other people, but isn't runtime infrastructure.

In that sense, in the sense of maybe our listeners might be familiar with the book Team Topologies, in that sense of serving a bunch of people by providing something that they can engage with as a service rather than have to go and log a ticket or have a conversation in the heat of the moment, that's where we see the kind of harness engineering at scale resembling a kind of platform.

I have to say, I don't think that I see many clients doing this effectively or well yet. What we're seeing right now is the pain point, where you have lots of different teams who are all solving almost exactly the same problem. Maybe even they're solving problems around testing or backlog decomposition, but also problems about local knowledge. How do we want to architect in this organization? What are the requirements to sort out a path to production in your particular company?

The first time, it's fine, a team solves it, puts it into Markdown, a script, or something else. The second time, well, they can probably cut and paste it, and it's okay. What you then see is that over time you have problems both in maintenance because the context you've created and embedded in a project represents a point in time, but also it's really just like copy-and-paste sharing, it's not a scalable solution.

I don't think that, though, if he's quite right to point out that we have emerging technical standards to make it possible, this is really a social and organizational problem. I don't think I can claim to have seen many people who do this well yet. However, I am convinced from the beginning chaos that we're starting to see as people try to share different bits of context that it is something that will become very important for organizations to solve.

Effy: I think similar to platform engineering and internal developer platforms, in organizations who don't have that maturity, as Chris said, we have a lot of reinventing the wheel. Every team is having to build their own CI pipeline to do the same eight checks on their code before they get to prod, and then compliance comes along and rolls out a new tool, and you do it with that. Security says, "Oh, we're shifting from this thing to that thing."

Suddenly you're having to overhaul your pipeline, and over time developers spend more time building that infrastructure, that is a common concern. We're often actually building the software, which is the unique, valuable thing. It creates a lot of waste, and also just in a lot of cases, it's not what we want to do. We're software engineers. We want to be building software and not spending all our time tinkering, trying to get our agent to work to integrate correctly with the new code scanning platform that security just mandated.

Lilly: We'll open a can of worms here, I think, with this question, but I want to ask it anyway, which is that all architecture and all infrastructure and platforms are opinionated and they push something of the intention of the designer onto the users. My feeling is that this is extra true of things like harness engineering or whatever we want to call it because it's not just pushing that structural intention, it's pushing semantic intention on top of things. I feel like that is something that we are yet to reckon with as this pattern rolls out. What do you folks think about that?

Chris: I think that people are not quite realizing that there is both the technical artifact and the understanding, and they think that by manipulating the technical artifact or pushing that on people that it automatically transmits the understanding. Certainly, I've seen a lot of people on day one, they use a coding agent, get excited. On day two, they set up their own system of work and divide things. Then on day three, they publish it or try to make other people do it.

I think the urge for that, which is very powerful in this, I've now got this new understanding, I want to share it with other people is quite laudable, but there's a bit of a problem where if the artifact outpaces the understanding, then you're going to end up with friction. I think Effy mentioned earlier the problem with taking wholesale skill libraries that might be open source and chucking them in your project, or maybe in this case, since we're talking about harness engineering at scale, spreading them around your organization, but just having copied or made available that collection of workflows and skills doesn't automatically make everyone who's received that technical artifact fluent in the concepts behind it.

It is important to share, but I think it's also important to share ideas and help people gain expertise so that they can get the most out of these things. I think it would be very tempting if you were someone at the center of a technocratic organization to want to focus all your time on pushing out Markdown files or scripts or whatever as a way to maybe centrally influence and improve the development in your organization. I would definitely say don't neglect the fact that for those things to really be effective, they have to be wielded by people who get why they're doing what they are and can also apply their own context and manage exceptions.

Effy: Particularly because we're talking about non-deterministic actors here who are prone to reward hacking. If you're mandating certain checks, certain steps, you need the human to not only understand that those exist, but when they should be applied and what the purpose of them is. Otherwise, they're going to blindly accept the AI output, and the AI might easily-- It's the case of, "Oh, the unit test failed, so the agent deletes the unit test so there's no failing tests." That's the thing where a human who fully understands the context and the reason we have the tests is they're going to say, "No, that's not good. Go back, restore the test, fix the failing code."

A human who's just blindly running some of these artifacts produced by another team without understanding the workflow, the intent, and what's aiming to be solved there might just readily accept that change in the AI, which is again coming down to that kind of cognitive surrender and human-in-the-loop theater.

Lilly: I've really liked that all of the four themes in this episode have been about AI and LLMs, and we've spent most of our time, I think, talking about people, which is where I think this should be at. Which themes do you folks think will still be strong here in a year, apart from presumably humans as the meta theme? Effy, I'll start with you.

Effy: I think, as Chris said, AI factory is one that hopefully we get a degree of solved or standardized, or at least it becomes as boring as it should be. I wouldn't expect that to stick around unless something majorly changes. I do think that there's a lot more to be explored around putting agents to work and multiple agent orchestration. We talked about some of the local innovations in that space, but there's still the big question of cloud agents versus local agents and still a lot of rapidly developing technology to enable you to run multiple agents in the cloud, for example, or run agents in a context that they're visible to multiple members of your team.

I do think that harness engineering is broad enough that there's always going to be something to talk about there. One of the big clusters of proposed blips that we dealt with the other week, which we haven't even really talked about here, is a lot of different frameworks and tools designed to solve the issue of memory and context knowledge, things that build local knowledge graphs, that provide tools for an LLM to access an indexed catalog of knowledge or an indexed abstract syntax tree of a codebase. That problem of memory and context is still far from solved. I think there's going to continue to be a lot of innovations in that space. Harness engineering is not going away anytime soon, would be my take.

Chris: I'd really agree. Harness engineering at scale is about large groups of people cooperating to get stuff done, and that has been a challenge of society since society became a thing. I don't expect that to become redundant anytime soon. Very possibly we might stop talking about it as one thing because it is such a potentially large area, starting to share context artifacts across projects and provide guidance, et cetera, that maybe we'll have a little bit of a more in-focus view of it, so we'll be able to pick apart the different parts next time.

Maybe I'll also mention the one that I don't think we've discussed yet, which is around frontier models. I think that might also be one where it'll continue to be massively relevant because we should continue to see innovation in models. We'll see more possibilities of different ways to run models or different levels of models. I'd hope, though, that we don't have to say beyond frontier models in the future, because, really, the idea that, "Let me just pick the shiniest toy of the set and use that," that's really quite a sign of immaturity in our profession that we have been doing that.

I'm sure I've been guilty of using an outsized model to sort my emails or the example that Effy gave earlier. I don't mean to project it to other people, but I hope that our fluency with choosing different tools among this set raises. We don't have to say beyond frontier models, but I really hope that we're talking about the next decision-model-type innovation next Radar because I think there's a lot more to come.

Effy: I also just hope that we do continue to have such a plethora of choices when it comes to models. We're starting to see that those open-weight models are getting good enough that some people are getting quite concerned. We're having Anthropic jumping up and down about the cyber capabilities of some of the latest open-weight models and suggesting that they should be curtailed.

We've already had the US start to apply export restrictions to frontier models when it comes to things like usage by China. We're now hearing, as of the time of recording, the first murmurings that that might go the other way and that China might start cracking down on what gets released globally by the Chinese labs that have been putting out a lot of the leading open-weight models in the near future because of that kind of one-sided capability gap at the moment.

Lilly: If you've been listening to all of this and you're wondering where you can get more of this, where to keep up, I've got good news to you. First thing is that Chris Ford is writing another book, Agentic Engineering at Scale.

Chris: Yes, that's the title.

Lilly: Which will be with O'Reilly out early 2027. You can find that at oreilly.com. Anywhere you get books, I believe that will be a thing. Also, Volume 35 of the Tech Radar will be out. At the time you're listening, I'm fairly sure. Thoughtworks.com/radar. Publication will be in early November. You can hear about all of these themes. You can drill down into the smaller details. You can see all of the stuff that people have been talking about and judge it for yourself. Effy, Chris, thank you so much for joining me. It was really great to have you here.

Effy: Yes, it's been good.

Chris: Cheers. It was a great chat. Thanks very much, Lilly.

Lilly: Join us next time on the Thoughtworks Technology Podcast. We'll be here to have more conversations about this and more. You can find us anywhere you get your podcasts. Like, subscribe, all that jazz. We'll catch you next time.