What we're talking about when we talk about context engineering

Episode transcript

Rachel Laycock: Hello, everyone, and welcome to the Thoughtworks Technology Podcast. I'm Rachel Laycock, the global CTO for Thoughtworks, and I'm joined today by...?

Bharani Subramaniam: Bharani Subramaniam.

Rachel: Okay, and?

Alessio Ferri: Alessio Ferri. Hi, everyone.

Rachel: We're going to talk about a hot topic as we were putting together our technology radar this week, which is context engineering. We had quite a hot debate about what it was, how you define it, what's going on in the industry. I have these two special guests with me who seem to be deep experts in this space right now, which is rare given how new it is. Let's start with what is your definition of what context engineering is?

Bharani: Context engineering is this emerging field where you curate what the model sees so that you get a better result. That is what context engineering is.

Alessio: Maybe before we go into the context engineering base, I'd like to define what the actual context is. Context is this concept of providing tokens into a model so that it can eventually produce an output. The engineering is all the practice that sits around that and applying engineering principle throughout the design of what goes into this model, how we build this context, and also how do we test it. Context is actually a very broad topic that could include, for example, memory, conversations, data that's being accessed by an LLL. It's an umbrella term that covers everything that goes into a model as its input.

Rachel: I've heard argue, though, that that is actually prompt engineering and that this is just a piece of prompt engineering. What are your thoughts on that? Do you think it's a different thing? Do you think it's an extension of prompt engineering?

Bharani: Yes, I think this is mostly true, and at times, false, because some people say context engineering is a super set of prompt engineering, but if you look at it, we started with a prompt, a carefully curated template, so that a user can structure the input to get the desired output. What really happened is when you're building complex systems, it is not just a one-hop, single prompt, single output. You are going to have a multi-hop. Then what is carefully curated is no longer true.

You're no longer worried about how you are structuring for a given prompt. You're going to deal with a history of such conversations. That's where I think you have to make a switch from how am I crafting my input to what the model actually sees? I think this shift is what, in my mind, that actually made us focus on, "Let me worry about the bigger picture of what model sees as opposed to how I'm crafting a single piece of input."

Rachel: What impacts does it have? What is a good version of this? Maybe give some examples of doing it well and the impact of doing this well versus not well?

Bharani: I think I would say that we are still discovering this, what works well, what doesn't work well, because if you think about it, almost all models right now are stateless in the sense they don't have any memory, and it's a pure function. If you think about it, the only thing that can affect the output is the input that you give to the model. What we have seen is that if you have an extremely large input, irrespective of how big the capability of the model is, even though a model can take, let's say, a million token, if you give a really large input, we have seen the actual quality of the response to it.

You would also burn a lot more tokens when you have a very lengthy, messy input. I think one technique that has worked for us in the past is that our time, and this is where it gets subjective, you need to compact and curate your history. You don't need the entire messy output of those tools. You may have to consider compacting because-- I say this, it's almost like you visit a doctor, you need all of your history, but also, you need a neat little summary so that they can quickly see what has happened and start from there.

Rachel: Yes, almost like when you present to executives and you need the summary review, and then all the appendix is later. Who's building the context? Who should care about it? Who has to craft it? What skills does it take?

Alessio: I think context engineering, from what Bharani was saying before, we started experimenting with prompts and understanding how we could ask to a model better the instructions that we wanted the model to do for us. Prompt engineering exposed everyone, exposed developers. Building with AI exposed developers that were using AI to build software. It was exposed to users that were interacting with models directly such as on ChatGPT. This technology is quite ubiquitous today, and everyone can access it.

I think that context engineering should be something that we all care about. It's not just who is building AI, who is building with AI, but it's also users, because as we interact more and more with these tools, as we feed, for example, information and we feed documents to these interfaces that allow us to work with the data that we as organizations own, then everything that we feed into that context has to be relevant. We need to almost help the model that sits behind to understand and focus on what's relevant for the task that we are putting in the prompt.

I think what we said before also about the fact that over time, we've seen context windows grow in LLMs, but as Bharani said, the attention focus hasn't grown as fast as those context windows. It's even more relevant today because if we don't do context engineering, the impact that what you were saying before, Rachel, the impact is we have uneccessary costs when we run these models. We lose focus from the LLM or potentially miss this important information, which can be quite impactful for the task that we have ahead. Maybe it misses a specific requirement in a document that were fed into an LLM or potentially even hallucinates because there's so much confusion in the context that it sees that the model is not powerful yet to handle that.

Rachel: I think escalating cost is certainly something when you think about leveraging these models at scale, it's going to be a growing concern, I think, in the industry. It'd be good maybe to dive into, and you can both share some of your own examples of like what-- because one of the things we talked about a lot, actually, in our sessions was the different techniques that exist. Maybe, Bharani and Alessio, you could tell us some of the techniques you've used and what have been the benefits of those, and when you might use one technique over another and what you've learned. I recognize it's early days, but I think people would be really interested to hear what have been our experiences so far.

Bharani: I would say that the most important principle is you need to build the entire context engineering around the KV cache. Imagine this. Let's say if you are giving the entire history for every single conversation, the model is not going to execute that extra. It's going to just compare, "Do I have it in my cache or not?" The most important engineering technique is to treat the context is append only.

You should be intentional about rewriting history because every time you rewrite, the model is going to recompute and it's just not the last instruction, it's going to recompute your new history. I would say the fundamental principle of context engineering is to treat-- Context is mutable, but treat it as append only. If you want change, be intentional about, let's say if you're combating history, then you need to be aware that it's going to cause inorderly computation. That, I would say, is the first principle.

Rachel: What about you, Alessio?

Alessio: We mentioned before the different users and people that have to be concerned about context engineering, and we've seen some techniques that are very accessible to anywhere, for example, using AI to build software. There is emerging techniques such as these, what we call AGENTS.md, and every coding assistant provides their own version or name of that, but ultimately, it's about providing context about, for example, the project that you're working with, how folder structures are structured like that and why have you done it with that intention so that the LLM can almost understand what your principles are and as it addresses the tasks that you fit it with, it knows about that context.

That's, I think, why quite accessible for everyone interacting with these tools to provide that extra level of context to a model. I think there's some very interesting techniques once some of them have been quite exposed by building internally at Thoughtworks, the CodeConcise accelerator. That is what we call RAG. RAG has many different facets to it. What we ended up building is GraphRAG, which I find is a very interesting technique, because it's not just about providing that context to the LLM, but it's also making the context understandable by people because that context is a graph, and you can navigate that. It's about making the context not only understandable by a model, but also trying to optimize for that context to be explorable by humans.

Rachel: Yes. That's funny because one of the other topics we talked about was AI-friendly code bases, I remember. Just that made me think about that as another topic we explored as to why identifying that good coding practices that we generally use to create modular, easy-to-read code actually makes it easier for the AI to read it as well. Ironically, that when you're generating lots of code with AI, it doesn't necessarily build it in a modular and clean way, so you almost need the humans in the loop. I guess that brings me to another question around this, which is, where are the important human-in-the-loop pieces of building out, doing context engineering well?

Bharani: The point Alessio touched is important. For example, we were thinking of, let's say if you have a piece of data, should you settle for a JSON format or are you good with, let's say, a CSP? Both are human-readable, but the CSP is going to be much more compact because you're not going to repeat your headers and things like that. It fundamentally comes down to how many tokens you want to spend. It should be human-readable, but at the same time, compact. I think that's where the empathy for the tokens comes in, because every token costs and everything counts in the latency as well.

Rachel: Right. It's that mechanical sympathy. It's like we have to have empathy for the humans so it's human-readable, but also empathy for how the machine is going to operate and the potential for spiraling costs or other infrastructure concerns. I know it's early days, but do you think that there's any long-term patterns emerging, or is this just are we still in the early days of some techniques work but we haven't really established what's long-term at this point?

Bharani: I think that is a pattern, I would say, almost in the AI industry right now. For example, lots of the prompting techniques are now fundamentally at the model layer. For example, the chain of thought is understood by the model and not just at the prompt layer. Over time, if you think of it, people draw an analogy to the entire operating system model. Like right now, you think of context engineering as where we are programming with C, perhaps, and you are manually managing your memory, your paging, what should be in the RAM, what should be swapped out and things like that.

You can draw an analogy where in future, maybe in the model, or maybe you would have a smarter operating system, an equivalent of that for AI where the context as the working memory is managed better either by some standardized scheduler-like processes, or natively in the model. I think we can expect that to come up because everyone who's building agents, everyone who's working with model has to deal with this problem at this point of time.

Rachel: Right. I guess I'd be remiss not to ask you where MCP fits into this. Where does that play into what you've been building, and what are some of the challenges you've had with leveraging MCP that you've already experienced? Because we did talk about that quite a lot over the last few days as well.

Bharani: In your mind, if you can think of, there are three important agents, I would say. If you can think of an agent that you're building and the resources in your organization, the protocol that bridges that gap is like the model context. You need some ATD that provides your contextual resources to the model, and that's the MCP server. I have multiple agents and microsystems, multi-agent systems. I need a standard for that. Okay, so that's your agent-to-agent. We talked about agent-to-agent protocol as well.

Then the third one is we are building these user interfaces for these agents, and you need a standard for that, that's the agent-to-UI spec. I think all these protocols, MCPs, agent-to-agent, or agent-to-UI, they're all emerging, but of all three, I think MCP is the most mature right now because we are trying to expose and make these applications a lot more context-aware with the internal data.

One thing to call out when building MCP is that the security is still a challenge, that the protocol is evolving, they're incorporating a lot of these recommendations from the security community. We've come across a lot of MCP servers that are just built over the weekend as a hackathon exposing a lot of sensitive data. That's something to watch out for. I think there's also an upcoming tool called MCP Scan that's coming up that we are positioning in the radar.

Rachel: Then I think the last thing I want to ask, because when we're talking about context and another topic we discussed quite a lot over the last few days, is around spec-driven development. This push, in the context of building software to give as much context, to let the AI run away and build things. Is there anything you want to raise with the audience of what's working and not working in that space and what we discussed over the last few days?

Alessio: I think we were saying before that context engineering might sometimes feel like-- and Bharani made this point, that it might feel like it's what we provide as input to a model. It's actually also what comes back from a model because most likely, what comes back is also going to be the input to another model.

As part of a lot of work that we're doing in Thoughtworks is trying to use AI-- is use AI to understand better and faster legacy systems. On the other side, we have also teams that are exploring how can we use these specifications that we produce, which have the right level of detail, which are not ambiguous for an LLM to be able to implement and with engineering practices around it so that we can test what the LLM is producing.

We are actually looking at, can we bridge these two gaps? Can we, for example, extract documentation and create these specifications from a legacy code and then feed that for forward engineering by another LLM to produce? I think I really like the nature of the LLM as in it can be quite creative at times at solving problems. When I say creative, it's quite interesting because how creative it can be, I don't know.

What I find quite interesting is the fact that when you take legacy code and you describe it into these specifications, finding the right balancing between providing a lot of detail and not enough detail is actually quite hard because when you provide too much detail, you might end up with a solution that looks too similar to the one that you already had before. When you find a way to actually describe the problem without actually describing how it's solutionized, but describe the problem and what you want to get out of that problem, then that leaves enough room for the LLM to be able to understand and use its training data to find a way that's good, given the modern techniques that we have to solve problems to implement that solution. I think that's actually very, very powerful.

Bharani: Just want to stress one point that there is these lots of different opinions saying, "Oh, you know what, you don't need engineers for this anymore." If you actually look at these specs, they are design and abstractions and good examples and structured communications. All of this is what I would expect from a software engineer to do. It's just that it's another layer of abstraction. We are like from programming languages with types and rigid structures, we are going into English. If you look at these specs, they are still engineering documents.

Yes, you can get a trivial example just with a plain English because model has seen enough hollow world programs out there. But when you start solving for enterprises and really sharp problems, you need to have a good abstraction so that we don't repeat yourselves. Expressing that in the document is going to be as challenging as expressing in any language. That is the tough balance or the hard balance.

Rachel: It's a really interesting space because it feels like the whole industry has been trying to create the citizen developer forever, and with local no-code platforms. I think when we try to go too far down that route, especially if you're building new complex systems, beyond the hollow world, getting the spec right is actually part of the whole agile software delivery feedback loop. It's like, that's the hard part.

If you lock that down too much too early, you're going to have to do too much rework. I do think in Alessio's example, where I'm seeing a more solid and workable use cases on legacy systems where you're trying to describe the behavior so that you can then-- a large portion of it, you would essentially going to want to reimplement. It's very early days and it's probably its own topic in and of itself. We do have to go today. I just want to thank the folks that have joined us, Bharani and Alessio. There will be more to come over the coming episodes about some of the deep discussions we've had this week as we're recording the Radar together. Thanks very much.