Transcript

Ken Mugrage: Hello, everybody. Welcome to another edition of the Thoughtworks Technology Podcast. It's an interesting thing. At Thoughtworks, we have this thing called the Global Tech Leadership Forum, and it's some of our technology leaders from around the world, and we get together on a regular basis and chat about things. On a recent one of those, one of our guests put in the type-in chat, "Do we even need to see the code anymore if we're talking about AI?" which, of course, lots of people had responses to.

I'm sitting there going, "You know what? This would make a good podcast," because the podcasts are supposed to be discussions. I'm going to ask our guests to introduce themselves real quick. I'm just going in the order on my screen. Razin, can you go and introduce yourself, please?

Razin: Yes. This is Razin. I am a principal consultant in Thoughtworks, which is also called as enterprise architect in some companies, and I've been in Thoughtworks for approximately 9.5 years now, almost 10 years, completing very soon, and working as architect in multiple projects.

Ken: Caer?

Caer Sanders: Hello, I'm Caer Sanders. I'm also a principal at Thoughtworks. My background is in a mixture of robotics and artificial intelligence. I'm also a writer, so my degree in education is in writing of all things, which seems relevant today.

Ken: Very much so. In fact, I may have to sign you up for a couple of blogs. [laughs] It's interesting because this discussion came up, and like everybody, I do initial research using the different LLM tools and so forth, and so I went in, and we have one that can see our corporate things, of course, and public, and I said, "Hey, write me a little brief." It got the guests' roles wrong because I didn't give it the original chat. It looked at the writings and the public information, and it assumed that Caer was the one that didn't care about the code, and Razin was the one, from the enterprise architect perspective, that was going to be, "Oh, no, I need to approve everything."

I just found it really interesting that it came back. I guess I'll start with Razin. You started it with the question. What's your stance? Are you thinking today we don't look at code? Is that a future thing? Where do you stand on this?

Razin: What I feel is, in 2028-ish, the code will not be any more relevant. Why I think so is because what will happen is the tools that we are telling ourself, like IDEs, et cetera, those will convert it into the new harness engineering tools, which would come up with lots of skills and harness and tools around it, which will make spec so good to be converted to binary directly or if it is converting to code and then to binary, I wouldn't be worry about the code. I would really not be worried about if the code is coming in between or not, so I would just be worried about what specs we want and whether the code is getting generated right or not, and I will just generate the right test, test it, and just figure it out.

Ken: You're not saying today; you are saying that's technically 16 months out?

Razin: Yes. Why I'm not saying today? Even today, it is relevant because I use lots of non-looking code things, but the biggest problem today is token dynamics, et cetera. Even today, right, for example, today, Meta has released one of the model called as Muse, and that is giving really, really good results in terms of coding. That started happening today, and then methods like BMAD have started coming today, but the biggest problem today is a token usage, so that's why I'm saying not today. When we will solve a token problem and local LLM problem at that point of a time, and that is not very far—that's is going to be max one year or one half year.

Ken: Caer, you said some opposing-- In fact, I think you were in a discussion that day with one of the leaders in observability. I'm not going to use her name, but you're certainly welcome to do so because I wasn't part of that conversation. What's your stance on this?

Caer: I've been working with AI for about a decade now and building AI platforms since before GPT-3 launched and brought AI upon all of us, and so I've had a lot of experience with it. What I'm noticing is that AI-generated code gets the job done in the vast majority of the cases, and I can't at all contest that, but when I'm working on a safety-critical system or a system where downtime costs millions of dollars an hour, I still have to read the code and make sure that I understand it and know what it's doing. I feel very strongly about that.

A lot of the work I do professionally is safety-critical. A lot of the work I do professionally means I have to read the code still. I was talking with one of these observability leaders in the industry, and we've been writing a lot about the split between AI skeptics and AI optimists. It's a very divisive split; you're either one or the other. I find myself in the middle because the skeptics are like, "We shouldn't use AI at all." The optimists are like, "We should use it for everything."

What I'm realizing is that this transformation feels a lot like what we saw with infrastructure as code 10, 15 years ago, when Terraform came out; there was this whole movement around servers are no longer precious pets, they're disposable cattle. That same analogy is now being applied to code, where they're saying LLMs have made code cattle instead of pets, but I think that's the wrong framing. I think infrastructure as code made servers less precious, and this AI-generated code movement is making code less precious, which is different. We still care about the code, but it's less precious as an artifact.

What matters more is all of the knowledge that goes into specifying it. I can go on along, talk about that in a moment. I'll get back to you.

Razin: Yes. I have a counter to that. Infrastructure as code is a means to achieve a real servers, which is an infrastructure, but your real infrastructure is still a code, whereas code is a means to achieve a real software that you want to achieve. What I'm saying is real software will still be important. You will really need a software; you will really need a test, but there is one intermediary in between, which is nowadays coming, which the intermediary is called as code. Basically, somebody is giving a requirement that is getting converted into epics or stories, et cetera.

Then one intermediary is coming which is converting that to the code, and then the code is getting converted to the real software. Then real software is something that is being used in the world. What I'm saying is that intermediary relevance will just go away. It will be the first thing people will come write the epics, story specs or whatever you want to call it. There are different names that people are calling, and then the final software and the test. Those are the three things you will be worried about. That intermediary thing, that is code, which will be of no value and no meaning to the people.

Ken: Interesting. I try to not put my own opinion in here, but rarely succeed. In the time period that Caer just mentioned when they were talking about, especially the code part, and they throw away in cattle and things, some people may know I was deep in the DevOps movement and I was actually the product manager for our continuous delivery tool for many years and it was very easy, Razin, to take that position and say, "Oh, it's not really the same," but we find that infrastructure's not just the hardware, it's the networking layer and the performance and which data center and that kind of thing.

Often, what we would do is we would put that on the development teams and say, "Hey, you're now a Terraform person, but don't worry, because--" all the things, and then suddenly we go down at 2:00 AM, and nobody would have any idea how to fix it. You don't see that as a problem at that time. Do you think it's just an economics problem today?

Razin: The way we are right now caring in the Terraform about the real cloud-- For example, AWS has its own language, which is a competitor of Terraform, and Azure also has its own, and GCP also has its own, but lots of people don't use those. People use Terraform, but while writing Terraform, they still think about nuances of which kind of a server I want or what kind of a thing I want. What I'm saying is, while writing spec, people will be worried about what kind of logic I want.

The logic that we write in code, the conditions, the loops we write in the code, those will be in a much better or a much simpler language that anybody can understand, but people will still be worried about those. Those in the specs say that this is the logic I want or this is the category I want, or this is the kind of observability I want, or this is the kind of hardness I want. Those kinds of things you will still keep writing on the spec, but the spec-- and which we even today write in the stories and epics, but stories and epics get converted to the code, and then I'm just looking into those code; those worries will go away.

All those things will be there in the spec, and then you worry about the final software, and you worry about the test. Those three things you worry about, and then everything else harness will take care of.

Ken: You're really putting all the weight in the harness.

Razin: Lots of weight on the harness, and that is going to be a really big industry for sure. Right now, lots of harnesses are provided by the model provider itself. I see a future where model provider and harness provider will be two different industries, and there'll be a really big weight on the harness provider. I see Thoughtworks being one of the leaders in that industry, AI/works being one of the best harness we are going to have.

Ken: We could spend an entire episode defining harness. Luckily we did, and it just published recently. For our listeners, I will answer-- we'll put links in the transcript. There's a great article on martinfowler.com that talks about what a harness is. Caer, what's your response? What do you think about that? Is a good harness good enough? Are guardrails good enough?

Caer: It's complicated. When you're working with a system, so much of the system's specification isn't written down. It's written and embedded in code; it's written and embedded in people's heads. It's all tacit knowledge. It's not captured anywhere. With harness engineering, what you're asking engineering teams to do is take all of that tacit knowledge and put it to paper in a very thorough way. If you can do that, I do believe that that could work. The lady I've been talking to in Durability, Charity Majors, she recommended to me that I look at the Phoenix Architecture by Chad Fowler, which is this whole series that he's written on regenerative software.

That series was the first time that someone convinced me that maybe it's a good idea to not let engineers write code anymore and only let them regenerate software from specifications. That could be its own episode right there on why that is. If you truly can get to a point where all that knowledge is captured in specification, in documentation, in conformance tests that aren't specific to your tool stack, then I think you could generate all the code and never look at it, and also nuke all the code every day and regenerate it from scratch if you had the budget for that.

Ken: Some of our listeners may or may not be familiar with — I think all of the guests are — Bridget Kromhout, who does a lot of DevOps things. Bridget used to say, ‘Kubernetes is easy; people are hard.’ Is that what I'm hearing from you, Caer? It's like, to get people to-- if you had a great spec, but you ain't going to get a great spec.

Caer: That's exactly it. I think people and social problems are the stickiest part of engineering now. They always have been. It was just hidden behind a layer of writing lots of code.

Razin: Yes. There is one solution to that also, and that's where, again, my harness engineering point comes into the picture. I will divide spec into two different parts. Definitely, people are not really going to be really good at writing all those things into the spec or going into the details or as much as possible. What if we divide into the two different parts where we let some experts, like, for example, Thoughtworks, write specs, or nowadays it is called skills, and then people also started calling it as plugins.

We write a plugin, which is a combination of skills, and the skills are actually giving all the prompts or the specs about how you should write the code, what are the good practices, what are the things we should avoid, what are the things we should not avoid. All that is bundled into the skills given by harness providers. Then the real creator of a software just gives what they really need. It's all about what they need. That's my whole argument why I think that the code is not going to be really important, because in code, we do lots of things together, both the things together, good code versus what the creator of the code really means.

Here, the creator of the spec will only be what the creator of the code means. Then, if they are not clear enough about what they need, they will not get a clear enough software, but then they will realize, "Oh, I'm not getting clear enough software. This is why my problems are." They will keep improving the spec. At the end, spec is what the real creator of the software needs. Software engineering parts are handled in the skills. The creator only says, "This is what I need," and they get that software. That's going to be the future: "I need this, I got that software."

If there is a problem on that, I got something else I did not explain very well, then I improve the spec. Finally, I give the spec that, oh, I wanted this software, I got this software, these are my tests. Done. I just get the software and start testing. Why would I care what happens to that spec, how the harness provider is taking skills, how they're creating something called as code, how they are dividing the code, good code, bad code. Why would I worry about it?

Ken: What about things like performance and carbon impact and all of that sort of thing? If a non-deterministic system is writing the code and it passes all my tests, and most people don't know what fitness functions are as far as response times and those sorts of things. What about that situation where the code that's created is just bad? I'm taking a stance here. I'm not saying it always is, but it's just not as performant. Now you're spending more time on your cloud costs, and that sort of thing. Are you worried about that, or you think the LLMs are going to be just better at it?

Razin: Why LLM have to be better at it is because if it is not good, and then if people start seeing that, "Oh, this LLM is not providing good software; I'm just giving a prompt, and then I'm getting a software, but it's not good enough," they will switch the LLM provider. The competition will make sure that people will start going to that standard, that, okay, people really get what they want, real performant code. Then the second thing is it's not only the LLM provider, it's a combination of both LLM provider and harness provider. Together will make sure that the code being created is really good.

Then people have also gone into the level of it's not only about people not caring about code. It's also about, I think there will be LLM models which will directly create binary from specs. Actually, the LLM themself also will not even generate the code. Elon Musk said that, "I want to work in that direction, that I will directly generate a binary from a spec." That is a different direction, and that will take five, six years. Like I'm saying, the harness provider and LLM provider, if they're not good enough, then people will switch them, and the competition will make sure that they're good enough.

Caer: I think that we may eventually get to a point where spec-to-binary is a reality. I think it'll be a very long time horizon because there's so many architectures to compile too, so many architectures to target, and so many optimizations that lower-level languages like C, C++, Rust are already doing today. It'd be really hard to recreate, add an L, an inference layer. That's one piece.

The second piece is a lot of my career has been spent in performance engineering, I guess you could call it, trying to get things like vector search down to five milliseconds per request. You start to really find what you can squeeze in your system to get there. In my recent experience, I'm using Claude Fable almost exclusively, which is a little wasteful, maybe. I've been using it to build some systems in Rust on the weekends and trying to apply the regenerative software principles to it. What I've noticed is that, even Fable, which is one of the most heralded frontier models, still does unnecessary memory copies and things like that.

I mention that because those are the kinds of things that are really easy to not do, and they tank your performance really quickly. It doesn't mean that LLMs can't do that. It just means that right now I'm still very much in the mind of having to read the LLM's code when I'm doing performance-sensitive work because they're still making really junior mistakes in their engineering. That could get better over time; I'm surprised it hasn't gotten better already, in a sense.

Razin: I think this kind of a problem will more be solved by harness side, the skill side, because if we are doing performance-related work, the skill needs to be kept enhancing that, "Hey, don't do this, don't copy memory," all those things. Harness providers will keep into the skill, and big skills will get generated by harness providers. That's why I was saying there'll be two different industries in future: LLM provider and harness provider, because both are equally important.

Writing into the skills that don't make this kind of a mistake and do this kind of a coding. For example, Thoughtworks saying that, "Hey, these are our practices. Do these practices. Do internally; this is how we generate code, small code." All those will be our future.

Ken: We originally had another guest that we were going to have at the same time, who's our regional CTO in Europe, Giles. One of the reasons we were going to have him is that he did some real research on the tokenomics or the economics part of this. He's going to publish all these results, so don't quote me on exactly. Basically, he built roughly a 150,000-line application. A lot of it was refactoring from enterprise, because that's a lot of what we do is enterprise modernization.

Then when he ran it, he noticed a thing go by in his terminal that said an edit to line 4,000 scrolled past his terminal. He's like, "Whoa, what is this thing doing?" He took it out, and he did some manual refactor to do it in stages and took his token cost from 159,000 to 27,000. 83% savings because the tools he was using today just couldn't break it up into logic. They could break it up into functions and that, but not logic.

They act like they're reasoning, but they're not really reasoning. Razin, you said a second ago that it's the economics part that has to be solved. What makes you think they're going to do that? Why is that in their best interest?

Razin: Yesterday I was working on one of the tasks. I'll give you an example. I asked LLM to do something; I gave it, and then it took 11 hours for a local model to do because it's very slow, and it took 190 million tokens, and then cost was zero because everything happened locally. We are seeing examples of local models, like Qwen3.6 yesterday was able to do that. Today, Muse has been released, which is claiming to be even better than what Qwen3.6 is able to do.

This is what companies need. Companies are not going to spend millions of money on API costs. They have to be reduced; otherwise, nobody will use Cloud, right? If it is not as usable as everybody wants it to be using, then they will go bankrupt. They have to reduce the cost. Then many companies can't afford so much API cost. People will go into the hardware side. People will go onto the local model side.

That's why I'm seeing the trends. I'm not just guessing, I'm seeing the trends that good models are coming, and people are moving in those directions. If I'm able to do a full learning management system kind of coding in 48 GB RAM, then I'm pretty sure when 128 GB becomes common for everybody or terminals become common for everybody, I think we'll be able to do much better.

Ken: Is this a case, though, of the rich get richer? What do I mean by that? Not everybody can afford to run things locally. Not everybody can afford to run 128 gig. I have a friend doing a desktop machine, and with the cost of RAM, I think it was $450, US dollars, for 32 gigabytes of good-quality, high-performance RAM. I look at another experiment that was run by someone in my office at Thoughtworks. The tools, using the correct model in Claude and OpenAI's models and those sorts, was $5 to $7 to create this application that she wanted.

There was a low-code, no-code platform that's prompt-driven that I don't want to name, of course, and it was $27 to do it. The cost to the person who can't afford the low-code-- It's like the same thing happens with food, right? If you're in a neighborhood that's a food desert and you can only buy groceries at the local corner store, you pay $6 a gallon for milk.

If you're in an affluent area that has a fancy store, you pay $3 a gallon for milk.

How do we, as technologists, make sure that, societally, everybody can do this, right? Is it going to be only the people that can afford local models, or how do we get it to be the business people who, let's be honest, they're creating software, and they're creating bad software in many cases—how do we get them to do this? Do you think it's a tool problem?

Razin: It's a harness problem. The harness is not matured enough to have skills that are giving LLM enough ways of doing the coding. For example, the BMAD method, right? I don't know the creator, so it's not a promotion or something, but the BMAD method is something which is right now a combination of skills where it is doing the full agile. You give a prompt, and it is creating a simple PRD, then UX document, then architecture document, and then it is creating the code, and then review code and then they are talking to each other. Two LLMs are talking to each other and solving the problem.

That kind of a harness, when it will come, and more people will look for reducing the cost. If the cost of the tokens will reduce or local hardware will come, then that kind of a competition will be making things affordable for everybody. Better harness will reduce token cost, and competition will reduce token cost. Then the competition between local models and API comes, right? Automatically, Anthropic or OpenAI kinds will reduce the cost of the tokens.

Ken: Caer, what are your thoughts?

Caer: I'm always cautious about playing around what a business team is going to do to lower my costs, when their incentive is to make as much money for me as they can. I'm less optimistic about AI costs going down. If anything, I expect them to go up as all the subsidies and incentives start to dwindle. That said, I have noticed that I have peers who have built software using Claude — let's say it was Sonnet 5 or something — and they've spent, I think, genuinely thousands of dollars in API calls a month to build a relatively simple application.

I'm able to do the same thing with a frontier model for $10, but the difference in our workflow is that one is very waterfall, where my peers are saying, "Here's my entire spec. I'm going to write it ahead of time. Go one-shot it." The other one is, "I'm writing a bunch of tiny concept documents of all the major criteria that I have for the system and the external behavior I want it to have. Let's do it one at a time slowly." That takes a little bit longer, but I get a better result for a lot less money. That's my own compromise when thinking about LLM cost, is just changing the way I use it, as opposed to hoping for external change.

Ken: One of the things that I've poked a little fun at some of the LLMs on LinkedIn and other places, I had asked a question eight or nine months ago just for correlation between payroll and success in US baseball. I said, the World Series had just ended, "Take the top five teams. Show me what their payroll was." I just wanted to see if there was a correlation. I knew that it might not be accurate on exactly the payrolls because, frankly, they make that stuff hard to track. For those that don't know how US sports work, they give them free cars and houses so they don't have to show it up as payroll, whatever.

What struck me is it got four out of the five wrong on where they finished. This was easy-to-find information of who won — we call it the World Series, even though it's only the United States because that's just the way we are — and the other top four on top of that. It got one of them placed right. The other four, it was wrong in where they even finished. I think about, if we watch the media, our listeners, if you watch the media, if they cover a story that is something that you know really well, you're going to be like, "Ooh, that wasn't really right," a lot of the time.

If you do a query to most LLMs about something you know really well, a lot of times, you're like, "Ooh, I have to tweak that. I'm going to ask Claude to do a function, a small piece of code, 20 lines," or whatever, in your IDE. You're like, "Ooh, no, I'm going to change that." If we are experiencing that level of accuracy, that doesn't worry you, Razin, or are you just going to be like, "As long as the test pass, I'm okay"?

Razin: There is a concept called as bug in software development. What is a bug? Bug is something that I wanted some software, and I did not get that software. Same thing is going to happen. How we handle bugs, same way we are going to handle in the future. If I'm not going to get the accuracy or the things that I wanted, I improve my spec, that, "Hey, I wanted this also, which I am not getting." Then consider it as a bug fix. That will go to the spec, spec will get improved, and then internally, it will do code or whatever it want to do, but it will fix the bug, and then it will come back with that accuracy.

If it doesn't come back, then it's LLM's problem. LLMs will solve it because I explicitly ask it to solve it, that I need this accuracy. This is how it will happen. On the first go, even today, we don't get the software that we want. We get things like bugs. We get things like improvements. We get things like MVP. Even in the future, we will never get it. Most of the time, even humans who really want a software to be created, they themselves don't know what they want to be created. They will give half-baked requirements, and they will get half-baked software. They will keep improving.

Ken: I have to admit, I was playing devil's advocate there because, recently, we were talking to one of the national health services for our country and saying, "Hey, this AI for medical was only right on initial diagnosis 85% of the time." 15%, that's a lot. That's wrong. It was funny because the CTO stopped and looked at our CTO and said, "How accurate do you think people are?" It was 82. [laughs] It turns out medical diagnosis is hard. Caer, I'm going to ask you this because you were talking about people overthink. Is there a tendency that you've noticed where we expect more out of these tools? We're like, "Oh, this wasn't perfect." The person maybe isn't either.

Caer: The phrase I'm thinking of is cognitive surrender. I don't know if people have higher expectations of LLMs than humans. That's a little unclear to me because, like this example, we accept 100% for the LLM, but people are only 82%. That could be the case. What I have noticed is that people are more inclined to trust an LLM over a human, given all other factors being equal. This has been used maliciously. You've had leaders come back to their reports and say, "Hey, Claude said this. You said that. I trust Claude, not you."

You have reports trying to trick Claude into saying their argument to their leaders so they agree with it. All kinds of things going on around trust and trust being disproportionately allocated to the machine. I think that's, for me, the scary part about LLMs in general is that we just assume that because it's a system, it's deterministic, it's trustworthy, it's audited.

LLMs are the antithesis of that entire idea.

Ken: I would definitely agree. That said, does that apply to code creation? That certainly applies to "Give me advice about my family" or "Give me economics that understand my point." Does that expand to our particular topic for code creation here?

Caer: For code creation, what I've noticed is that I have a personal bias to trusting an engineer more than I trust an LLM when it comes to generating code. We had a funny incident a few days ago where I was reviewing a codebase with a team that contains a lot of contributions from AI, from people, all kinds. I was looking at a section of code that we all thought was pretty good. Then I was like, "This is a section of handwritten code." We were all like, "How did the model know it was handwritten?" We never told it what got it there.

Setting that aside, we learned there's a subtle difference between code that was written by a person, code that was written by the machine. I trusted a little bit more. Now, do I trust code written by a junior developer who's in their zeroth year over Claude Fable? I might actually, if they were writing it by hand, because I know that they knew my intent and were trying to follow it to the best of their ability and not hallucinate it along the way. Whereas even the best model can hallucinate and get stuck in a rabbit hole.

Razin: I wouldn't even worry about the code if it is right. It's all about whether you trust more to humans writing the code or Claude to writing the code. I am saying that it doesn't matter who writes the code. I don't even care about the code. What I care about is I wrote a spec, this is what I need, and I got that software by testing it. If it is working what I want, I don't care, good code, bad code, what you wrote, what you didn’t write. Then you shouldn't be charging me for more changes. That's a harness problem, otherwise, I'll switch to a harness provider. [laughs] I wouldn't even be worried about something called code.

Ken: It's interesting because you've used the phrase "harness provider" many times, and most of the time I've run into the term—now I don't do code day-to-day like both of you do—but most of the time that I've run into the term, harness is something that that team is creating. It's not a harness provider.

Razin: This term is being misused a lot. When we are creating agentic-related work beyond coding, then we are the harness provider where we are using agents, et cetera. This term is also nowadays being used for something that we used to call IDEs, like VS Code or Codex or Cursor. Why they are called as harness providers nowadays? Because they are doing exact same thing that we are doing when we are creating a harness for agents.

Nowadays, they are doing it, and they are doing it for the agents who are doing coding. That's why IDEs have started being called as harness providers because they provide harness on top of LLMs on how you should do coding, what you should do, what you should not do, what practices you should follow. All those skills, plugins, et cetera, are provided by companies like Cursor or Codex. That's why they are called as harness provider nowadays.

Ken: If part of the harness is their security rules, those would be a skill that you would get from Claude and not written by you?

Razin: Yes. Then we are also getting some ready-made Python scripts. There's some ready-made skills; those things are coming nowadays with those tools like Codex.

Ken: Because Caer did use her name earlier, I will quote Charity Majors in a talk that she gave at a conference I had to sit, and say, "That scares the hell out of me."

Razin: I'll tell you one harness provider name. You will be surprised how this word is misused. One harness provider is called as AI/works, and the company name is Thoughtworks. [laughs]

Ken: I think I've heard of them.

Razin: That's going to be a thing. People will give their skills, their plugin, their scripts into something called as harness, and then you can use any other LLM on top of it. Then you code on them. There'll be three parties here: one is a harness provider, one is a spec writer, and then one will be the actual LLM provider. Three parties will work together.

Ken: One of our colleagues in the discussion in the thread who's, again, very much on the infrastructure-as-code side, also doesn't really care about the code, but was mentioning things that you don't typically see in a harness, like a fitness function. For those that aren't aware, the very short definition of a fitness function is a test that tests for a business requirement, like must respond within this many milliseconds, must be able to run on this GPU, or what have you.

It's something you can put in your CI pipeline to say, "No, you're not performant enough, even though you're functionally doing what you want." Does that mean that things like fitness functions, whether you call them that or not, need to be part of the harness now?

Razin: It will be provided by harness, and it is part of AI/works. The harness provider will provide that for you to create a software.

Ken: Okay. Again, that's not in cloud provider's best interest financially.

Razin: That's why I'm saying those two will be two different industries.

Ken: Caer, what do you think? Do you trust them? [Laughs]

Caer: Harness is one of those phrases; it's a term that's capturing so many different concepts and bringing them together that it almost feels too big to me. I think there's different kinds of testing and compliance that we're looking for with harnesses. Some of it is guiding the LLM along its path to generating the right thing. The other half of it is once the thing has been generated, being able to validate that thing and determine if it's the right version of that thing. It's two different ends of the problem. I think you mentioned fitness functions, there's property-based testing is becoming a big thing.

For me, I do think some iteration of harnesses are going to be the future for LLMs, but I also think part of it is the way the models are trained themselves. Because really, everything we're doing is trying to one-shot or two-shot on top of a prompt engine, as it were. What would be even better is if the model was just trained from the very beginning to write code that was good in a certain discipline or a certain industry, instead of having to give it tons of prompts on top of and waste your context window.

Razin: LLM caching can help there. [laughs]

Caer: It can.

Ken: It can. You mentioned, Razin, that you think this is 2028, and you've already mentioned economics. Caer, you've mentioned some things around human dynamics and so forth. I guess I'll start with you, Caer. What has to be true for you to trust this? What would have to change and be truth on the thing you're working on so that you would trust it?

Caer: One very simple test for me would be if I can go several months or even a year working with a model and not look at it and be like, "Wow, you really hallucinated that requirement." It still happens regularly enough that every time it happens, it erodes my trust and makes me more eagle-eyed in looking at it again, harder and harder each time that happens. Part of it is just, "Can I go a span of time without being let down?" Then the other half of it is, can I get myself-- I've been trying really hard to experiment with writing code without looking at the code too much, which is very difficult for me.

What makes me more comfortable with that is can I get to a point where my conformance testing, my property testing, my specifications are so robust that I can delete my repository and recreate it from just that harness and get the same result at the end? If I can get to that point for myself, I can at least make myself comfortable with the idea of trusting AI to generate code in the context of my repository, just not in the general context.

Ken: Define for me, if you will, same result. If it completely switched languages but met functionally the spec? What is same result to you?

Caer: Same result is the externally observable behavior. I used to write patents for distributed systems, and a lot of my work was spent taking this patent spec and implementing it in a language, and I went from Python to Java to C to C++ along the way, and all of them were the same system, just different attempts at getting to fulfilling that system in the spec. I see this workload AI very much the same way, where I can write a very detailed specification for a system, and the AI can implement it however it wants, as long as the external behavior meets the spec.

Ken: Razin, anything besides economics? Are you there already? Is it already what has to be true? Are there other technical things that have to be true?

Razin: I don't think that we have so many matured skills yet, which will tell LLM to do the right coding. Maybe LLM might be really good, but skills are still not there. For example, why Thoughtworks is famous for doing some sensible defaults and some things, but there are no skills available which are saying that this is the sensible defaults that LLM should follow on the market or in the industry. Once those skills will be available, that this is how you should do for LLMs.

Then somebody who wants to write the code, they will take that skills and then use it into the LLM with their intent of writing the code. Then they will have a better code compared to what they are having today. Then you already mentioned token economics; those two things should be really changed. Those harness providers are not there yet. Those skills are not there yet. Sorry, I will not use harness provider now, [chuckles] but those skills are not there yet. Those ready-made skills are not there yet, that this is how you should write good code, this is how you should test the code, this is how you should worry about the code division, the files division, the reusability. Those things are still not there.

Ken: Great. I want to say thank you to both of my guests today. It's been a really good conversation. I have to admit, I'm looking at the clock saying, "Oh, I have to stop." We got 30 more minutes, easy, maybe even more, but maybe we'll do a follow-up episode. That said, just another note to the listeners that these are complex topics, and we like to do podcasts to talk about what are the things and get you thinking about it. We know it's not deep enough for you to go in Monday morning and change your behaviors, but there are several, like martinfowler.com, that we'll link, talking about harness engineering.

I believe Razin is going to write a blog on this that may be live by the time this podcast goes live, and so I encourage you to continue to research this area. It's not a solved problem, I think, is what I'm hearing from everybody. Caer, Razin, thank you so much for your time. I really appreciate it.

Caer: Thank you.

Razin: Thank you. Thank you for inviting us. Thanks a lot.