Refactoring with AI

Episode transcript

Rebecca Parsons: Hello, everyone. My name is Rebecca Parsons. I'm the Chief Technology Officer Emerita for Thoughtworks. I'm joined today with three guests to talk about the topic of AI refactoring. First, I'd like to introduce Adam, who is the founder of CodeScene. Adam, please say hello to everybody.

Adam Tornhill: Hello, everybody. Really good to be here.

Rebecca: And Birgitta, a Principal Technologist with Thoughtworks, who is leading our efforts in AI-assisted software delivery. Birgitta?

Birgitta Böckeler: Hi, everybody.

Rebecca: Martin Fowler, our Chief Scientist, author of the book Refactoring. We are here to discuss AI refactoring in all its glory. One of the things that had disturbed me very early in many of the studies that were done on the power of tools like GitHub Copilot was, they focused on speed, and they focused on task completion. No one ever said anything about code quality, and nobody ever said anything about, and, "Oh, by the way, how is this going to help us with all of the legacy code that we have?" That's the topic that we are going to discuss today. Let's start with the question, how do we characterize code quality? What does it mean for code to be good code, or bad code?

Adam: The interesting thing with code quality is that there's not a single metric that can capture a multifaceted concept like that. I mean, no matter what metric you come up with, there's always a way around it, or there's always a counter case. What we've been working on for the past six to seven years, is to develop the code health metric.

The idea with code health is that, instead of looking at a single metric, you look at a bunch of metrics that complement each other. What we did was that, we looked at 25 metrics that we know from research that they correlate with an increased challenge in understanding the code. The code becomes harder to understand.

What we do is, basically, we take these metrics, there are 25 of them, stick them as probes into the code, and pull them out and see what did we find. Then, you can always, weigh these different metrics together, and you can categorize code as being either healthy, or unhealthy. My experience is definitely that when code is unhealthy, when it's of poor quality, it's never a single thing. It's always a combination of factors.

Birgitta: It's interesting what you said that the metrics that you chose like one of the main factors you use to choose these metrics is, do they improve readability, yes?

Adam: Yes, definitely. The reason for that is, and that's also what I think is interesting with what Rebecca said initially, that today we are focusing AI a lot on improving throughput, shipping more code faster. If we look at the studies on what developers actually do, what we see is that writing new code is a very, very small part of our day job — it's like 5% in the latest study I saw. That's a couple of hours a week. Whereas understanding existing code, that's where you spend the majority of our day. It's like 60% to 70% of our work week trying to understand the existing system. That's where I think readability to me is super important.

Martin Fowler: I would like to throw in here the point. To me, a healthy codebase is a codebase where people can more easily work in it to make changes, and evolve that code base. I focus on starting from that outcome end of it. Then, it is appropriate then to look at things like readability, because that's contributing to that ability to change in there.

I've always believed that readability of a codebase is key to being able to make changes quickly, because you need to be able to understand the code, in order to change it effectively. I'm 100% with the idea that readability is key, but we have to remember that the driving force here is this ability to make changes rapidly and safely.

Birgitta: We haven't even talked about just correctness, right? I guess, Adam, that's not something that can be determined by code health metrics, because the developer has to determine if it's what they actually wanted, and that can only be checked with tests, I guess, right?

Adam: Yes, that's definitely true. That said, what we see in the studies we have done is that, there is a pretty strong correlation between unhealthy code and defects. I think it's no wonder, I mean, changing existing code might be one of the hardest things that humanity ever attempted. If that code is overly complicated, full of accidental complexity, we're making our job much harder. I'm not surprised about those correlations.

Birgitta: It's a vicious cycle at some point, like a spiral.

Rebecca: It's interesting you bring up accidental complexity, because we actually did quite an interesting study several years ago. Thoughtworks, since its beginning, had been doing code submissions on a relatively small number of problems. We analyzed it using one of the code metrics that existed at the time, which again, much like what you're talking about with code health, was a composite metric.

It was developed by Eric Doernenberg called Toxicity. One of the things that we noticed is that, there was a barrier that, if your code was simpler than this particular barrier for a particular problem, it was wrong. That you need a certain level of complexity to actually be able to get it right, which is why it's silly when people say, "What's the minimum level or the maximum level of blah…?"

You can't really answer that because for some problems, have to have a certain amount of complexity. That essential complexity really does exist, and we can't get away from it. I think that's where, again, what Birgitta mentioned, in terms of testing, we need to make sure we have a test harness that is robust enough to allow us to ensure that we have addressed the essential complexity.

Let's move on to why should a business care about code quality. As a former university professor, I still have something of an academic bent, and I would like to say that there would never be any bad code anywhere on the planet, and we all know that that's a fantasy. How do we talk about the business value of code quality?

Martin: I would bring that-- This is back to that outcome point that I made earlier on. If people can make changes more rapidly in the codebase, then they can implement new features more quickly, they can make changes to the software. If any problems do come up, they can fix them. All of this is much faster when you've got a high code base quality, and therefore, it's better for supporting businesses, particularly, as they need to change rapidly to deal with changes in market conditions.

Birgitta: Responsiveness, right?

Martin: Yes.

Adam: I very much agree with that. In fact, me and my research colleague, Dr. Markus Borg, we published a study, I think it's a year and a half ago, called Code Red: The Business Impact of Code Quality. What we did was that, we wanted to prove that scientifically that there is this relationship between healthy code, and the ability to change and add new features and fix bugs more rapidly. And what we found was quite dramatic.

It's like on average, if you have a healthy code base, you can move more than twice as quick. You can decrease the number of defects that you ship into production with a factor of 15. That's quite a business impact. It means that if you have an unhealthy code base, and we have a healthy one, we can implement whatever capability you have, twice as quick as you can. That's the bottom line to any business.

Birgitta: I think it's really important to have studies like that, because I think it's not so much about the question like, why does it matter, or does it matter? Because I think most people would say, "Of course, it matters," right? Of course, code quality matters, but still we see people take decisions every day [chuckles] in software delivery context, where they're trading off these types of things like readability, maintainability, all of that for other more short-term concerns.

I think showing how much it matters, can really maybe help move that needle. If people really see the dimensions of what you have in your study, for example, might really help you if you think, "Yes, but in this case, it doesn't matter as much" If you have those numbers in the back of your mind, it might help you take more responsible decisions in the moment.

Adam: Yes. I could add to that. In addition to that, another scenario that I personally really like with it, is that having this type of data makes it possible to actually build something like a business case for refactorings, and larger improvements. Because these can now come with a business expectations that you're going to become not only quicker, but also better. You can actually quantify and measure that, and that's something I think is important.

Rebecca: You mentioned refactoring there, and this is something that I know you've done some studies on as well. As you mentioned earlier, Adam, it's actually a very small percentage of a developer's time that is spent on actually writing new code, and refactoring existing code is one aspect of a developer's normal workflow. Can you tell us a little bit about what you've been learning around using AI tools in support of refactoring, and what kinds of things that you've learned so far?

Adam: Sure. I'd be happy to. Basically, the challenge we were looking at was that, we have a tool that is capable of identifying bad code, prioritizing it, but then of course, you need to act on that data. You need to do something with the code, improve it. This is a really hard task, so we thought that maybe generative AI can help us with that. What we started out with was a data lake with more than 100,000 examples of poor code. We also had a ground truth, because we had unit tests that covered all these code samples.

We knew that the code does, at least, what the test says it does. What we then did was that we benchmarked a bunch of AI service as like Open AI, Google, LLaMA from Facebook, and instructed them to refactor the code. What we found was quite dramatic, in 30% of the cases, the AI failed to improve the code. Its code health didn't improve, it just wrote the code in a different way, but the biggest drop off was when it came to correctness, because in two-thirds of the cases, the AI actually broke the tests, meaning, it's not the refactoring, it has actually changed the behavior. I find it a little bit depressing that in two-thirds of the cases, the AI won't be able to refactor the code.

Birgitta: Now, what types of ways did it break the behavior?

Adam: In super subtle ways. That's the tricky thing. Maybe we didn't make our life as easy as we could, because we chose to start out with JavaScript, which is a notoriously tricky language, but it could do really, really odd things like moving a "this" reference to an extracted function, which would alter its meaning, sometimes it removed entire branches, sometimes it chose to negate the boolean expression, and these things that are really, really hard for human to track down and spot.

Birgitta: There's a little line with a dot, like an exclamation point that is gone, or in the wrong place, it's hard to see, right? I get now what you meant about, like it's trickier with JavaScript, and then you mentioned this as an example. It's notoriously prone to bugs in the way that this is being used. 

Can you repeat that rate again? In how many percent of the cases did it succeed?

Adam: The best AI service performed correct refactoring in 37% of cases.

Birgitta: 37%. In the other cases, in the other-- Doing some quick math here, 63% of the cases, [chuckles] it either broke the behavior, or it didn't break the behavior, but it also didn't make the health better. You were using your health metrics to check if it was actually making the smell go away.

Adam: Exactly. Because to me it feels strange to say this when Martin is on the call, but to me a refactoring, it's not refactoring, unless it improves the design of the code. Am I right, Martin?

Martin: Mostly, yes. The reason I say, "Mostly", is it's the intention that matters, so that there are times when you'll make a refactoring that actually makes the code a little bit worse for the moment, but because of the change you are about to make, it'll make that change easier. Again, refactoring is often a somewhat directed process. You're not just saying, "Oh, I want to wave hands over the code to make it better." Although sometimes you do.

Sometimes you say, "Oh, I've got a change coming up. The code isn't in quite the right shape for that change. It's perfectly okay as it is, but not for the change I'm about to make, so I need to refactor the code, so that the change I'm about to make is going to be better." Kent Beck has a lovely quote about this. It's “when you are going to make a change, first make the change easy — warning, this may be hard — then make the easy change.” I think that's the case that we have to bear in mind there, but certainly, always the intention of a refactoring, should be to make the code better.

Birgitta: Yes, and I would say if we-- We're generally talking about, can AI not only help us write new code, but can it also help us improve code, or make code quality better? It's also been my experience that you can ask a coding assistant, most of the products you can ask, "What's wrong with this code, or what can I improve about this code?" It gives me some suggestions, and sometimes they help me but, for me as a more experienced programmer, it's often very basic stuff. Like, "Oh, the error handling is still missing." I haven't actually had impressive results yet with “how can I improve this?”

There was also a really widely shared study recently from GitClear that was called Coding on Copilot. Data Suggests Downward Pressure on Code Quality. What they were talking about a lot was how they had some data that they interpreted as: codebases that are partially written with AI, grow faster than our code bases used to grow previously. Suggesting that, because now it's easier to create new code than to change existing code, maybe we just take the easier road sometimes, and just let Copilot write more code for us, or a coding assistant write more code for us, instead of refactoring the existing code. I think you've also read that study, Adam, right? What did you think about that one?

Adam: Yes, I read that study, and I think it's interesting, because the conclusions, it's actually pretty much what I would expect from widespread mainstream AI adoption, because AI now makes it so easy to write a lot of code that shouldn't be written in the first place. I do agree with that, but what I find in that study is that, I don't really think the data would support the conclusions they make. And I think one of the main limitations is that they simply don't know which code was AI generated, and which wasn't, so I think we have to be a bit careful in interpreting this.

Birgitta: I think it was a blanket conclusion from just saying, "Oh, this is about the time when AI coding systems became really popular," and then looking at all of these repositories and taking this conclusion, but I also agree that the hypothesis totally makes sense to me.

Rebecca: When we think about understandability, obviously, code quality is crucially important, but what are the approaches? You all are faced with an existing codebase that you don't know anything about, and now we have these great AI tools that are supposed to help us, and ultimately, take away all of our jobs, [chuckles] which I don't believe is going to happen anytime soon, but how can these tools help in this critical aspect of software development, which is understanding an existing codebase?

Birgitta: Yes, I think for the understanding a codebase, I've actually seen some quite promising first features like this that I think are not at the scale yet, where maybe where it really makes it easier to navigate, but there's a lot of combination going on of large language models with other technology that actually understands the structure of the code, because the large language model doesn't really understand the code — it only understands the tokens, or it only knows the tokens.

In GitHub Copilot, for example, you can, these days, now also ask your workspace, your code base a question. They're also working on combining that with your other GitHub repositories that you're hosting on GitHub. There are a bunch of products and startups that are working on specifically helping you navigate code, and understanding it and asking it questions. I've tried a few of these tools, and sometimes it works, sometimes it doesn't.

It's sometimes not the perfect answer, but it's still, even if it's not perfect, it helps me understand things quicker in spite of that, interestingly. Even if I sometimes see, "Oh, this is not actually the case," it still points me in a direction and helps me navigate things faster, and understand them faster. So I think there's actually a lot of potential there for the pure understanding part.

But at the same time, large language models can also only be as good as the code that they're trying to explain. If you have a codebase full of variables that are just like X, Y, Z and A, B, C, then a large language model, the same as a human, [chuckles] will not know what the domain behind this is.

Adam: It's also my experience. Doing this benchmarking study, we have so much data that we haven't published yet. One of the things we see as well is that these large language models, they do seem to break down pretty quickly when it comes to understanding code. I think it's particularly concerning for legacy code, because one of my experiences is that if you look at the codebases that would benefit the most from these techniques, they tend to have extremely long functions: functions that are 500, 600 lines of code, it's not that uncommon. You stuff that into a large language model, and it will break down, guaranteed.

Birgitta: I think, especially, when there's like a lot of cyclomatic complexity or deep nesting, I think the architecture of the large language models can't really deal with this deep nesting, yes.

Adam: Yes, I think it will improve rapidly, but we are not there yet in my experience.

Martin: Is there any work at trying to do this kind of stuff with these kinds of models that operate on the level of the abstract syntax tree, rather than the code text itself?

Adam: We actually do a bit of both. When we did a benchmarking study, we operated at the raw text, but when we did other innovations on top of the AI, where we use startup machine learning models, then we found that using the abstract syntax tree is much, much better because in many, many cases, you can actually erase the differences between different languages. You can have one model that understands multiple languages, as long as they're in part of the same language family, same paradigm.

Birgitta: In terms of your study, so you found this result that only in, what was it, 36%, 37% of the cases, the refactorings were actually good. Then, you went the next step. Something that I see in a lot of places at the moment as well, where people are saying, "Okay, these results are not perfect, but can we put a next step behind these suggestions that we get from these tools, and assess our confidence in this, and maybe not even show the results to the human that we can automatically determine that we have very low confidence that they're actually correct." You all also did that, right? You went the next step, and tried to filter out the ones that you could determine with automatically that they're probably not correct.

Adam: Yes, that's right. Because at first, 37% correctness, it's pretty depressing. When we looked closer at the data, we saw a couple of interesting patterns. We saw that the different AI services, they seem to be good at different things. We tried to use that to our advantage. Instead of going directly to, let's say, OpenAI or Google, what we do is we go to something we call a model selector, which is basically a pretty simple machine-learning algorithm that has been trained to understand that, this type of code, pick OpenAI, this other type of problem, pick the Google AI.

Already that improved that success rate. Then, we also made sure to add some fact-checking on top of the AI response. We fact-check it to make sure it improves the code health, but it doesn't do anything strange like removing entire branches that we don't expect to be removed. If we find that any of these fact-checking tests fail, then we throw away the result, and ask another AI for a new one. Once we get something that passes our checks, we can safely pass it on to a user that can inspect the code, and apply it if necessary.

Rebecca: Did you find that there were patterns and instances of where you didn't ever get something that you found acceptable?

Adam: We never published that, but there were definitely some patterns that the AI seemed to be really challenged with. It's usually about the things that you don't really want to do as a human programmer anyway, but so many code bases are full of it, like continuous statements inside deep loops, the AI really struggle with that, for example.

Birgitta: Something I learned from reading your paper is that when you were doing this, you had to determine, does this change the behavior? Yes, or no? Does the behavior change compared to the previous state of the code? I learned from your paper that, that's actually not a solved problem. You had to go back to your data lake, and use all of your previous experience with analyzing code bases to make this possible, right?

Adam: Yes, that's right. When we make this claim that we can guarantee that the code is semantically equivalent, it sounds like we have solved the halting problem, which would have been cool, but we didn't. Yes, we had a benefit of lots and lots of data with a known ground truth is the refactoring correct or not. That helps a ton, of course, in building this algorithm.

It's also the case that we could constrain the problem, instead of doing a general semantic equivalence check of all types of code, we could constrain it to these specific refactorings, and we could even constrain it even deeper, because we knew that this is a specific code smell that we have instructed AI to refactor in a certain way. That makes it much, much easier to fact-check the output.

Birgitta: Also, the context in the CodeScene tool that it had, like what code smell it is about and so on, you do use all of that information to do your confidence assessment?

Adam: Yes, that's correct.

Rebecca: Shades of my work with compiler optimization, oh, so many years ago. [chuckles] Based on all of what you've seen, what would you recommend for developers getting into using these coding assistants, in terms of mitigating the risk of generating bad code, or using refactorings that are subtly wrong? Is there any guidance for people who are starting to use these tools on how they can use your work to mitigate against the risks that you've uncovered?

Adam: I do think that traditional tools like linters are super useful together with AI tools. I also think it's obviously important as well to keep an eye on the code quality, so I do recommend having something that can give you feedback on code quality, where we're all on as you write the code. Then, the last thing I want to point out, and I think this is more about more of a concern for the team, but that is the whole code familiarity aspect, because the quicker we're able to generate new code, the harder it is for the team to understand that code. I have found as well in the studies we have done that unfamiliar code, no matter how healthy it is, it's still going to be a challenge for you, it's going to be a barrier to entry.

Birgitta: Yes, and a lot of people are asking this question these days, like, "Oh, how do I know if using a coding assistant, or using any gen AI for any of my tasks and software delivery, how do I know that it's actually worth it? How do I measure this?" For me, it's still the same way that we measured the outcomes before, we just have new tools now, but there is a perspective on this that when you're using AI for this, especially monitoring the risk, there's certain things that we can maybe now specifically monitor that maybe before we didn't even look at as much.

Not only our common code quality metrics, but based on the study, or this hypothesis, maybe I want to look at the growth rate of my code base. Maybe I've already maintained it for a year or two, and I have historical data about the growth rate before, and now maybe I look at it, and of course putting into context, like how many features are we working on these days? What are we actually doing?

It always has to be put into context. Maybe I can actually look at this growth metric, or I can look at like, is the size of my change sets to production actually getting a lot bigger, because whenever-- The commits are getting bigger because I'm writing more code every time I use Copilot?

Rebecca: Maybe even things like copy-paste detection, keeping track of some of those that's-- At least, one of the studies that I read, that they found that there was more copy-pasting going on as a result of-- Not necessarily as a result, but there was a correlation, not necessarily causation with the use of the coding assistant, and an increase in copy paste.

Birgitta: Then, also what you said, Adam, about the familiarity, that's, I think, again, maybe like a qualitative survey, not something that one can measure with a tool, but really asking developers, "Do you feel like your debugging time is going up?" When you have an incident, and nobody really understands the code. Again, these more qualitative surveys about developer, and debugging experience and stuff like that.

Rebecca: Then, of course, there is the perennial issue of what does developer productivity mean anyway? We won't go off on that tangent at the moment, [laughs] I don't think. I do have one other question. Again, Birgitta and Adam, you've both spent a lot of time with this. As we use these tools more, as more of our code bases have been touched by these tools, do you see fundamental changes in what it means to be a software developer?

Are we going to worry more about different kinds of things than we did before? Can you talk a little bit about, what you think the things are going to look like with respect to these coding assistance? Obviously, the models-- Not obviously, but likely, the models will continue to improve. What about the whole day in the life of a developer? How might that-- How do you think that might evolve?

Adam: I think that AI coding is less overhyped to date than it was maybe a year ago. A year ago we still heard about the end of programming, and all of that, and I don't think anyone realistically believes that today. However, it's very clear that we're only seeing the beginning of this trend. Yes, the models are going to improve, the tooling is going to improve, and I think that we're entering our challenging phase, because over the next decades we will have this hybrid model, where humans need to understand code that's developer and AI.

I think it's really, really challenging. What I think that means is that, as organizations that build products, that develop software, I think it's going to mean that we need to reemphasize the importance of all these basic engineering principles like good test automation, healthy code. Maybe we even see a renaissance in the importance of code reviews, as a way of maintaining familiarity. I think a lot of these old techniques are going to become even more important today in our AI assisted world.

Birgitta: Yes, I like what you said about the hybrid mode. At least, if I understand you correctly, it's like as long as we still need multiple humans collaborating on code, and we don't have that one AI that does everything, as long as we're not like 100% or almost a 100% there, still a lot of the things that we're doing today about collaboration and reading code and all of that will matter to some extent.

Yes, I agree with that. I also think it's here to stay, and it will get better. Even though there are a lot of risks, and flaws, and things that don't work, maybe some of them don't work yet, some of them will never work. Yes, it's like for us as a profession now to figure out how to use this responsibly. For me, my favorite example is test-driven development. I still think it's relevant, but maybe it will change in the shape that we're doing it. Because a coding assistant generating tests, is just too tempting for people to not do it.

If you then generate your tests and your code, and you don't even check if your tests are effective, then you're going down a really slippery slope. How does TDD change? We still get the benefits, but we also maybe use an AI to help us write those tests faster, maybe for the edge cases. That would be an example where maybe one of the practices I think that is most immediately affected, like even today, and not only in like a year or two or longer.

Rebecca: Martin, you've clearly thought a lot about TDD. What are your views on how you think this is going to impact a practice that has been so central to XP?

Martin: I'm, again interested not so much in the AI tools generating code, as the AI tools telling me things about the code base. Particularly, with a code base that I'd need to know. On the TDD prospect, I'm immediately interested in are there areas of the code base that are very well covered by tests? What are the kinds of changes that could go.

Again, it's like the mutation testing problem. Are there parts of the code base that we could introduce a bug without the tests detecting that the buggers been introduced?

I would like the AI to find that kind of thing out for me. It might also look at areas where I might have tests that are overly detailed in the internal implementations and say, "Can they be replaced with tests that are less dependent on that," particularly if it's an area where I'm looking at changing some of those internal cases. That kind of thing is interesting to me.

Birgitta: All my labs and tests, right? Where are things that I'm testing over and over again on each level of the pyramid.

Martin: Yes. Or even in less obvious cases where it's even the same place, but because I wrote tests in different cases, in different parts of the system. I'm interested in AIs that will just tell me things about, "Hey, I need to change this particular kind of output somewhere. Where's the code that figures all this out? Where is this being touched through the code base?"

Even simple things of, where-- I'm hitting this database table. Tell me all the bits of the code that contribute to hitting this database table or not. A lot of this we can do with searches, but I suspect with an intelligent tool that's-- I don't want to say trained on the code base, but has ingested the code base, hopefully, would be able to answer much more interesting questions than I can currently do by throwing Regex at the code base at the moment.

Birgitta: I think finally it's-- I think also worth pointing out that all of these things that you just mentioned, we don't necessarily have to only turn to generative AI just because it's the hype of the moment to do this. Adam, I thinking at CodeScene, you've used machine learning before all of this this happened to suggest refactorings and all of that. Do you also see a resurgence now in interest of using AI to improve code health, even beyond large language models?

Adam: I do. In particular what I see is on importance in data in having not only enough data, but the right type of data. Because all these algorithms are extremely data hungry.

Birgitta: Yes. In this case, the code is the data of course. Right?

Adam: Yes.

Rebecca: I'd like to thank you all, Adam, Birgitta and Martin, for joining me today with this discussion of AI refactoring and code health, code quality, and how our lives as developers might be changing, and all of the both exciting potential, and very real risks that we will be facing. Thank you, Adam. Thank you, Birgitta, and thank you Martin.

Birgitta: Yes, thanks. I got a bunch of new ideas of what I will try to ask my coding assistant tomorrow.

Rebecca: [chuckles] Excellent!