Continuous delivery in the wild

Podcast transcript

Alexei Boas:

Welcome to the Thoughtworks Technology Podcast. My name is Alexei. I'm the head of technology for Thoughtworks Brazil and I will be one of your hosts this time together with Mike Mason. Hello Mike.

Mike Mason:

Hi, Alexei. Happy to be here. My name's Mike, I'm also one of the regular podcast hosts.

Alexei Boas:

And we're delighted to have Pete Hodgson with us this time. Pete, would you mind introducing yourself a little bit to our listeners?

Pete Hodgson:

Sure. Hello, hello. My name is Pete and I'm a software delivery consultant. I used to be, well, I still am a software engineer and I spent some time at startups and then I spent some time at this global consulting company called Thoughtworks and then I went back to startups for a little bit and now I'm doing independent consulting. And a lot of what I do is helping teams get better at delivering software. Sometimes that's architecture and stuff like that, but a lot of times it's also around process and how to get stuff into production on a good cadence.

Mike Mason:

Pete is a good friend of Thoughtworks, used to work with him very closely, obviously quite aligned with, I would say, Thoughtworks worldview on things and the right way to build software. And today's episode is called, Continuous Delivery in the Wild, and we're going to be talking to Pete about a recent ebook that he's written for O'Reilly talking about continuous delivery. Pete, why don't you tell us a little bit about the premise of this Continuous Delivery in the Wild book that you've written?

Pete Hodgson:

I spent quite a lot of time talking to people about continuous delivery and hearing thought leaders and stuff like that talking about continuous delivery. And what I notice is people spend a lot of time talking about the cutting edge stuff like really advanced engineering orgs are doing, which I think is great. But I don't really hear that much being talked about about just companies that are just quietly doing this stuff well and they might not necessarily be doing the most amazing, cutting edge practices, but they're succeeding really well and they're getting the value out of these continuous delivery principles without necessarily doing all of the whizzy, super fancy stuff that you'd go to a conference talk to hear about, although maybe you should be going to a conference talk to her about the successful stuff.

The premise was I want to get a real talk, not conference talks. And so, a company called Split sponsored this little project and I went in and found a bunch of organizations that were succeeding in continuous delivery by my own definition where they were releasing to production at least once a day, often more than that. That was the criteria, but besides that, that criteria, the people I talked to were quite varied. They were like big enterprise-y companies, little start ups, companies that have been around since the fifties, companies that have been around since last year. I got a really wide range of companies, but all of them, this unifying thing where they're all doing pretty well with CD. And I just basically spent some time with a bunch of the teams asking them how they did CD or how they did software delivery.

Mike Mason:

Let's start with that definition of CD because I think that's worth pushing on a little bit because I think a lot of people would define it in different ways. Your definition for this is organizations that are able to deploy to production daily.

Pete Hodgson:

That was my criteria for companies that I feel like are succeeding with continuous delivery to some extent. The way I would define continuous delivery, and whenever I'm around Thoughtworks people I always get a little bit more nervous about sticking my neck out, my working definition continuous delivery is that you're able to release code from your main line branch or master or your trunk or whatever, wherever you release into production, you feel confident that at any time you could deploy from that code from master to production. That's, to me, is the central thing. And there's a bunch of like principles and practices that are mixed in there, but I think that's the definitional thing for me, I think, for continuous delivery.

Mike Mason:

I'm not an expert, but I feel like when Jez and Dave were working on the original book, one of the key things was to make the IT department not a blocker for getting a deployment out of the door. It was to make deployment or do a delivery of software to be a business decision rather than one that was mired in three months of getting it through the test environment or whatever. In a lot of ways it doesn't even need to be once a day, it just needs to be at will, you could do it.

Pete Hodgson:

Absolutely. Yeah, no, I think that's totally true. There isn't anything, definitely not as far as I'm aware, in that original book. And I think really in the principles of continuous delivery it's not about you must be able to deliver every X days. Particularly when it was first becoming popular, I think a lot of it was about exactly what you're saying to giving power back to the product owner or whatever to release software at the pace that felt right.

And a lot of the research that Nicole and Jez and Gene Kim did with that amazing Accelerate book and the DevOps report, I think what has emerged is there's some value in giving control back to product people. But I think there's this really like interesting, maybe it was originally understood problem, maybe it was by smart people, it's not intuitive to me, but there's this really interesting thing that by compressing that feedback cycle, by allowing yourself to release more frequently, you end up releasing more frequently and then you get these really interesting effects around reducing batch size and smaller changes going out. And nowadays. I think there's some confusion or at least there's some stuff gets mixed up in a lot of people's brains between continuous delivery and continuous deployment and this idea of frequent releases, et cetera, et cetera. And I think that's because people who are doing continuous delivery and have the capability to release whenever they want realize that using that capability to release a lot is really, really useful. And so they do it.

And so people see, oh, people who are doing continuous delivery are releasing all the time. Therefore, continuous delivery is releasing all the time. And I don't think that's exactly right. I think it's actually about the capability to do that. And what ends up happening is most organizations do that because it's super valuable to be able to have small, rapid flow or a frequent flow of small changes into production. It's super, super valuable. And that was definitely one of the things that when I was talking to these organizations as part of this book, the super common theme was the value in reducing batch sizes.

Alexei Boas:

Cool. Well, that's very useful. Thanks. You did mention different companies you talked and that seems interesting to see all these patterns emerging in several different contexts and contrasting contexts. Were any patterns that you found over that research, is there anything surprising or anything that you feel like sharing?

Pete Hodgson:

Yeah, I think what was interesting to me, broadly what was really interesting, was that there was these core principles that lots of organizations shared, but then the way they applied those principles were quite varied. The key thing that I just kept on hearing was around this thing about reducing batch size, which essentially means trying to get into a cadence of releasing frequently or deploying frequently. But the way that different types of organizations achieved that was really different. Taking that same principle of we want to be able to deploy frequently, but then looking at your architecture. Some of these organizations were two years old and when they started building their systems, everyone was already all excited about fine-grained SOA or microservices or whatever you want to call it. They have these fine-grained systems like loads of little independently deployable artifacts and other organizations for various reasons had these giant ginormous monoliths or these large-ish monoliths] that they're in the process of breaking down.

That difference in architecture was reflected by a difference in the way you do release management, for example. There was this pattern that I saw with more monolithic systems. And my definition for a monolith from this in terms of this research is like a monolith is something where a deployable artifact is owned by more than one team because I think that's actually the ergonomics. That captures what makes a monolith different to work with. People that were working in those architectures, it's not these practices that work with microservices don't make sense in the context of that.

Let me take a step back. If you've got a microservice-y architecture and each deploy glass factory is generally owned by a single team, maybe let's say seven engineers. It's actually feasible to just say an engineer writes the code and walks it all the way to prod, like literally they make the commit or a series of commits, they get signed off or whatever, it's ready to go, it passes quality checks. And that engineer walks those changes all the way to production. That's not really feasible to do with a monolithic system because changes are landing so quickly because there's so many people making changes to the same deployable artifact and the people making those changes are far away from you organizationally, like they're not on the same team, so you don't necessarily know what those changes are.

It's not feasible to just say like, "Oh, I'm just going to deploy the latest code." You can try it, but what I found from talking to these organizations is most people had realized at some point that's not feasible. Then you think like, well, how can we achieve the principles that we're going for here of releasing frequently, but in our context of we've got a large monolithic architecture and the pattern that a lot of teams who landed on is this idea of like a release bus. Essentially, a scheduled release that goes out, let's say, every day or twice a day or every couple of days, maybe.

I could go into the details of that specific pan, but it's interesting more broadly to realize that the thing that works for some other company X, just because they're succeeding with it doesn't mean it's going to necessarily work in your organization. Facebook, I think, is a really good example where Facebook have a ginormous monolith and they're really, really good at releasing frequently. But that doesn't mean that if you don't have a ginormous monolith, you should do it the way Facebook does. Facebook is doing great. They've got loads of great [inaudible 00:12:21], but if you've got like a small microservice architecture, you don't need to do a lot of the complicated stuff that they have to do because they've got however many thousands and bazillions of engineers working on the same code base. Context is king, I guess.

Alexei Boas:

Looks like focus on the outcomes that you want to get and then the principles that you want and not exactly on the specific practices that other companies are doing. They might not make sense to you.

Mike Mason:

Pete, I'd love to hear about people doing branching or not doing branching or doing trunk-based development or not. I did a couple of books on subversion, so I'm dating myself in terms of version control tooling. But I remember having to tell someone once, branches are not free. They cost you something in terms of maintaining them and so on. But I'm just curious, did any patterns arise around the use of branching or not?

Pete Hodgson:

Yeah. Well, Mike, now that we've got Git branches that are in fact free, you're obviously a dinosaur. Subversion is, I don't even know what that, obviously I'm being facetious. I think it's worse now because that idea, which I was being mean and sarcastic about, that idea of like branches are free is I think more prevalent because branching is super cheap. Merging is not. It's not the branch, branch is great. Whew. Branching is not the hard part and it never was. And Git definitely made branching easier and to some extent, made merging easier, but that's like a whole different series of podcasts probably.

One of the things I think is really interesting around CD is the relationship with CI, continuous delivery and continuous integration. Nowadays, again, a whole other podcast series about what does CI mean because a lot of people would say it's Jenkins or Travis or CircleCI, it's in the name. And the curmudgeons that came up with the term would say like, no, CI is a practice, it's about integrating on a daily basis. And so I've wandered for a while, do you need to do CI in order to do CD? If you define CI as I'm integrating my changes into a shared main line at least once a day, do I need to do that in order to be practicing CD? And I had this hypothesis that no, you can actually succeed with CD without doing true CI in terms of merging all your stuff, at least once a day.

And I think, at least with the companies I talked to, that was what I found, is most of them were doing branching. Most of them were doing some flavor of GitHub flow-ish. What I think a lot of people are quite familiar with modern branch management, where you've got like master and then you've got feature branches and you branch off of master, you do your work, you submit a pull request or a merge request, and then you land it on master and then eventually you deploy it to production. I think there's nuance and details in how people are doing that stuff, but a lot of the organizations I spoke to were doing some version of that.

A few of them were doing trunk based development and were having a great deal of success with it. And in general, the ones doing trunk based development, I would say, were doing better in general with continuous delivery and were able to release more frequently. They were getting to the point that they were getting to like single piece flow. Literally, I have a commit, and commit, I walk that maybe one line change all the way out to production, which is great. I didn't hear anyone saying like, yeah, I wish we weren't doing that or like we tried that and it didn't work. Everyone who's got to that point, I think they were happy they were there.

But there are a bunch of organizations that had feature branches that weren't around for more than a day, so they technically weren't doing CI, but they were doing okay. And there was a very common theme of we don't like long lived feature branches. There was a common theme of a few of the more well-established orgs or orgs that had a heritage that went back to the before CD practices said like, oh yeah, we've got a few teams that they have like shared team integration branches or whatever. And there was this universal thing of, but we're really trying to get away from that. There's a general thing of like, we don't like feature branches, we definitely don't like release branches and things like that.

But it wasn't like, oh gosh because we're doing feature branches, we can't do CD. There was a desire to keep them small, but interestingly, I didn't hear that many people saying we like to keep them less than a day. The trunk based development people, the people who were succeeding with trunk based development, were definitely doing CI and they're very happy doing it, but the ones that weren't quite there, we're still quite happy, which I think was really interesting.

Mike Mason:

Curious were there any other kind of, I don't know what you'd call it, like a CD cliche or a supposition about the way organizations work? Did you prove anything true to yourself or maybe more interestingly, did you disprove anything or discover something that was a bit of a realization to you?

Pete Hodgson:

There was a lot of stuff that was that reinforced existing things that I would have believed, like people doing trunk based development were doing great, generally were happier. A lot of the standard practices that I've seen in general people applying when they're trying to get to continuous delivery, so brunch by abstraction and feature flagging, all of those kinds of techniques were definitely in play. I think I was surprised that there was actually less focus on those branch by attraction techniques than I would have guessed. And I think that was generally because a lot of times when I saw not that much feature flagging going on, it was for younger startup-y orgs that were quite happy to just be blasting out features all the time and didn't really feel like they needed to put that much control on when a feature was rolled out. That was a little bit of surprise.

I think what was interesting to me that I didn't expect was that almost all of the organizations that were really doing well had invested a lot of time on what I would call a custom delivery platform. Something on top of the CI/CD infrastructure. They were using Travis or Circle or Concourse or Jenkins, I'm sure a lot of Jenkins, to do the nuts and bolts of a change happened in source control, I need to do some stuff now, like I need to push my artifacts somewhere. The nuts and bolts was being done in general by a CI/CD system, but there was this whole layer of stuff on top of that, that no one was using open source tooling or vendor tooling or no one was borrowing or buying that. Everyone had implemented custom tooling on top of that. And it was interesting because various orgs had done the same thing, but for their organization. Various orgs had hand rolled basically the same set of capabilities. It feels like there's a gap there around why are we all like reinventing this wheel, but I'm not sure-

Mike Mason:

I'm not sure the wheel is the same for everyone, though, because everyone has a different regulatory process and the department down the corridor that you've got to satisfy in your process and all that thing. I don't know.

Pete Hodgson:

Yeah. It's true. And it's like, everyone's fitting together a slightly different thing. Everyone's deploying to a different system or different set of systems. They've got a different tech stack. They've got, again, different regulatory things going on, different STLC is the development process is baked into the tooling. I don't know. It feels on one hand that's true, but then, I feel like we've probably said that in the past about a bunch of other things where it's like, oh, well, all of the pipes are a slightly different shape. It's like, well maybe if we just all agreed to make the pipes the same size or the same fittings, then you could like raise the level of abstraction. I'm not sure.

Alexei Boas:

And Pete, just to get a sense of the investment and just to be clear, this is not a side project. The companies who were having success on that made an intentional strong investment in building that and making that available to the whole company, is that so?

Pete Hodgson:

Yeah, for sure. I think the companies that were still struggling a little bit with really finding their mojo with CD, it was like a little bit of a side project or it wasn't like an intentionally a thing that everyone had realized was like, oh, we're building a thing. But for the companies that were really cranking on this, it was a full on internal product. It was a platform that was treated as a product. There were teams of people who worked full time on building this product. I would like to think that there were product managers or similar that were out there, talking to their customers, IE product delivery teams and trying to figure out how to do better, analytics around how was adoption of this platform going, all of that stuff. Now, it was a big investment, which implies that it was valuable because if I had just seen it at one or two places then maybe they were just people were wasting their time on something that was fun to do, but for it to be a very consistent theme tells me that there was a lot of value in doing it, even though it was a lot of cost as well.

Alexei Boas:

Cool. And one thing you did touch on that, but I'd love to explore that a little bit further. You talked about different types of companies, different types of architectures. Is it so that you have seen high levels of maturity with new and old companies, small and big, different types of architecture? Because sometimes we get this myth that, oh, I am this kind of company, I have this kind of legacy of history, therefore, I cannot get to a high level of maturity on continuous delivery. What did you see related to this and that variability?

Pete Hodgson:

That was interesting for me. I think probably in the back of my mind, maybe this is partly because I'm based in San Francisco and if I'm not drinking the Kool-Aid, then it's in the water, I guess had this a little bit of an assumption that the startups would be doing like cool, avant-garde, super sophisticated stuff and maybe the Midwestern, auto e-commerce thing would be a little bit further behind. What I actually found that there wasn't really that much of a correlation there.

Some of the older, more established companies that were not in the Bay area, who are just in the middle of the country, were just quietly kicking butt, basically. They were like, yeah, we don't really, most changes flow out to production within maybe a day of us starting them. And no, we don't really do branches anymore. And yeah, it's quite a simple system, so it's not that hard. I'm like, well, actually, yeah, you're doing really well. And it was almost like they didn't realize quite how well they were doing, which is a really good sign. If you're just like quietly succeeding, sometimes you don't realize that the thing that you're doing is actually really successful. That was definitely interesting to me.

When I reflect on it, I think part of the reason for that is because most of the time when you go to conferences and like who invests in an engineering blog because they're trying to recruit engineers? It's the Silicon Valley startups of the world. You get these breathless posts about how... I'll pick on Twitter from like 10 years ago because probably everyone involved is no longer involved in it. But like Twitter would like reinventing how to do like a [crosstalk 00:25:57]-

Mike Mason:

Messaging.

Pete Hodgson:

In rails. And it-

Mike Mason:

And it nearly killed them.

Pete Hodgson:

The amount of fanfare.

Mike Mason:

It nearly killed them.

Pete Hodgson:

Yeah, like multiple times. They really tried to do it like multiple times and the amount of fanfare and excitement that was done around this, the amount of hype that that got was way more related to the amount of marketing that was involved than the actual quality or good idea-ness of the thing. It's mean for me to pick on that specific example. The thing is every organization does really dumb stuff, but like only in Silicon Valley are they so proud about it.

Alexei Boas:

And how about the value aspect, Pete? You did mention, going back to what we were discussing, the definition of continuous delivery. Companies who had the ability to deploy frequently usually leveraged that and just the value of that. What have you found related to that? Are companies extracting value and business value out of this? Is this something the IT department is pushing and the rest of the company doesn't get it? What did you see connected to that?

Pete Hodgson:

That's interesting. The actual, legit research, I was talking about Accelerate and the DevOps research, Dora I can't remember what that acronym stands for.

Mike Mason:

Agency.

Pete Hodgson:

There you go, DevOps Research and Assessment, Nicole and those folks did actual science, which demonstrates that reducing software performance ties to organizational performance and part of software performances is driven by continuous delivery and these practices of reducing batch size. I was not doing science. And there's probably like a science word for like wasn't really doing science. It was a experiential report or something. I was just going and talking to people about it and trying to understand what was going on. There was definitely an interesting correlation where the science says reducing batch sizes is a good thing. And all of these organizations had that feeling. And when they were slipping away from that and saying like, oh yeah, we had a period where we were only able to deploy every two days. It was kind of they knew that was bad and they wanted to get back to like every day.

There was a sense and I think, when I asked for a couple of those organizations, I said like, well, why are you doing this? What is the benefit? And the biggest thing I heard was around responsiveness to the needs of the business. Essentially, we're able to deliver stuff back to the people that are asking for it fast enough that they stop yanking the wheel. Because I've seen that where business people are like, we need blah and every time I ask for it, it takes ages, so maybe if I ask harder or if I asked more often or if I micromanaged the crap out of it, then maybe that will work. They're like yanking the steering wheel and causing a bunch of noise. When I asked people like, what's the benefit? They played that back to me, like now we're able to respond to the business fast enough that they're stopping yanking the wheel and micromanaging us as well.

Mike Mason:

That's really interesting, Pete, because one of the questions you always get about continuous delivery or one of the opposite arguments is well, I actually don't want to release features that often because I'm going to be changing it under the feet of the users all the time and people are going to get annoyed by that. Or you get sort of these arguments about non-consumer facing website businesses that in theory can move slower and therefore, wouldn't appreciate that the smaller batch sizes. But what you're saying about satisfying the business more often so that you can nudge the ship in the right direction rather than trying to steer it hard to port, that's super interesting.

Pete Hodgson:

It's a universal thing that I think applies to almost everywhere. Maybe there's some exceptions around systems that are in almost a hundred percent maintenance mode and it's being changed very, very infrequently, then I could maybe buy that the investment isn't worth it. But now, I think the fundamentals are if you can make small changes more frequently than there's less cost and there's less friction. And it doesn't matter what the thing is that you're making the small changes to, it's just the fundamental thing is you're reducing risk and all of the lean principles and they apply like, not literally universally, but they apply very, very broadly I think.

For example, this is not related directly to the research I did for this book, but a client of mine is a FinTech company and they have mobile apps and when I first started... One of the first projects I did at Thoughtworks actually was for a big bank and it was when the iPad first came out, so it was a while ago. And the idea of releasing native apps frequently was crazy because oh, you can't really do it, it doesn't make sense in our context, blah, blah, blah. And even now this other FinTech I'm working for, they've moved to weekly releases of their native apps and they want to go faster and that's not that crazy now. And it turns out that it's just as useful for them as it is for the web people because it's the same fundamental forces at work, whether you're deploying to the app store or deploying to Kubernetes, it's the same ideas.

Mike Mason:

We're getting closer to the end of the time here for today's podcast. I wonder, Pete, if you thought there were any takeaways for people who are thinking about CD, I don't know, maybe they're struggling or maybe they're doing some CD and they want to check that maybe they're following the principles that you think are the most important. What kinds of takeaways would you give from this research that you'd done?

Pete Hodgson:

I think that the fundamental thing is around reducing batch size and deploying more frequently. And what's interesting is if that's your goal, what you should plan for is that act of deployment, you need to figure out how to scale that out because if today you deploy once every two weeks, it's possible to have there's these four people that do the deployment. Even if they're the DevOps people in the team or something, that is feasible to do. If you want to get to the point that you're deploying daily or more frequently than daily, then you've got to figure out how to scale that out and the only way, almost universally, it's not almost universally, everyone I spoke to, the way they scale that out is by pushing it down into the individual product delivery teams. And the way that you do that is you build tooling and you replace those four people who are able to do the deployment with rather than them doing deployment, they're spending their time building the platform that lets them do the deployment. Sorry, that lets the product engineers do self service deployment. And that's something that you should plan.

If you want to succeed, you've got to get to that point eventually. You can get there incrementally. Maybe first start getting used to the ideas and speed up, maybe automate the process a little bit, but it's still those four people that are running it, blah, blah, blah. You don't go there in one go, but expect that at some point you've got to be able to have an engineer walking their change all the way to prod without bugging any centralized people. And that's a really big cultural shift and big investment in tooling for some places, but I think it's the only way that you can actually succeed with this in a sustainable way,

Mike Mason:

Just want to echo something that you said earlier as well, which is that if you do build such a thing, a delivery platform for your organization, you need to be treating it as a product, having internal customers for it. You don't have a team in the corner who go and build it in an ivory tower and then you have all the problems with this sort of building stuff in a vacuum thing. That's not what you're talking about. You're talking about building it the right way.

Pete Hodgson:

Yes. And that's actually new obsession is platform teams. How do you succeed with platforms? Maybe I've just been unlucky, but I've been at so many places that have done that, where there's like a platform team who mostly they tend to be picked from as the people that are good at this stuff and they are picked because they're good at building software, not necessarily good at product management. And there tends to be this thing of like, oh, well, in my tiny little part of the org, the way we did it is X, all we need to do is build the software for that and then hit people over the head until they start using our software. And if they don't use our software, that's their fault because we were clearly hitting them.

Alexei Boas:

And Pete, before we wrap up, tell us a little bit about book. The whole conversation has been about the book, I realize that, but what's in the book? And can you tell us a little bit about that?

Pete Hodgson:

Sure. It's basically summarizing these interviews. I went and talked to these organizations and essentially, asked them all to describe what's your path to production. How does a small feature change get through to your user's hands? A lot of the stuff we've been talking about, like what were the similarities and what were the differences?And talk about things like, branch management and code review, the stuff that we were talking about around minimal branches, how people are doing code review, which is actually quite interesting.

And then something we didn't talk about actually is these patterns for testing to integrating your changes. If you've got 20 teams that all owned their own services, how do you check that your change is going to work well with all those other changes? That was interesting to me because there was just so many different ways that people are doing it, really in a lot of different ways.

And talk a lot about deployment and of the release bus idea and other ways that people are succeeding with, with deploying monoliths. And some stuff around feature flagging and decoupling deployment from release and how do you close that feedback loop around I made a change, what's the impact of that change and that was interesting to me. A lot of organizations that are doing okay, don't have that much sophistication there because they don't need it. An interesting thing to me. And really, the goal is for this to be real world, actual like, it's at least working for one organization because they're succeeding. And it's a free download, so it doesn't cost you anything and a reasonably short read.

Mike Mason:

Well, Pete Hodgson, thank you very much for joining us today. Where can people find you online if they're looking for you?

Pete Hodgson:

I think they should come and look from me because I love talking about this stuff. If there's anything where you've disagreed heartily with what I've said or you want to ask more questions, come find me. I am PH1 on Twitter and my website is thepete.net.

Alexei Boas:

Well, it was a great conversation. Great to have you with us and thank you very much for joining and thank you all for listening.

And if you have any feedback for us, don't hesitate to reach out or to leave a rating or comment on your preferred platform. Thank you so much for listening, bye.