Realising the full potential of CD

Podcast transcript

Alexey:

Hello, and welcome to the Thoughtworks Technology Podcast. My name is Alexey. I'm the head of technology for Thoughtworks Brazil, and I will be one of your hosts this time. And this time I'm joined by Rebecca Parsons, one of our co-hosts. And Rebecca, we are going to be playing kind of a dual role on this one, right?

Rebecca:

That's right. Continuous delivery is something that I'm incredibly passionate about, and so even though I am officially here as a cohost, I will probably be sticking my oar in, but I will try to let you hear from our guests.

Alexey:

Yeah. And we're delighted to have three guests with us this time, Ken, Aravind, and Scott. So maybe let's do a quick round of introductions. Ken, would you mind introducing yourself and going first please?

Ken Mugrage:

Sure. So my name is Ken Mugrage. I am a principal technologist with Thoughtworks. I've been here just over 11 years, but spent most of that time working on continuous delivery and dev ops.

Alexey:

And how about you, Aravind?

Aravind:

Sure. Hi, I'm Aravind. I'm a consultant at Thoughtworks. I've been here quite a long time. Over the last few years I've been a product manager for GoCD, an open source continuous delivery server started at Thoughtworks, but I have a development background from before that.

Alexey:

Okay. Thank you. And finally, Scott, would you mind introducing yourself to our listeners?

Scott:

Absolutely. Hi, my name is Scott Davis. I'm a web architect and principal engineer with Thoughtworks. So I'm really excited to talk about CD from a front end perspective.

Alexey:

Okay. Thanks so much and, well... All right. And our topic today is how to realize the full potential of continuous delivery and go beyond just building a pipeline. And well, why don't we start with a definition. So we we've all seen different people consider continuous delivery to be slightly different things. Why don't we start there? And Rebecca, since you mentioned about your passion for that, would you like to give us your take on this? What is continuous delivery?

Rebecca:

Well, in its simplest form, I think of continuous delivery as the end to end process of realizing some element of business value from end to end. And so this is how I am able to automatically promote software through the entire process of going from developer writing code to whoever the end user is being able to interact with that feature and derive value from it. And this necessitates us to automate all of the different deployment steps and the setups, automate the level of testing that's necessary to understand what the consequences of the code changes are, automate the environment set up so that we know what it takes to actually construct one of these environments and know what environments are actually running in production. There's a strong emphasis on measurement, and do we know how well the software is performing? Not just from a technical perspective, but is it delivering the business value?

Rebecca:

In fact, there have been discussions around should we redo the agile definition of 'done' to say, you can't say a story is done until you get feedback in production that it's actually delivering the value that it was supposed to. Now, I think that completely breaks the short feedback cycle that is so crucial to make agile software development work. But I do think that there is something to that notion that we can't really say we've accomplished something unless we actually see the value in production. So continuous delivery to me is the whole end to end. What does it take to deliver value into production and know that it is respecting all of the functionality requirements as well as a nonfunctional or cross-functional requirements, the compliance requirements, that entire process from end to end. That is continuous delivery to me.

Alexey:

Yeah, that's great. Well, that definitely brings a much, much broader perspective and allows us to also look at different things and then try to bring unusual things into that process.

Scott:

What I really like about that definition that Rebecca brought up was she continued talking about delivering value. And I know as I deal with developers across a number of years of experiencing the industry, it's really common for us to say, "Aw, my code is done. So therefore it's done." But the idea of delivering value means it's not really done until it's in the hands of the users. And so that idea of delivering value, I like saying software is pass fail. You know, it's either in the hands of the user or it's not, And any amount of effort that you put into that up until it actually is in the hand of the developer is worth nothing. It's not a value proposition for doing a whole bunch of work that your users can't benefit from.

Aravind:

Especially when people talk about measurements these days, they seem to equate number of builds that have run, number of pipeline instances and so on, and not about the real business value. And that is something that we need to be very aware of. I see that people use it as a proxy, but when you start using that as a real measurement, rather than what it is a proxy for, that becomes the challenge.

Alexey:

And about how that happens in practice? So for example, Ken, I know that you have seen many different contexts and challenges companies face when trying to build their continuous delivery practices and infrastructure. Are there a couple of stories or maybe one key story that would highlight some of these less common elements of continuous delivery, less discussed elements of continuous delivery, that you could share with us?

Ken:

Sure. So the big thing is, as was mentioned, that automation of course is an important part, but it's not just about the automation. Rightfully so, a lot of companies are concerned about saying, "Okay, product team, you now have full automation to production. And anything you do is just going to automatically be there in users." I mean, that is the end goal, but at the same time, I could write a very poor application, put the Thoughtworks logo on it, stick it on AWS and do damage to my company's reputation. So we want to do all these things, but you hear a lot of times about people adding a deploy step to the end of their CI pipeline, and now that's continuous delivery, and that's not what it meant.

Ken:

So one of the best examples I've seen is there's a big retailer that really needed to be able to react faster. And this was a few years ago, but they wanted to be able to react faster and do continuous delivery and put things out to users faster and get feedback and all of the good things. But at the same time, their, their company was worried about damaged reputation and risk and compliance and all of those kinds of things.

Ken:

So one of the things they did is they did automated testing. Because you can do automated testing and security compliance. They're not a complete substitute for people skills, of course, or should you say people's skills, but what they did is they did those tests in parallel with the development teams. So if the development teams would create a thing, some kind of deployable unit, when their deployable unit was going through tests... So let's say they create a WAR file and it's going through different kinds of functional tests or unit tests or whatever they happen to be. At the same time, a separate pipeline would pick that up and run compliance checks and security checks and those sorts of things on it.

Ken:

I think we're all fans at Thoughtworks especially of autonomous and cross functional teams. But the reality was that there needed to be earned trust that wasn't there yet. And so by using these separate pipelines, they didn't slow down their development cycle. They didn't make the functional tests wait for the security test to be done, for example, or the other way around. So they still got the fast feedback, but they also made sure that only things that were in compliance would get to production. And that's a pattern I've seen more and more, and one that I encouraged a lot of folks to consider.

Alexey:

Yeah, that's cool. So that's far beyond the common usages of tools and techniques that I see companies make. So Aravind, you have been very close to continuous delivery tools, development, and so on. So could you tell us a little bit about advanced features that can be useful in those situations and how to build those things and what patterns and anti-patterns to consider and to be mindful about when trying to build those kinds of things Ken was talking about?

Aravind:

Deployment tools essentially evolve in step with changes in software development. We have seen continuous delivery tools, for example, change depending on the kind of software that's being built and platforms being targeted. For instance, smartphone applications, webapps, microservices, containers, container schedulers, these things, lots of different changes. And in an enterprise setting, there are also other tools and platforms that you might need to integrate with. ERPs, mainframes, lots and lots of databases, lots of different complexities in an enterprise.

Aravind:

But some aspects of delivery tooling remains relevant across all of these different approaches to software delivery. For instance, good visualization and traceability remains valuable, whatever you're developing, whatever kind of delivery that you're doing. GoCD, for instance, has something called a value stream map, which I see people still discover and enjoy. Good modeling constructs, for instance, backed up pipeline related abstractions, I've always appreciated. For me, it's really strange to see how many tools in the space still just have a build or a task as the level of abstraction and allow people to try and express the whole complexity of their enterprise using those really small building blocks, right?

Aravind:

These days, for instance, Kubernetes is all the rage. So you can find tools which integrate really well with container schedulers, such as Kubernetes, and sometimes even using Kubernetes as building blocks themselves for CD. I'm not sure how well that approach really works, because to me, Kubernetes is a platform for building things on top of it. I'm not so sure about CD tools yet, but we will see. That's definitely evolving right now. And finally, there are tools which provide better deployment patterns built into the CD tooling itself. For example, canary, blue-green deploys, deploys across huge fleets of servers, and so on.

Aravind:

So there are definitely choices and what you pick depends on what you need and what you're comfortable with. There's definitely a huge spectrum. Sometimes I feel many organizations don't even understand continuous integration, let alone continuous delivery. So there's a way to go for a lot of people, I think.

Alexey:

Yeah, it's interesting, because it does tie back to the maturity the organization has with its own engineering practices and engineering maturity. So it's not just about the tooling or how to organize things, but it goes back to some of the being able to execute an engineering practices and deliver them. I guess that's where, at least in my experience, that's where many of the challenges are, right?

Scott:

And I think we're hearing a pattern kind of form here, when Ken talks about a good CD pipeline is all about earned trust, and when Aravind talks about a good CD pipeline is all about good visibility. What we're really beginning to talk about here is how do we trust our output? And if our releases were happening once a year, because you had to burn things to a CD and ship them out and everything like that, a delivery pipeline could be very manual and very precious and very bespoke or even kind of one off on a per release basis. But when we're talking about books like the State of the DevOps reports, where they say highly successful organizations are deploying to production tens if not hundreds of times a day, it does require a different model.

Scott:

And we're talking about the automation aspect of this, but automation without earned trust and without good visibility and something that doesn't deliver value is missing the point. We can automate all of these tasks, but if we don't have visibility in those tasks, and if we don't have trust in those tasks and those tasks, aren't really delivering value, then the automation is a fun exercise, but ultimately not value-delivering.

Ken:

You know, that brings up an important point that a lot of times gets mixed up, the difference between continuous delivery and continuous deployment. So in continuous deployment, you have an automated process that is, if everything passes, just going straight to production. That's the folks that are doing tens and hundreds and what have you of deployments per day. One of the things that I really liked that the DORA version of the State of DevOps report came out with is they're now saying elite performers are deploy on demand, because continuous delivery means that you can deploy anytime you want to, not that you do. Okay. So like, if Adobe gave me new versions of Photoshop several times a day, I would get very upset. And so there's a lot of software categories that are not a good candidate for continuous deployment, but even if your model is print a CD or USB in today's world or whatever it is, even if that is your distribution model, you should be able to deliver at any time.

Rebecca:

Yeah. And the way I like to think about it is that the actual move into production goes from being a technology driven decision, when can we safely do this, to a business decision on when is the right time for us to do this? And you know, it's interesting, Scott, you talk about automation and trust. If you do have a blind deployment process, this continuous deployment setting, that's one level of trust. The other way that I look at it, though, is much of this automation, particularly in these steps of deployment and configuration, are what helps to mitigate the risk of the deployment. Because no human being is at their best at three o'clock on a Sunday morning, we all make mistakes, except the computers don't make a mistake.

Rebecca:

And so I think one of the great benefits that organizations can get out of the discipline, the end to end engineering discipline of getting to the point of continuous delivery, is that you can take the business risk to experiment because you are not adding on to that risk of the experiment an, "Oh, by the way, maybe we'll crash the database," or something like that. And so I think to me, that's a critical part of enabling that end to end business value is being able to have some of those experiments.

Aravind:

Yeah. The peace of mind that you get from knowing that you've deployed this thing so many times, that itself is what going the journey of continuous delivery to me. For example, in GoCD, we ourselves doubted whether we could deliver GoCD continuously. If you look at the literature, a lot of them, people talk about SAS services, you know, services in the cloud as things that you continuously deploy. But as Ken was mentioning, even if it's something that you're doing like burning a CD, it is possible to deploy that. I've heard from people who were doing CD with real microchips, which they were developing and burning them onto test rigs and running, doing that continuously. So it's definitely possible.

Scott:

All of this reminds me of something that a frequent cohost here, Neal Ford, often says, that if you as a human are doing something that is easily automatable, what happens when you go home for the day is the computers all get together behind your back and laugh at you. We need to be constantly automating these kinds of things because it frees up us silly humans to do more and more interesting things and more and more innovative things. There is no joy in doing these repetitive tasks over and over again. As Rebecca mentions, this is how we add trust to the system as we automate it, so we know these are happening in a consistent, known, good way over and over again. And just one of the added benefits are my servers aren't laughing at me behind my back.

Ken:

Yeah. And you made a really good point there, is that it frees up the humans. I think part of being responsible and continuous delivery and going that extra mile that not everyone's going, the reason that you automate this, to me anyway, the security and the compliance that you can, is so that the people with that deep expertise can go do the things you can't automate. More penetration testing, more edge cases, et cetera. People are always, "Oh, you're going to automate people out of a job." No, you're going to automate them into more interesting jobs. And so I think that giving people the ability to do that and apply those skills is vital.

Alexey:

Yeah, and that connects back to, Ken, you were talking about automating some auditing aspects, like security aspects, compliance, and things like that. And Scott, I know you had been looking at how to bring into the CD pipeline monitoring or fitness functions, performance, and things like that. How does that work? Because that frees even up more people, more of people's time. It also lowers the risk because you know in what state the application is right now, and you can work on that. You can also see the impact of changes and things like that. So that broader view of monitoring into the continuous delivery pipeline, at least in my opinion, provides a lot of peace of mind because you know what's happening, you know in what state that is. So Scott, can you talk a little bit about how to add some of those things to the pipeline?

Scott:

Absolutely. From a front end perspective, I get really interested in tracking quality in the forms of things like accessibility and performance and privacy and security. APPS is a kind of cloying but effective little acronym to kind of keep these in mind. I consider these all fitness functions, but it would be really awkward for me to describe what a fitness function is when Dr. Rebecca Parsons is on this call right now. And and my first exposure to this idea of fitness functions was from her book from last year, Building Evolutionary Architectures with Neal Ford and Patrick Kua. So before I get into the dirty details of accessibility and performance, Rebecca, do you want to give a nice high level overview of what a fitness function is?

Rebecca:

Well, fitness functions are designed to unambiguously assert whether or not a particular condition is met. You can think of it, for those who are familiar with the field of mathematical optimization, with the objective function. One example most people are familiar with is the traveling salesman problem. And what you were trying to do is minimize the distance that a salesman has to travel but visit all of the different cities, and so the fitness function in that case is the distance that salesman traveled. And what we are proposing in the construct of evolutionary architecture is using this notion of fitness function very broadly to say, as you look at your architecture and determine things that are architecturally critical to the success of your system, and then you characterize that requirement in the bounds of a fitness function that then you can hopefully automate.

Rebecca:

But actually being able to automate the fitness function isn't the defining characteristic. The single most important characteristic of a fitness function is you and I will never disagree of whether the function passes or not. So something being maintainable is not a fitness function. Something having a cyclomatic complexity less than 15, that can be an effective fitness function. But we want to think about all of these characteristics that we want our systems to display. How would I measure that? How would I objectively say, therefore, whether or not my system is satisfying this particular requirement?

Scott:

Yeah. And I've said this a frequently in the past, that I love the idea of taking all of the engineering rigor that we apply to software development kind of historically, and typically on the backend, we can apply that same engineering rigor to front end technologies as well. So performance I think is a really easy one. And as Rebecca pointed out, we want these to be measurable and quantifiable. We don't want to just be able to say, "We need our website to be fast," or, "We need it to be performant." We can get to the detail of, "I want this to have a time to interactive, a TTI, of under five seconds on a mid range Android smartphone on a slow 3G network." I mean, if that's our unambiguous definition of what our performance targets are, as Rebecca mentioned, and then we all agree on what these are and we can throttle networks and we can throttle CPU usage, and we can do things to emulate that kind of scenario.

Scott:

So making these quantifiable, and again, the engineering rigor we apply to our code, we assume that we're writing unit tests around our code to make sure that our outcomes match our expectations. And so if unit tests are at the micro level, kind of testing the bricks of the building, not the building itself, then these fitness functions are the same kinds of tests that we can apply to our architecture, but with all the same rigor and quantifiability behind it. We can measure time to interactive in milliseconds. In terms of accessibility, there are some very easy things we can do that are essentially syntax errors that would be caught by any HTML linter. I mean, we can talk about images that don't have alt text on them and headers that are out of order. You can't go H2 H2 H1 H3 H4 H5 based on how they appear, because those have deep semantic meetings. And it really breaks accessibility when people are trying to see the outline of your document by measuring it in terms of which header levels they're in.

Scott:

So there are a lot of these very quantifiable things that we can use to measure the health of our architecture and these fitness functions, if they can be applied to a CD pipeline, we can use things like lighthouse tests that are free with Google Chrome, but we can also NPM install it. And once we can build this into a pipeline, we can have achievable, quantifiable, measurable goals around accessibility, around performance, around privacy, around security. This is automation with trust taken to the architectural level.

Alexey:

Okay. Any final thoughts? I guess we're coming close to the end of the episode. Any final thoughts you want to share with our listeners?

Ken:

I think I would do, and it's been said a couple of times, but you know, if there's one takeaway, continuous delivery is not adding a deployment step to the end of your continuous integration run. It does require more rigor. It's what reduces the risk, and so think about all of the things that are going to affect your company. What is the response time, you know, those kinds of stuff, but also we're recording this in the middle of a global pandemic where we have a lot of systems that are legacy, and legacy is not a bad word, that have never had these patterns applied to them because they're not quote unquote "modern", that we also need to think about, is what can we do to systems, without completely rearchitecting them, that can bring them along so that when things happen, and I stress when not if, that we can react as a business and as a community, as people, to things that require technology changes. So let's not leave those things behind and only focus on the shiny new Kubernetes.

Rebecca:

And I would just also reiterate a point ken made earlier. Very often when I talk about continuous delivery, people get all nervous about, "I don't want somebody to be able to check in code and have it plopped into production." If you read the book Continuous Delivery, there is absolutely nothing wrong with putting a big green button on some manager's desk, and that thing isn't going to go live until that person smacks the big green button. Continuous delivery means you can go from checking it in code and dropping it into production with nobody touching it. It doesn't mean you have to. It doesn't mean you have to deploy 10 or 100 times a day. It means you could if you had to, and it's the level of risk reduction you, get even with making smaller steps along that continuous delivery journey, can be enormous. You can get significant business benefit just by incremental steps and starting down that path, but don't feel like you failed if you don't allow something to drop immediately into production, because that's not the point.

Aravind:

Yeah. For me, I think the point I was thinking about segues really well, because when I talk to people in larger organizations, they're always worried about where to start and whether they can even start. And every time I've seen them succeed in larger organizations, they've always started really small with a small team to demonstrate to the rest of the organization that this is a viable path to improvement. And that goes back, to me, to what Ken was saying about trust earlier. That trust, the way that I saw it, is trust of the people on the ground to go with this journey. And it can start with a small step. It doesn't need to start with a huge plan to continuously deliver everything. It can start really small and can go.

Alexey:

Okay. Thank you very much. This was a great conversation and great to have you all with us. Thank you very much for joining.

Mike Mason:

I'm Mike Mason, and join me on the next episode of the Thoughtworks Technology Podcast, where we will be talking with Eric Dörnenburg about the fact that 50% of code is now running in the browser.