Where’d my tail go? - Using Continuous Delivery principles to shorten our release tail
This is the story of how the ThoughtWorks Studios Mingle team used continuous delivery principles to reduce the testing period at the end of our release cycle from 2-3 weeks to 1 day.
Background
In his blog post, Shortening the Tail, Jim Highsmith talks about tail length as a true learning metric that can help a team understand how it is progressing toward achieving its goal of continuous delivery.
Well, about a year ago, the Mingle team’s tail length for regression testing was 2-3 weeks, which was almost 25% of our desired release cycle. That’s a lot of time! We weren’t happy with our tail and we knew we needed to improve if we wanted any chance of delivering valuable software more than every three months.
What’s going on?
To understand what was contributing to our long tail, you should first know that Mingle is currently shipped as installed software with official support for multiple operating systems and databases. Our support for multiple operating systems and databases requires that we run installer specific regression tests for each operating system-database combination, in addition to tests for upgrading from previous versions of Mingle. Now here’s the kicker - the installer specific tests were all manual! With each combination taking about one day, it’s pretty easy to see why our tail was weeks rather than days.
CD principle: Automate almost everything
Neal Ford, another ThoughtWorker, likes to joke that when people do the things computers could do, all the computers get together late at night and laugh at us. Needless to say, the computers were having a good chuckle at our expense. The good news for us was that everything we were doing could be automated. All we were doing was installing our application on operating system ‘x’ pointing to database management system ‘y’ and doing a quick smoke test to make sure that common usage scenarios still worked. With a little pairing between our devs and testers we were soon able to automate this process and make it a part of our Go pipelines. By automating our installer regression tests and adding them to our Go pipelines we were able to get feedback--much sooner--about our installers on every change to the application.
So what’s the result of all of this automation and feedback? Now that we’re identifying and fixing installer issues earlier in our process - and using machines to run through previously time consuming manual testing - we now have a release tail that has been shortened from 2-3 weeks to 1 day.
More good things can happen when your tail is short
A lot of good things can happen when your release tail is short. For the Mingle team, we were immediately in a much better position to reliably--and regularly--release valuable upgrades to our customers. Bug fix releases can now be turned around in a matter of days rather than weeks. Other, less obvious, benefits came from freeing up members of our team, allowing them to focus their energy on truly unique problems that computers aren’t (yet) able to solve. A year ago our testers were less involved at the start of any new product release because they were busy running manual installer testing. Today our testers participate in all activities at the very beginning of any release which has helped improved our flow.