Enable javascript in your browser for better experience. Need to know to enable it? Go here.
Five ways to go wrong with continuous integration

Five ways to go wrong with continuous integration

Continuous integration doesn’t get rid of bugs, but it does make them dramatically easier to find and remove.
Martin Fowler
chief scientist, Thoughtworks

Continuous integration (CI) has become a staple in the world of agile software development, largely because it allows teams to detect problems early. In CI, developers integrate code into a shared repository several times a day, and each check-in is verified by an automated build. 

 

However, despite its many benefits, CI can sometimes go terribly wrong — often because of harmful 'antipatterns' or bad practices. This article explores the antipatterns and bad practices that make CI ineffective and how you can avoid them. 

What is an antipattern?

An antipattern is just like a pattern, except that instead of a solution it gives something that looks superficially like a solution but isn’t one.
Andrew Koenig
author, member of the Large-Scale Programming Research Department at AT&T's Shannon Laboratory and project editor of the C++ standards committee

Antipatterns are common 'solutions' to development problems that actually do more harm than good. While antipatterns appear to solve problems, they also create new problems and for CI to work properly antipatterns must be replaced with more effective and reliable alternatives.

 

Five common CI mistakes

 

#1 Infrequent check-ins

 

This antipattern makes development teams go against the fundamental principle of CI — they let code stay on local laptops for too long without it being committed and checked-in. 

 

This is usually because the feature they’re working on needs an exhaustive set of changes to be complete. But, as the feature is being developed, other project members continue to check-in their work. As more code is gradually checked-in, merge conflicts arise that are difficult to solve and can severely delay integration.

 

How to avoid it

 

Split the feature into smaller, independent tasks that are followed by a commit and check-in. Small commits and frequent check-ins will make integration faster and hassle-free. Small commits also allow for specific tasks to be reverted, if necessary, rather than the whole feature.

 

For example, tasks for implementing a write to a database could include:

 

  • Add a repository layer to a write to database

  • Introduce a new endpoint and integrate it with the repository layer

  • Introduce different error responses when a write fails

  • Add validation checks for the request body

     

#2 Feature branches

 

This antipattern dictates that all feature development should take place in a dedicated branch. This makes it easy for multiple developers to work on features without disturbing the main codebase. 

 

However, if feature branches don’t get changes from the mainline (the main branch for the repository) until it’s finished, the project isn’t even close to following continuous integration. In fact, this approach is more like continuous isolation, or CI theatre.

 

An isolated feature branch leads to multiple conflicts because by the time the feature is completed, the mainline will have tons of changes. This forces all developers to sit together and merge or important changes might be missed. 

 

Let’s look at an example.

A figure representing 2 developers(Professor Plum & Reverend Green) simultaneously working on 2 features. They develop the features on long-lived feature branches while there are few commits on the mainline. They are shown working in isolation on their own branches and occasionally take changes from the mainline branch but never push their changes to the mainline till the very end. Developer 1(Professor Plum) finishes her work early and pushesh to mainline. When developer 2(Reverend Green) tries to merge his code after, he runs into a huge merge conflict because now he has to incorporate all the changes from other developers.

Source: Feature Branch by Martin Fowler

 

The image above shows a repository containing many long-lived feature branches. The branch by Professor Plum has local commits P1, P2..P5 and she has taken changes from the mainline at points P1–2 and P3–4. Finally at P1–5 she merges her feature to the mainline.

 

The branch by Reverend Green, which started at the same time, has local commits G1, G2..G6 and has changes from the mainline at G1–2, G3, which are before P1–5. When he decides to merge it to the mainline, he needs to incorporate all the changes done by Professor Plum. And because there are a lot of commits, it will be difficult to integrate them all. 

 

How to avoid it

 

Avoid long-lived feature branches and maintain only one branch as a source of truth. Begin all development work with checking out from this branch — commonly known as the 'trunk' (or 'main' in Git terminology) — and pushing to it. This is called trunk-based development, because everyone develops against one branch: the trunk.

 

In projects which need to have pull requests approved before the merge, use short-lived branches for each task. You can merge these branches when the task is completed. These branches can be deleted and the next task could be taken on a fresh branch taken from mainline.

 

This approach increases the number of PR approvals, but also it makes the PRs shorter and crisp, which can be approved quickly. The larger a PR is the more review effort it requires.

This figure is similar to the previous one representing 2 developers(Professor Plum & Reverend Green) simultaneously working on 2 features. However instead of working on long-lived branches they are shown working on short living branches existing for only one commit. Both developers are shown syncing their changes to and from the mainline at frequent intervals. Since there is a sync to and from the mainline branch, there are no major conflicts encountered throughout the feature development.

Source: Feature Branch by Martin Fowler

 

In the image above, Reverend Green makes a local commit G1 and merges it to the mainline. Professor Plum makes a local commit P1 and before she can merge it into mainline, she needs to take commit G1, merge it locally with her changes and then push. Reverend Green does the same when he needs to push the next commit. It’s a series of small merges, without needing a huge merge event.

 

#3 Broken builds

 

In reality, a broken build is not a problem — the build pipeline is designed to give early feedback when something goes wrong. The problem, however, arises when a build stays broken for a long time. 

 

If a broken build isn’t fixed quickly, it blocks other check-ins. Check-ins on a failing build then create more files, more changes and more dependencies that make it difficult to detect and isolate the defect.

 

How to avoid it

 

Be disciplined about fixing broken builds as soon as possible. When fixing the build requires a lot of time, revert the commit instead, so that one person can look into the build issue, while others get unblocked to check-in.

 

#4 Build time

 

One of the main reasons for taking a CI approach is to get faster feedback. A long-running build defeats this purpose by increasing the wait time for developers after check-in. Often, this leads to our first mistake, infrequent check-ins, because developers tend to check-in multiple commits at once to avoid long wait times.

 

How to avoid it

 

One reason for a longer build time is running all sorts of checks and tests as part of the build. To overcome this, configure the build to just compile the code and run quick unit tests. Push all the other checks and tests that need time, such as testing integration among different components, to the next stages in the pipeline.

 

The goal is to find the right balance between tests/checks and build time so a build is stable enough to move to the next task. Use the test pyramid (see image below) to achieve this balance.

Five ways to go wrong with continuous integration

In this method, only the unit tests will be run as part of the build, which take minimal time. As you can see from the pipeline image below, everything else can run later without blocking developers from moving to the next task.

Five ways to go wrong with continuous integration

A demo pipeline that runs different kinds of tests in different stages

 

#5 Build feedback

 

If teams don’t realize their build is broken, they’re likely to keep making check-ins on a failing build, leading to the issues we highlighted in mistake #3. 

 

How to avoid it

 

Set up a CI dashboard showing the status of the build, and configure CI servers to notify teams by email or on Slack about broken builds. As soon as the team receives a broken build notification that should become their priority.

Five ways to go wrong with continuous integration

Source: GoCD - Slack Build Notifier by Ashwanth Kumar

 

While doing this, though, it’s important to avoid alert fatigue. When people become inundated with build notifications, they’re likely to start ignoring them, rendering the whole process ineffective.

 

So, instead of configuring the CI server to send notifications after every check-in, you should notify people when: 

 

  • A new check-in fails, so that your team can fix it 

  • A new check-in fixes a broken build, so that those who were waiting for the build to be fixed can check-in their code 

     

Avoiding CI mistakes: a checklist

 

  • Make small, independent and frequent commits to the remote repository

  • Use trunk-based development or short-lived branches for small tasks

  • Make fixing a broken build your top priority, and don’t allow check-ins until the build is fixed

  • Reduce build time by getting faster feedback if something goes wrong

  • Set up a dashboard and a notification mechanism to inform team members about build status. Only send important CI notifications to avoid alert fatigue

 

Above are the antipatterns or bad practices of continuous integration that occur in a day-to-day project. Hopefully, the solutions shared will help avoid these problems and put your project one step closer to continuous integration and eventually continuous delivery.

Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.

Keep up to date with our latest insights