ThoughtWorks
  • 联系我们
  • Español
  • Português
  • Deutsch
  • English
概况
  • 工匠精神和科技思维

    采用现代的软件开发方法,更快地交付价值

    智能驱动的决策机制

    利用数据资产解锁新价值来源

  • 低摩擦的运营模式

    提升组织的变革响应力

    企业级平台战略

    创建与经营战略发展同步的灵活的技术平台

  • 客户洞察和数字化产品能力

    快速设计、交付及演进优质产品和卓越体验

    合作伙伴

    利用我们可靠的合作商网络来扩大我们为客户提供的成果

概况
  • 汽车企业
  • 清洁技术,能源与公用事业
  • 金融和保险企业
  • 医疗企业
  • 媒体和出版业
  • 非盈利性组织
  • 公共服务机构
  • 零售业和电商
  • 旅游业和运输业
概况

特色

  • 技术

    深入探索企业技术与卓越工程管理

  • 商业

    及时了解数字领导者的最新业务和行业见解

  • 文化

    分享职业发展心得,以及我们对社会公正和包容性的见解

数字出版物和工具

  • 技术雷达

    对前沿技术提供意见和指引

  • 视野

    服务数字读者的出版物

  • 数字化流畅度模型

    可以将应对不确定性所需的数字能力进行优先级划分的模型

  • 解码器

    业务主管的A-Z技术指南

所有洞见

  • 文章

    助力商业的专业洞见

  • 博客

    ThoughtWorks 全球员工的洞见及观点

  • 书籍

    浏览更多我们的书籍

  • 播客

    分析商业和技术最新趋势的精彩对话

概况
  • 申请流程

    面试准备

  • 毕业生和变换职业者

    正确开启技术生涯

  • 搜索工作

    在您所在的区域寻找正在招聘的岗位

  • 保持联系

    订阅我们的月度新闻简报

概况
  • 会议与活动
  • 多元与包容
  • 新闻
  • 开源
  • 领导层
  • 社会影响力
  • Español
  • Português
  • Deutsch
  • English
ThoughtWorks菜单
  • 关闭   ✕
  • 产品及服务
  • 合作伙伴
  • 洞见
  • 加入我们
  • 关于我们
  • 联系我们
  • 返回
  • 关闭   ✕
  • 概况
  • 工匠精神和科技思维

    采用现代的软件开发方法,更快地交付价值

  • 客户洞察和数字化产品能力

    快速设计、交付及演进优质产品和卓越体验

  • 低摩擦的运营模式

    提升组织的变革响应力

  • 智能驱动的决策机制

    利用数据资产解锁新价值来源

  • 合作伙伴

    利用我们可靠的合作商网络来扩大我们为客户提供的成果

  • 企业级平台战略

    创建与经营战略发展同步的灵活的技术平台

  • 返回
  • 关闭   ✕
  • 概况
  • 汽车企业
  • 清洁技术,能源与公用事业
  • 金融和保险企业
  • 医疗企业
  • 媒体和出版业
  • 非盈利性组织
  • 公共服务机构
  • 零售业和电商
  • 旅游业和运输业
  • 返回
  • 关闭   ✕
  • 概况
  • 特色

  • 技术

    深入探索企业技术与卓越工程管理

  • 商业

    及时了解数字领导者的最新业务和行业见解

  • 文化

    分享职业发展心得,以及我们对社会公正和包容性的见解

  • 数字出版物和工具

  • 技术雷达

    对前沿技术提供意见和指引

  • 视野

    服务数字读者的出版物

  • 数字化流畅度模型

    可以将应对不确定性所需的数字能力进行优先级划分的模型

  • 解码器

    业务主管的A-Z技术指南

  • 所有洞见

  • 文章

    助力商业的专业洞见

  • 博客

    ThoughtWorks 全球员工的洞见及观点

  • 书籍

    浏览更多我们的书籍

  • 播客

    分析商业和技术最新趋势的精彩对话

  • 返回
  • 关闭   ✕
  • 概况
  • 申请流程

    面试准备

  • 毕业生和变换职业者

    正确开启技术生涯

  • 搜索工作

    在您所在的区域寻找正在招聘的岗位

  • 保持联系

    订阅我们的月度新闻简报

  • 返回
  • 关闭   ✕
  • 概况
  • 会议与活动
  • 多元与包容
  • 新闻
  • 开源
  • 领导层
  • 社会影响力
博客
选择主题
查看所有话题关闭
技术 
敏捷项目管理 云 持续交付 数据科学与工程 捍卫网络自由 演进式架构 体验设计 物联网 语言、工具与框架 遗留资产现代化 Machine Learning & Artificial Intelligence 微服务 平台 安全 软件测试 技术策略 
商业 
金融服务 全球医疗 创新 零售行业 转型 
招聘 
职业心得 多元与融合 社会改变 
博客

话题

选择主题
  • 技术
    技术
  • 技术 概观
  • 敏捷项目管理
  • 云
  • 持续交付
  • 数据科学与工程
  • 捍卫网络自由
  • 演进式架构
  • 体验设计
  • 物联网
  • 语言、工具与框架
  • 遗留资产现代化
  • Machine Learning & Artificial Intelligence
  • 微服务
  • 平台
  • 安全
  • 软件测试
  • 技术策略
  • 商业
    商业
  • 商业 概观
  • 金融服务
  • 全球医疗
  • 创新
  • 零售行业
  • 转型
  • 招聘
    招聘
  • 招聘 概观
  • 职业心得
  • 多元与融合
  • 社会改变
遗留资产现代化持续交付技术

Modernizing your build pipelines

Mario Fernandez Mario Fernandez

Published: Nov 5, 2018

Doing Continuous Integration is a lot easier if you have the right tools. In our project at a german car manufacturer, we were tasked with developing new services and bringing them to the cloud. We had a centralized Jenkins instance, shared by all the teams in the department. It didn’t fit our needs and made it harder for us to deliver software quickly and reliably. In this article we’ll explain how we  migrated to a new solution and recreated all of our build pipelines, and explore what we learned from our experience.

The problems with the clients existing Jenkins instance included:
  • It was a snowflake, which meant that understanding its configuration was hard. Making changes, or recreating it would have been very time-consuming.
  • Understanding the legacy code was difficult, changing it even more so. That’s because an internally developed  domain-specific language (DSL) to write pipelines as code had been used to write most of the build pipelines
  • Maintenance had been sparse and ad hoc, because it wasn’t owned by any team
Our old pipelines were bloated and slow, and couldn’t support our cloud transition. We wanted to adopt a cloud-native approach, instead of doing a lift and shift. That is why we decided to rewrite our pipelines instead of migrating them. To increase the ownership of the team, we decided to use a new CI tool as well, owned and operated by the team. This would allow us to address the problems mentioned above.

Making a choice

We wanted to make this process as transparent and objective as possible. These were our requirements for a new tool:
  • Full configuration through code. Manual changes decrease maintainability. We want to reflect every change in version control
  • Pipelines as first class citizens. Build pipelines are a crucial part of our infrastructure, and we want the best possible support from our tool
  • Container friendly. We want to make every step in the pipeline reproducible. For that we want to use freshly built Docker containers for all our steps
  • Visualization. The status of the build should be easy to check at any time, as pipelines are checked much more often than they are written
  • Support. Active community, regular releases

We researched alternatives based on these criteria. We started with tools familiar to the team, such as ConcourseCI, TravisCI, CircleCI, GoCD, and Jenkins 2.0.

All of them offered what we were looking for, but in this particular case, we opted against suggesting GoCD — which is developed by ThoughtWorks. We’re actually big fans of GoCD, but when we’re working with new clients, we sometimes choose not to recommend our own products, in case they’re worried about bias.

We spiked some of the tools in the team and wrote down our findings. In the end, we decided to use Concourse. We liked the support for containers, the simplicity of the UI and that the pipeline was fully defined through a YAML file. However, we believe that the lessons we’ve learned can be applied to any modern CI tool.

Building high quality pipelines

Our pipelines have a bunch of responsibilities. For a frontend application these include:
  • Multiple linters, for TypeScript, CSS, Shell Scripts, Terraform and Docker
  • Unit tests, both for JavaScript and for Shell Scripts
  • Dependency checks with npm audit
  • Applying infrastructure changes
  • End to end tests
  • Building the application and deploying it to a CDN

Figure 1: Building pipelines
That only defines what the pipeline does. But what makes a pipeline more useful than another?
  • It’s fast. Quick feedback helps you react faster
  • It’s reliable. Having to run a flaky build over and over is extremely frustrating
  • It’s maintainable. A pipeline should be maintained without too much effort
  • It’s scalable. Usually there are multiple pipelines to maintain
  • It’s visual. Dependencies are easy to understand and a failed build can be traced back quickly

Fast pipelines are crucial. You want to encourage your developers to push their code early and often. That can only happen if they can get feedback on their changes quickly.

Parallelization reduces the build time. Ideally, all the tasks that aren’t dependent on each other should run in parallel. The CI should do that transparently for you.

 - aggregate:
      - put: dev-container
        params:
          << : *docker-params
          build: git
          dockerfile: git/Dockerfile.build
      - put: serverspec-container
        params:
          << : *docker-params
          build: git/serverspec
          dockerfile: git/serverspec/Dockerfile.serverspec
Figure 2: Building multiple images in parallel

Quick feedback isn’t only achieved with speed. Tasks that fail randomly can kill the feedback loop. This was a problem for us, as we often had to trigger unreliable tasks over and over until they worked.

In our experience, the best way to ensure reliable builds is to use containers to run each task in isolation. Keeping a persistent workspace across builds, as Jenkins does, can save you time but it’s a recipe for flakiness.

You need to create the containers that are used in the pipeline, with the binaries and packages for all the defined steps. We prefer building them as part of the pipeline itself. Even if they’re not production containers, you should still follow best practices to build them. There are plenty of articles available on how to write high-quality images.

FROM node:10.11-stretch

ENV CONCOURSE_SHA1='f397d4f516c0bd7e1c854ff6ea6d0b5bf9683750' \
    CONCOURSE_VERSION='3.14.1' \
    HADOLINT_VERSION='v1.10.4' \
    HADOLINT_SHA256='66815d142f0ed9b0ea1120e6d27142283116bf26'

SHELL ["/bin/bash", "-o", "pipefail", "-c"]

RUN apt-get update && \
    apt-get -y install --no-install-recommends sudo curl shellcheck && \
    curl -Lk "https://github.com/concourse/concourse/releases/download/v${CONCOURSE_VERSION}/fly_linux_amd64" -o /usr/bin/fly && \
    echo "${CONCOURSE_SHA1} /usr/bin/fly" | sha1sum -c - && \
    chmod +x /usr/bin/fly && \
    curl -Lk "https://github.com/hadolint/hadolint/releases/download/${HADOLINT_VERSION}/hadolint-Linux-x86_64" -o /usr/bin/hadolint && \
    echo "${HADOLINT_SHA256} /usr/bin/hadolint" | sha1sum -c - && \
    chmod +x /usr/bin/hadolint && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*
Figure 2: a container to build JavaScript applications

Testing containers

We always use ServerSpec to test our containers following a TDD approach. This applies both to the containers that are only used in the pipeline, as well as to the containers that will run on production.

require_relative 'spec_helper'

describe 'dev-container' do
  describe 'node' do
    describe file('/usr/local/bin/node') do
      it { is_expected.to be_executable }
    end

    [
      [:node, /10.4.1/],
      [:npm, /6.1.0/]
    ].each do |executable, version|
      describe command("#{executable} -v") do
        its(:stdout) { is_expected.to match(version) }
      end
    end

    describe command('npm doctor') do
      its(:exit_status) { is_expected.to eq 0 }
    end
  end

  describe 'shell' do
    %i[shellcheck].each do |executable|
      describe file("/usr/bin/#{executable}") do
        it { is_expected.to be_executable }
      end
    end
  end
end
Figure 3: testing that the development container has the right versions]

Getting this to work can be very challenging, as it requires running Docker in Docker. We have an example that you can use if you want to do this in Concourse. Use it with this entry point so that permissions are set properly.

platform: linux
inputs:
  - name: git
run:
  path: bash
  dir: git/serverspec
  args:
  - -c
  - ./entrypoint.sh ./run


Using containers helps with reliability and reproducibility, but it can hinder the speed of your pipeline because you have to repeat steps like downloading dependencies and running build scripts multiple times. Caching dependencies, such as npm packages, is an acceptable tradeoff.

Low maintenance and scalability

Having the pipelines reflected in code helps to keep the maintenance cost low. Our pipelines mostly call shell scripts that also work locally. This allows us to test the steps before pushing, which allows for a faster feedback loop. In many ThoughtWorks projects, it is common to find a go script for this, as explained in more detail in this article.

As your pipelines grow, and as you add new pipelines, complexity grows. Our old pipelines had so much copy-pasted code that changing anything took a lot of effort.

One way to avoid duplication is to parametrize tasks. You can write the definition of a task in a file, and use it from the pipeline. It is configured by passing variables to it. We use this extensively. We have different linters that we want to run as separate tasks, so we built a generic linter task, such as:

platform: linux
inputs:
  - name: git
caches:
  - path: git/node_modules
run:
  path: sh
  dir: git
  args:
  - -exc
  - |
    npm i
    ./go linter-${TARGET}


Then, we can integrate this into a pipeline like this:

 - task: lint-sh
      image: dev-container
      params:
        << : *common-params
        TARGET: sh
      file: git/pipeline/tasks/linter.yml


We have a library of tasks to reuse code across pipelines. It works well because of the effort we made in keeping the pipelines consistent. Some tasks, like updating the pipeline, almost never change and can be freely reused.

We try to avoid over-abstracting, however. Sometimes is better to just have two different tasks and reduce the coupling between them.

The same applies to the containers where the tasks are executed. You have to balance between trying to reuse containers and coupling different pipelines through these images. One container that we reuse is the one that runs the ServerSpec tests.

Visualization

One antipattern that we saw in our old pipelines was running multiple checks together in the same step. That forces you to read through multiple logs to find what failed. Instead, we clearly divide the steps, which helps identify errors more quickly.

Failing pipeline

Consistency also helps with visualization. The effort we spent building shared tasks has made the pipelines more similar to each other.

Lastly, having a dashboard for your pipelines gives the team one place to check everything’s working correctly, and it serves as an information radiator.  

How it all fits together

At the start of our journey, we decided to recreate our build pipelines and to switch to a new CI tool, operated by the team.

In order to avoid replacing one snowflake with another, we used Infrastructure as Code from the beginning. Specifically, we use Terraform to provision all the infrastructure located on AWS.

We set it up on EC2 instances, using AutoScaling groups. For the database, we took Amazon RDS because we did not want to set it up ourselves.

We hypothesized that this investment would help us deliver software faster while spending less time maintaining and debugging pipelines. How did it work in practice?

The good

Being code driven means that our infrastructure is always in a known state. Owning the solution allows us to adapt it to our needs. One such case was replacing PhantomJS, which is no longer actively developed, with Headless Chrome to run frontend tests.

Building everything in isolated containers makes for safe and predictable builds. We know exactly which version of which binary is being used in each step. The containers themselves are tested as well as part of the pipeline.

Thanks to our effort in reusability, creating new pipelines is a breeze. Refactorings are easier to apply and also happen more often. The new pipelines are a lot easier to debug as well.

The bad

The thing we liked the least about Concourse is the documentation. It’s pretty thin in many places. Caching, for example, isn’t explained particularly well.

Switching tools takes time getting used to. Jenkins can restart a job from scratch, whereas Concourse needs a new version of a resource to be there. One big drawback of Concourse is that you cannot pass any artifact between jobs, which forces you to persist anything that you want to keep in external storage, such as S3. This makes working with intermediate artifacts quite inconvenient.

We like the dashboard quite a lot, particularly now that it’s the default page, starting from Concourse 4. However, we do miss support for CCMenu. Tasks that are triggered manually aren’t easy to visualize in the pipeline, and you cannot see who triggered certain builds.

The ugly

Operating infrastructure needs more effort for the team. Concourse is a lot more complex to set up than Jenkins, which basically only needs a war file. You can see the complexity of the architecture in the official page. We’ve had downtimes related to the CI Infrastructure that, while serving as a learning experience, have impacted our productivity.

Upgrading from Concourse 3 to Concourse 4 was more painful than expected. The database schema changed in between the two releases. In the end, we ended up spinning up a new instance and discarding all the build history. On the other hand, being responsible for our own tool has encouraged us to be more proactive in keeping it up to date.

Conclusion

All in all, the positives greatly outweigh the negatives. We’ve reached a state where creating new pipelines for new services and infrastructure can be done quickly and reliably. We’re a lot happier with the state of our build infrastructure than we were before.

If you want to check more details, we have prepared a repository with some sample code that can be used as a starting point.

Technology Radar

Don't miss our opinionated guide to technology frontiers.

Subscribe
相关博客
云

Using Pipelines to Manage Environments with Infrastructure as Code

Kief Morris
了解更多
持续交付

5 Traits of a Good Delivery Pipeline

Marcos Brizeno
了解更多
持续交付

Enabling Trunk Based Development with Deployment Pipelines

Vishal Naik
了解更多
  • 产品及服务
  • 合作伙伴
  • 洞见
  • 加入我们
  • 关于我们
  • 联系我们

WeChat

×
QR code to ThoughtWorks China WeChat subscription account

媒体与第三方机构垂询 | 政策声明 | Modern Slavery statement ThoughtWorks| 辅助功能 | © 2021 ThoughtWorks, Inc.