Infrastructure Pipelines

Infrastructure-as-code tools like Terraform recently have become popular (see Infrastructure as code | Technology Radar). One main promise of such tools is automation. Still, we rarely see Continuous Delivery implemented for infrastructure code. This article advocates the usage of  Continuous Delivery pipelines for infrastructure provisioning (see Pipelines for infrastructure as code | Technology Radar) and provides some best practices on how to get started. Our target group is developers who - like us - love to drive automation.

Why?

Accidents happen

This slightly adapted quote shows how small changes in a workflow can have dramatic effects when it comes to infrastructure code. Therefore we should reflect on why we want to have automation in the first place. It:

• Keeps things reproducible
• ​Reduces time spent on documenting manual steps
• Provides reliable processes
• Teams focus on the actual change rather than the steps to apply them
• Improves traceability of changes

Our goal should be to fully automate the process of applying infrastructure changes. The application development world saw the move to automated build and delivery pipelines a few years ago (Automated deployment pipeline | Technology Radar), now the infrastructure world is catching up :)

Infrastructure code is CODE!

A good take on automating your infrastructure code is treating it the same way as your application code. Therefore some steps are necessary before automating and reaching the familiar workflow of: write code, commit, push & trigger pipeline.

Prepare your Infrastructure Code for Automation

Before you start building infrastructure pipelines, make sure your infrastructure code is ready for automation. We have therefore provided a small checklist in the appendix of this article. Here is a rough overview.

✔ Keep it in version control: Changes in the code base will be the trigger for all further steps in your pipelines.

✔ Modularize your infrastructure code: Building your infrastructure code in small reusable modules allows for re-use of code and facilitates testing. 

✔ Shape your infrastructure code around your applications: Keeping your infrastructure code together with your applications code helps you in building smaller blocks that are related to the applications and changes in either parts of the code will be easily managed.

✔ Tests: Tests are the safety net in any automation. 

✔ Staging: Having the same build steps across multiple environments, ensures that when changes are promoted to production, they behave as expected.

✔ Immutable Infrastructure: If you have the choice, try to use immutable infrastructure resources - doing so comes with the security that rebuilt infrastructure will always be the same.

✔ Your infra code should be idempotent, no matter how many times you apply the same code, there should be no changes beyond the result of the initial application.

Automating your infrastructure provisioning

Now that the infrastructure code is ready: How do we get started with pipelines? Generally, we can follow the same steps that we use for pipelining application code:

Design your infrastructure pipeline

With the design of an application pipeline in mind, we can now start designing an infrastructure pipeline. We expect our pipeline to be:

• Reliable - a reproducible process to provision our infrastructure
• Fast - quick feedback cycles if the pipeline succeeds or fails
• Specific - concrete feedback on what went wrong when errors happen

To achieve these goals our infrastructure pipeline should look something like this:

Prepare stage

In this stage we want to validate our infrastructure code. The outcome of the stage is a package that contains all the artifacts we need for applying our infrastructure changes, e.g. validated infrastructure code.

Pre-Production stage

The purpose of this stage is to provision our infrastructure in a production-like environment. Therefore we generate a report of changes that will be made to our infrastructure, then we  apply these changes and finally we can test our (new) infrastructure.

Production stage

We repeat the exact same steps that we ran in the previous stage - but in our production environment - with the added safety of having run them in pre-production.

Considerations

Separate your infrastructure code into smaller repositories. Running pipelines for smaller portions of your infrastructure provides faster feedback cycles and reduces the blast radius when deployments go wrong.

Store and reuse any artifacts that become relevant again in subsequent steps e.g. terraform providers and terraform plan.

There must be no secrets in your infrastructure code.

A CI/CD system that can execute infra changes is a very powerful entity, make sure only the right people can access it and that it is handled securely. Apply least privilege principles to the roles that run your pipeline.

Make sure not to test things that your infrastructure-as-code tool already “tests” implicitly. There is no use in checking whether a resource has been created after the apply succeeded.

Linting and validating your code helps you have a consistent code shape.

You want to keep your testing pyramid balanced, so here is a hint on how a good test suite might look like:

• Unit tests: test the small piece(s) of logic that you have in your declarative code (e.g. usage of variables and conditions)
• Integration tests: test a Terraform module in isolation. Spin up the module’s infrastructure, assert it does what is expected and tear it down again
• End-to-end tests: Post-build you can test that once your infrastructure is provisioned, it does what it was built for. e.g. “can access my database with a newly provisioned user and the credentials stored in a vault?”

Summary and Outlook

Hopefully we convinced you that automating your infrastructure provisioning is a good idea. We hope that we helped you in shaping a good foundation of how a pipeline for your infrastructure code and pipelines can look like. We saw the benefits of using pipelines for running infrastructure code, described how to shape your code to make it ready for automation and provided a sample structure for infrastructure pipelines.

Your next steps could be to start with a pipeline to deploy a small part of your infrastructure and replicate that for other parts that need provisioning depending on how you have split your infrastructure code.