There are many well documented benefits of continuous delivery such as lower risk releases, lower costs, higher quality and faster time to market.
With streaming data pipelines there are other benefits too. If we are consuming data from an upstream source taking the pipeline offline to update it can mean (depending on our architecture and how long we've been offline for) that we miss data or at minimum have additional load to process when we come back online. These spikes after being offline can impact data latency SLAs. Downtime can also strain downstream applications and consumers of the data. Continuous delivery allows us to make safe incremental changes without having to take the pipeline offline.
For teams moving to a continuous delivery model for the first time it can be a little daunting.
Some of the practices like trunk based development require a shift in how developers write code. Everything we check in will now end up in production. We have to adapt our developer workflow and path to production accordingly.
Here is a sample developer workflow and build pipeline for continuous delivery/deployment:
Key components to making this work:
We have used similar setups to continuously deploy changes to streaming data pipelines running under heavy load with no downtime or impact to other systems.
Disclaimer: The statements and opinions expressed in this article are those of the author(s) and do not necessarily reflect the positions of Thoughtworks.