Deploy Service by EbDeployer

infrastructure
Posted by Xiao Li

1 June 2015

Back in 2013, our team member WPC built EbDeployer to create a new, automated, zero-downtime deployment solution for internal services.

The goal of this blog post is to explain four levels of concepts structured in EbDeployer to help you when deploying services:

- EB App (ElasticBeanstalk application)
  - Environment
    - Component
      - EB Env (ElasticBeanstalk environment)

Basic relationships between the above concepts:

eb app and envs relationship

An EB app deploys one package usually mapping to one codebase. We use EbDeployer deploy one service to one EB app. For example, the following screenshot is a deployment pipeline for our internal service in Go:

deployment pipeline example

For multiple services, we will set up multiple deployment pipelines to deploy each service to its own EB app, the following screenshot shows three of our internal services deployment pipelines:

service deployment pipelines

Because we built in zero downtime for each service deployment, all services’ deployments can be done as needed. Please note that “Environment” and “Component” are not ElasticBeanstalk concepts, but rather they are concepts we implemented in EbDeployer to help organise and facilitate service development and deployment.

Environment

An environment refers to one of the development, test, staging, or production environments, etc. For supporting delivery of high-quality services to end users, we launch development, testing, and staging environments and dogfood the changes before progressing to a production environment. Different teams may use different names or launching environments for different purposes, but the basic idea and requirements are the same: we need to launch non-production environments that are close to, or similar to, the production environment.

While we always have the ability to duplicate a production environment exactly, EbDeployer is also designed for productivity. We believe developers should have control of constructing environment resources. In the EbDeployer, there is a common section for you to define configurations that are applied to all environments, and then environment-specific configurations for you to overwrite any of common configurations as well. We apply different environment configurations to reduce costs and build-time, while still keeping environments similar enough to the production environment.

For example, we commonly enable blue-green (strategy option in EbDeployer configuration) deployments because of the practice’s inherent zero downtime for testing, staging and production environments: but we use inplace-update strategy for redeploying our development environment as fast as possible. We also turn off phoenix_mode option for development, testing or (sometime staging environment) for faster build cycle time, and only turn on phoenix_mode in production. Autoscaling group configuration is another common case, as you won’t have same load with your production environment, non-proudction environments will need different autoscaling group configurations.

EbDeployer environments configuration example snippet:

Resources are shared inside an environment, but replicated across different environments. EbDeployer creates an AWS CloudFormation stack for each environment as defined by the resources template configuration. The stack name will be EB app name + environment name.

ElasticBeanstalk’s environment concept is not the same as our high-level concept. We think EB Env is a building block for our service, hence, EbDeployer has implemented its environment concept on top of the EB Env by adding an environment name to the EB Env name, for example: prod is environment name of EB Env name prod-web-a-3cbbc22 which is created by EbDeployer. EB Env name has a length limit (4 to 23 chars), so when you name your environment, please try to use short names. For example, we uses “stg”, “prod” as our staging and production environment names to make the most of limited character space.

Components

When you share same model/code between your web request processing and background job processing, how do you deal with deployment? The EbDeployer component is designed for you to create different types of EB Envs inside the same EB app to do different types of jobs. Because most of the infrastructure should be shared between these EB Envs (in our cases, we share all infrastructure resources and the only difference between these types of EB Environments are our configurations of our Rails app), we think EbDeployer should handle this case, and designed it with that in mind. Similar as with environment configurations, you can make component-specific configurations to overwrite environments and common configurations.

As the environment replicates everything, it also replicates components. Hence you get the simplest configuration for multiple components deploying to multiple environments individually. Similarly, EbDeployer adds component names to EB Env names, so as we noted earlier, you should consider using short names for your components.

For example, the following snippet is components configuration of our one internal service:

In this example, we have two components: one background processing component named “bg” and one web request processing component named “web”. There is no specific configuration for web component, hence it inherits all configurations from “common” and “environment” configurations. The background component is a little more complex, as it cannot run in parallel, so we changed autoscaling MinSize and MaxSize to “1” so that it would only launch one EC2 instance in EB Env. There is also a specific environment variable for a background component named LAUNCH_MESSAGE_WORKER, which enables our Rails application to launch a message-processing worker to do background jobs. We also changed our default deployment strategy (blue-green deployment) for the bg component because blue-green would launch two EB Envs for deployment, which will then cause two EC2 instances to process the background jobs simultaneously.

One thing you may notice is we’re not using ElasticBeanstalk worker tier for background processing. EbDeployer does support deploying worker tier, and we will continue to support it. But in our experience, we don’t like the way the ElasticBeanstalk worker tier works. It tends to complicate things. Instead, we use a web server tier to launch the EB Env processing our background job. We control the background job processing by our code, and built message-sqs.

Last thing you need to know about Component is you don’t have to use it. If you only had one component, you don’t need to have a components section defined in your EbDeployer configuration.

EB Env (ElasticBeanstalk Environment)

As I’ve covered above, EB Env is our building block. It is used to construct components and environments. We also use it to build blue-green deployment strategy for any of the components or environments. So while one component may have one or two EB Envs deployed: one for the inplace-update strategy, and another for the blue-green deployment strategy. When we started work on EbDeployer, there was no Deploying Application Versions in Batches. So far we have not tried it in any of our production services, because its benefits are not obvious enough for us to replace blue-green deployments. But if you had any experience with you, we will like to hear it and open to support it.

But to see what we’re currently using EbDeployer for, please see below. The following screenshot shows an example of an EB app that has nine EB Envs (three environments, two components, and blue-green for web component) deployed by EbDeployer.

elasticbeanstalk environments deployed by EbDeployer


comments powered by Disqus