EbDeployer - Automate your Deployments on AWS Elastic Beanstalk

Using AWS specific tools

As we showed in the previous article, we used CloudFormation to build some of our services. Further to CloudFormation work, AWS has gone ahead and built a complete Web application stack infrastructure - they call it Elastic Beanstalk.

Elastic Beanstalk (EB) provides the following features:

1. Complete stack management - Computing, Storage, Database, Load Balancer all connected together

2. Stack lifecycle management - The application is deployed for the entire stack and only reports success when all the components are healthy

3. Built-in monitoring and log retrieval

4. Simple, configurable scaling and deployment

Elastic Beanstalk gives us everything in terms of managing the application lifecycle on AWS. We found that if we are able to build Mingle into self-contained application without the need of manual orchestration, EB would be of great benefit for us.

Migrating to Elastic Beanstalk

Given our better understanding of AWS infrastructure, we started by moving our simple web services to EB. These services were independent enough that they would not affect Mingle’s performance or availability. We picked Cycle Time Analytics(CTA) as one of the candidates to serve on EB. Since this service was small and independent, we were able to learn EB better when deploying it.

Pushing these services on to EB is straightforward enough. However, we had a few goals in mind, in addition to just deployment:

1. Automated Deployments to allow for Continuous Delivery(CD)

2. Zero Downtime for the user

3. Rollback and no data loss

4. One-click Seamless deployments

5. Automatically running smoke tests - to ensure that the new deployments did not break anything.

​Building EBDeployer

Pengchao Wang started building what we now call EbDeployer - a tool for automated Continuous Delivery on AWS Elastic Beanstalk. EbDeployer automates blue-green deployments for EB. EbDeployer was built with the above goals in mind. Since it has been built, we have been able to deploy multiple times a day, with automated deployments through our CD pipelines.

Blue-Green Deployments

EbDeployer is based on the ideas from Blue-Green Deployments by Martin Fowler.

Here is a brief explanation of what it means to have blue-green deployments in your infrastructure:

While working on some form of Continuous Delivery,  the problem of keeping the application up and running, while simultaneously updating it with new features, is common. Blue-green deployments allow you to have a clone of the production environment, so you can deploy to the clone, verify the functionality and switch it with the production environment.  This way, new features are available to the users without any perceived downtime. The diagram below illustrates how it would work with a rack of servers.

Although the basic idea seems simple, without proper tools, such deployments could be a challenge. Some of the human aspects involved in maintaining this system and doing it accurately make it error prone.

A few reasons why blue-green deployments are hard to maintain:

1. High cost of duplicate infrastructure

2. Maintenance of the duplicate infrastructure

3. Ensuring environments and data are in sync

4. Dead cost of the inactive infrastructure when it does not serve users

5. Human involvement in putting pieces together and running tests that make it error prone

EB addresses, these hard to maintain issues by design, as follows:

1. EB is available on the cloud - no sunk costs in maintaining your own infrastructure

2. EB automatically scales the environment based on load - thus reducing cost when the load on them is reduced

3. EB manages the entire development stack

4. EB is built on top of CloudFormation - thus providing protection around eventual consistency issues and more reliable state of the system

5. Inactive environments are tuned to use minimum resources based on load - hence saves on cost

EbDeployer - why we built it

EbDeployer brings harmony by automating blue-green deployments for EB.

EbDeployer leverages the low friction in EB to allow for maintaining and deploying to blue-green environment stacks. EbDeployer provisions environments using the EB API and builds active and inactive environments. Once these are built, the following deployments always happen on the inactive environment. They are verified through smoke tests and then switched with the active environment - thus achieving hassle free blue-green deployments.

EbDeployer manages to perform all these actions without any burden on the developer.  A simple configuration file is all it takes to setup your environment components. Some of these components can be pre-built if that is preferred. These components can be stacked and connected the way your application needs them.

A sample configuration looks as follows: 

application: eb-deployer-simple-java

common:
  region: us-west-1
  smoke_test: >
    Timeout.timeout(600) do
      until `curl http://#{host_name}`.include?('Hello, World')
        sleep 5
      end
    end


  option_settings:
    - namespace: aws:autoscaling:launchconfiguration
      option_name: InstanceType
      value: m1.small

# You can define environment here. Each environment can overriden any common settings
environments:
  dev:
    strategy: inplace-update
  prod:
    option_settings:
      - namespace: aws:autoscaling:asg
        option_name: MinSize
        value: "2"

    inactive_settings:
      - namespace: aws:autoscaling:asg
        option_name: 'MinSize'
        value: "1"

More examples of detailed configuration can be found on EbDeployer website.

EbDeployer also provides a standard template, thus making it really easy to use and deploy to AWS. A typical blue-green environment built with EbDeployer looks as shown in the picture below.

Smoke testing and deployment strategies

Once active/inactive environments have been set up, the blue-green environments are ready.

Subsequent deploys will continue to deploy to the inactive one and switch over once it passes the set of smoke tests. Yes, EbDeployer allows you to run smoke tests along with your deployment.

You can choose different blue-green strategies depending on your use case. For example, in a development or testing stack, we would like to destroy the inactive one to save on cost/hassle. So after testing the deployment, you can choose to terminate the old one.

EbDeployer also allows you to choose the deployment style - phoenix mode or inplace update.

• If there are no changes to configuration of the stack, inplace updates are faster

• Phoenix mode can be useful if you want to ensure a clean environment without any stale state whatsoever

​Delivering EbDeployer

Pengchao Wang(WPC) and I presented on how EB becomes better with EbDeployer at the aws re:invent 2013 (Our talk starts at 27m 45s). Feel free to browse the presentation below and use EbDeployer for your cloud app.

Since its initial build, we have moved a number of our applications, including the core Mingle application to EB. We have been doing daily, as well as automated deployments, to all our environments in blue-green fashion.

A few new features have been added and the community is finding it useful. EbDeployer has now been forked a number of times and has also inspired others to build similar tools - like cf-deployer.

More documentation and open sourced code can be found on our EbDeployer website. 

Check out other related posts on the topic of Continuous Delivery on the Cloud:  

1. Implementing Blue Green Deployments with AWS

 

2. Continuous Deployment of a Static Website to AWS with Snap CI

 

3. Deploying to AWS using Elastic Beanstalk "