Since the time Mingle was first released as a SaaS product, it has gone through a lot of changes. Most of those changes have been towards enabling Mingle to work in distributed environments on the cloud. In our journey, we learnt a lot about the cloud and how to build for it instead of fighting it. I will try to unfold our progress and showcase a tool - EbDeployer, which we built along the way (to enable continuous deployments on AWS).
Being on the cloud
Mingle has been on the cloud since December 2012. Since our first launch, we have been building more functionality and adapting it to fit the cloud. As explained in ‘how we moved to the cloud’ article, we built features along with cloud adaptability for Mingle in the last few months. Most of what we learnt helped us understand distributed computing - especially with AWS, better.
Using AWS specific tools
As we showed in the previous article, we used CloudFormation to build some of our services. Further to CloudFormation work, AWS has gone ahead and built a complete Web application stack infrastructure - they call it Elastic Beanstalk.
Elastic Beanstalk (EB) provides the following features:
-
Complete stack management - Computing, Storage, Database, Load Balancer all connected together
-
Stack lifecycle management - The application is deployed for the entire stack and only reports success when all the components are healthy
-
Built-in monitoring and log retrieval
-
Simple, configurable scaling and deployment
Elastic Beanstalk gives us everything in terms of managing the application lifecycle on AWS. We found that if we are able to build Mingle into self-contained application without the need of manual orchestration, EB would be of great benefit for us.
Migrating to Elastic Beanstalk
Given our better understanding of AWS infrastructure, we started by moving our simple web services to EB. These services were independent enough that they would not affect Mingle’s performance or availability. We picked Cycle Time Analytics(CTA) as one of the candidates to serve on EB. Since this service was small and independent, we were able to learn EB better when deploying it.
Pushing these services on to EB is straightforward enough. However, we had a few goals in mind, in addition to just deployment:
-
Automated Deployments to allow for Continuous Delivery(CD)
-
Zero Downtime for the user
-
Rollback and no data loss
-
One-click Seamless deployments
-
Automatically running smoke tests - to ensure that the new deployments did not break anything.
Building EBDeployer
Pengchao Wang started building what we now call EbDeployer - a tool for automated Continuous Delivery on AWS Elastic Beanstalk. EbDeployer automates blue-green deployments for EB. EbDeployer was built with the above goals in mind. Since it has been built, we have been able to deploy multiple times a day, with automated deployments through our CD pipelines.
Blue-Green Deployments
EbDeployer is based on the ideas from Blue-Green Deployments by Martin Fowler.
Here is a brief explanation of what it means to have blue-green deployments in your infrastructure:
While working on some form of Continuous Delivery, the problem of keeping the application up and running, while simultaneously updating it with new features, is common. Blue-green deployments allow you to have a clone of the production environment, so you can deploy to the clone, verify the functionality and switch it with the production environment. This way, new features are available to the users without any perceived downtime. The diagram below illustrates how it would work with a rack of servers.
Although the basic idea seems simple, without proper tools, such deployments could be a challenge. Some of the human aspects involved in maintaining this system and doing it accurately make it error prone.
A few reasons why blue-green deployments are hard to maintain:
-
High cost of duplicate infrastructure
-
Maintenance of the duplicate infrastructure
-
Ensuring environments and data are in sync
-
Dead cost of the inactive infrastructure when it does not serve users
-
Human involvement in putting pieces together and running tests that make it error prone
EB addresses, these hard to maintain issues by design, as follows:
-
EB is available on the cloud - no sunk costs in maintaining your own infrastructure
-
EB automatically scales the environment based on load - thus reducing cost when the load on them is reduced
-
EB manages the entire development stack
-
EB is built on top of CloudFormation - thus providing protection around eventual consistency issues and more reliable state of the system
-
Inactive environments are tuned to use minimum resources based on load - hence saves on cost
EbDeployer - why we built it
EbDeployer brings harmony by automating blue-green deployments for EB.
EbDeployer leverages the low friction in EB to allow for maintaining and deploying to blue-green environment stacks. EbDeployer provisions environments using the EB API and builds active and inactive environments. Once these are built, the following deployments always happen on the inactive environment. They are verified through smoke tests and then switched with the active environment - thus achieving hassle free blue-green deployments.
EbDeployer manages to perform all these actions without any burden on the developer. A simple configuration file is all it takes to setup your environment components. Some of these components can be pre-built if that is preferred. These components can be stacked and connected the way your application needs them.
A sample configuration looks as follows:
application: eb-deployer-simple-java
common:
region: us-west-1
smoke_test: >
Timeout.timeout(600) do
until `curl http://#{host_name}`.include?('Hello, World')
sleep 5
end
end
option_settings:
- namespace: aws:autoscaling:launchconfiguration
option_name: InstanceType
value: m1.small
# You can define environment here. Each environment can overriden any common settings
environments:
dev:
strategy: inplace-update
prod:
option_settings:
- namespace: aws:autoscaling:asg
option_name: MinSize
value: "2"
inactive_settings:
- namespace: aws:autoscaling:asg
option_name: 'MinSize'
value: "1"
More examples of detailed configuration can be found on EbDeployer website.
EbDeployer also provides a standard template, thus making it really easy to use and deploy to AWS. A typical blue-green environment built with EbDeployer looks as shown in the picture below.
Smoke testing and deployment strategies
You can choose different blue-green strategies depending on your use case. For example, in a development or testing stack, we would like to destroy the inactive one to save on cost/hassle. So after testing the deployment, you can choose to terminate the old one.
EbDeployer also allows you to choose the deployment style - phoenix mode or inplace update.
-
If there are no changes to configuration of the stack, inplace updates are faster
-
Phoenix mode can be useful if you want to ensure a clean environment without any stale state whatsoever
Delivering EbDeployer
Pengchao Wang(WPC) and I presented on how EB becomes better with EbDeployer at the aws re:invent 2013 (Our talk starts at 27m 45s). Feel free to browse the presentation below and use EbDeployer for your cloud app.
A few new features have been added and the community is finding it useful. EbDeployer has now been forked a number of times and has also inspired others to build similar tools - like cf-deployer.
More documentation and open sourced code can be found on our EbDeployer website.
Check out other related posts on the topic of Continuous Delivery on the Cloud: