AWS Elastic Beanstalk goes down for a split second on deploy

Question

I'm testing out Elastic Beanstalk for my node.js express.js app. When I deploy an update, the app goes down for a few seconds, and I get 'bad gateway' from ngnix.

Is this because I only have 1 instance right now? How do I get this ready for production? As in zero downtime.

Answer 1

Try to switching the Deployment Policy to Immutable. You probably have it set to "All at Once" which will update the existing instance and cause you to experience downtime while that is happening.

With the Immutable deployment policy, Elastic Beanstalk will create a new instance and make sure it is healthy before switching over.

See http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.deploy-existing-version.html

Answer 2

Bruno has the right idea using Blue-Green deploys for zero down time.

In my experience once you've swapped urls it can take up to 20mins to drain all the active connections from your old environment to your new one. This is because even if your TTL is set low, browsers normally cache the DNS for up to 15mins.

I monitor the old environments until the connections die off.

An added benefit is if you notice something wrong with your new deployed build during this 20min cool off period, you can instantly 'roll-back' by hitting swap urls again to switch back to your previous environment ;)

Answer 3

You can get the same undesirable behavior even with an Elastic Beanstalk environment with more than 1 instance. The reason is that it might happen that all the instances will run the update process simultaneously.

If you want to avoid that downtime, there's a built-in feature on Elastic Beanstalk called swap URLs. It's a very simple implementation of a Blue-Green deployment strategy.

Basically, in a Blue-Green deployment strategy you have 2 sets of resources: the Blue resources and the Green resources. Assume that, initially, the Blue resources are receiving and handling production traffic. The strategy works as follows: (1) you deploy the new version of your application into the Green resources; (2) you run any warm-up procedures you might want, if you have any; (3) you use some sort of "traffic manager" to send production traffic into your Green resources; (4) once all the traffic has migrated from the Blue resources into the Green resources, you can optionally shut down the Blue resources.

In Elastic Beanstalk, you need 2 Environments for this. Let's assume that we have 2 environments with the following names and URLs:

• First environment called Env-Blue, with URL myapp-prod.elasticbeanstalk.com.

• Second environment called Env-Green, with URL myapp-staging.elasticbeanstalk.com.

Let's also assume that you have your own domain name, www.myapp.com, which points to myapp-prod.elasticbeanstalk.com. You are currently running v1.0 on Env-Blue, and you are working hard to get v1.1 out.

When you are ready to deploy v1.1, you deploy it into Env-Green -- and you do that because it's the current "staging" environment. Then you can connect to it, make sure it works exactly as you wanted.

Finally, you select both environments on Elastic Beanstalk and select the Swap URLs feature. Your environments will then look like this:

• First environment called Env-Blue, with URL myapp-staging.elasticbeanstalk.com, running the old v1.0.

• Second environment called Env-Green, with URL myapp-prod.elasticbeanstalk.com, running the new v1.1.

Since www.myapp.com still points to myapp-prod.elasticbeanstalk.com, your users, who are connecting to www.myapp.com, will now be sent to Env-Green, which is running v1.1. As the users refresh their DNS caches, they will gradually migrate to v1.1.

A very important final note: for this strategy to work properly, it's very important that you have configured an appropriate TTL for your domain records. If it's too long, your users might cache your DNS for a very long time, and it would take them a while to migrate. And during that period, you'd have to keep 2 sets of resources running (that you'd be paying for).