AWS Beanstalk: None of the Instances are sending data, and can not ssh to ec2

Question

I have encountered this problem

None of the Instances are sending data

when deploying a new version of my app on beanstalk, the only diff between the two versions are minor and I'm quite sure it's not the cause. This is what I see:

• I can not get logs from beanstalk console
• I can not ssh to EC2 instance of this beanstalk configuration (although the status of that instance is 'running'.
• Last time I encountered the same issue with a ec2.micro instance, it got solved when I upgrade to ec2.small. What I thought is probably the resource usages make it unresponsive (weird though, since it's only deployment, not even serving any traffic). I don't want to upgrade (again) without really understanding what's happening here.
• The cpu pattern is like 80% -> 60% -> 20% in during the first 5mins of deployment, then stay at 10% constantly afterwards.

The only way that I can get some server log is to get system log from aws console, and here it is:

Here is the log: https://pastebin.com/PWWjPr3b

Here is what I saw when I issue a ssh:

OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/okidogi/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to ec-******.eu-west-2.compute.amazonaws.com 
[35.177.76.128] port 22.
debug1: Connection established.
debug1: identity file aws-eb type 1
debug1: key_load_public: No such file or directory
debug1: identity file aws-eb-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4

Then it's pending there.

Appreciate if someone can help on this.

Answer 1

Feng, glad to hear that did the trick for you. However, that idea was more of a diagnostic test than a long term solution. Since switching to Immutable worked for you - this would point to the possibility that your build process requires more memory than is available on a (already running) small unit like a t2.micro or t1.micro. By using an Immutable strategy you begin each deploy with a new instance that has more available resources than one that's already consumed with running your app.

This is a problem that is common but hard to diagnose as it presents many different ways depending on platform and framework. You can read more here: https://medium.com/@deanslamajr/an-insufficient-memory-deployment-failure-d9f1cb9b5c0.

My prefered way to address this is through swap memory as I've outlined in an answer to a similar question: AWS EB deploying Node app: failed to run npm install

I would advise trying the .ebextensions strategy mentioned there and also switching back to All at once deployment strategy to test if that truly solves your problem.