CODEWITHSUNDEEP
node.jsamazon-elastic-beanstalkamazon-cloudwatchaws-codepipeline
mayanxoni
mayanxoni

Reputation: 83

Incorrect application version found on all instances

I have a Node.js application (written in JavaScript) deployed on an Elastic Beanstalk environment using Codepipeline. Which means, on every new git push, Codepipeline deploys a new version of the application on to the specified Elastic Beanstalk environment. It was working great so far, but lately, every new version is failing with the following error:

Deployment completed, but with errors: During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version. Failed to deploy application. Unsuccessful command execution on instance id(s) 'i-XXXXX'. Aborting the operation. [Instance: i-XXXXX] Command failed on instance. Return code: 1 Output: Engine execution has encountered an error.. Instance deployment failed. For details, see 'eb-engine.log'.

I checked all the logs in CloudWatch, particularly the mentioned 'eb-engine.log' too, and found the following logs:

`

Platform Engine finished execution on command: app-deploy Running command /bin/sh -c /opt/aws/bin/cfn-init -s arn:aws:cloudformation:us-east-1:XXXXXXXXXXX:stack/awseb-e-vcrthwt2fn-stack/XXXXXXXXXXX -r AWSEBAutoScalingGroup --region us-east-1 --configsets > Infra-EmbeddedPreBuild Error occurred during build: Command 01_enable_swap failed An error occurred during execution of command [app-deploy] - [PreBuildEbExtension]. Stop running the command. Error: EbExtension build failed. Please refer to /var/log/cfn-init.log for more details. Executing cleanup logic CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment > failed. For details, see 'eb-engine.log'.","timestamp":1698824033276,"severity":"ERROR"}]}]} Running command /bin/sh -c /opt/aws/bin/cfn-elect-cmd-leader --stack arn:aws:cloudformation:us-east-1:XXXXXXXXXXX:stack/awseb-e-vcrthwt2fn-stack/XXXXXXXXXXX --command-name ElasticBeanstalkCommand-AWSEBAutoScalingGroup --invocation-id 89ba36e0-3e4a-439f-b6ff-eff89c74bd2f --listener-id i-XXXXXXXXXXX --region us-east-1 Instance is Leader. Executing instruction: stopSqsd This is a web server environment instance, skip stop sqsd daemon ... Executing instruction: PreBuildEbExtension Starting executing the config set Infra-EmbeddedPreBuild. Downloading: bucket: elasticbeanstalk-us-east-1-XXXXXXXXXXX, object: /resources/environments/e-vcrthwt2fn/_runtime/_versions/PROJECT_NAME/ code-pipeline-1698824021435-c67f26f41d43caabae9c88c869142f218534a3d3 Download successful19343927bytes downloaded Executing instruction: ElectLeader Running leader election for instance i-XXXXXXXXXXX... Calling the cfn-elect-cmd-leader to elect the command leader. envBucket: elasticbeanstalk-us-east-1-XXXXXXXXXXX Using manifest file name from command request Manifest name is : manifest_1698824028934 Download app version manifest Downloading: bucket: elasticbeanstalk-us-east-1-XXXXXXXXXXX, object: /resources/environments/e-vcrthwt2fn/_runtime/versions/manifest_1698824028934 Download successful151bytes downloaded Trying to read and parse version manifest... Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:XXXXXXXXXXX:stack/awseb-e-vcrthwt2fn-stack/XXXXXXXXXXX -r AWSEBBeanstalkMetadata --region us-east-1 checking whether command app-deploy is applicable to this instance... this command is applicable to the instance, thus instance should execute command Engine command: (app-deploy) Downloading EB Application... Region: us-east-1 envID: e-vcrthwt2fn Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:XXXXXXXXXXX:stack/awseb-e-vcrthwt2fn-stack/XXXXXXXXXXX -r AWSEBAutoScalingGroup --region us-east-1 Starting... Starting EBPlatform-PlatformEngine reading event message file Engine received EB command cfn-hup-exec

`

Although I tried several solutions like:

My last resort has been to delete the environment and pipeline and build new ones everytime I push my code (like the EBS blue-green deployment strategy). To save my time there, I ended up using a service called: Render, which shockingly works like a charm with the same application that's failing to deploy on AWS. The only catch is that it shuts down the instances in case of no activity and increases delays in response times due to cold boot.

Any help will be greatly appreciated, because this problem has eaten so many days out of development time.

Upvotes: 4

Views: 484

Answers (1)

clay
clay

Reputation: 6017

I too saw this error recently and have changed my deployment strategy in ElasticBeanstalk environment -> Configuration -> Updates, Monitoring and Logging, from Rolling to Immutable. This change has not seen the same errors you mentioned, so I'm considering it a fix.

I think the Rolling updates sometimes reused the existing instances and, for whatever reason, the application version did not work correctly and Beanstalk complained. (Maybe an interaction with the health checks? Or some failed/slow startup?) With Immutable deployments, A new EC2 instancE is launched with only the new app version, and all parts of the deployment must be ready before the scaling groups are switched on the Load Balancer. While it's more "cloud heavy" and maybe a little slower, this consistency was better for my project and I'm sticking with it.

You'll appreciate the work you did on blue-green deployments though. When you need to upgrade the platform for new NodeJS versions it needs to be a new environment rather than an automatic upgrade.

Upvotes: 0

Related Questions