EC2 user data hanging on "Checking init scripts..." ubuntu php

Question

I have an EC2 setup with user data that sets up php on the machine, since adding command to install imagick my user data script has started hanging when i install the php-dev package.

I have tried: - Executing the package installations one by one - Running the script step by step in a terminal window on a fresh Ubuntu 18.04 instance (I always get a successful installation)

My user data:

sudo apt update
sudo apt install -y nginx
sudo apt install -y pkg-config
sudo apt install -y imagemagick
sudo apt install -y php7.2-fpm php-common php-mysql php-pear php-xml php-mbstring php-intl php-curl php-gd php-zip php-json php-simplexml php-dom
sudo apt install -y php-dev
sudo apt install -y php-imagick
sudo apt install -y composer
sudo apt install -y nodejs npm jq

Output of the EC2 every time i try this

...
[  102.919299] cloud-init[1250]: Unpacking php-dev (1:7.2+60ubuntu1) ...
[  102.949693] cloud-init[1250]: Selecting previously unselected package pkg-php-tools.
[  102.961152] cloud-init[1250]: Preparing to unpack .../29-pkg-php-tools_1.35ubuntu1_all.deb ...
[  102.972353] cloud-init[1250]: Unpacking pkg-php-tools (1.35ubuntu1) ...
[  103.033142] cloud-init[1250]: Setting up libarchive-zip-perl (1.60-1ubuntu0.1) ...
[  103.049041] cloud-init[1250]: Setting up libltdl-dev:amd64 (2.4.6-2) ...
[  103.061142] cloud-init[1250]: Setting up libtimedate-perl (2.3000-2) ...
[  103.074622] cloud-init[1250]: Processing triggers for install-info (6.5.0.dfsg.1-2) ...
[  103.219307] cloud-init[1250]: Setting up shtool (2.0.8-9) ...
[  103.233891] cloud-init[1250]: Setting up libarchive-cpio-perl (0.10-1) ...
[  103.246200] cloud-init[1250]: Setting up m4 (1.4.18-1) ...
[  103.257928] cloud-init[1250]: Setting up libcroco3:amd64 (0.6.12-2) ...
[  103.269881] cloud-init[1250]: Setting up libsys-hostname-long-perl (1.5-1) ...
[  103.283145] cloud-init[1250]: Setting up libmail-sendmail-perl (0.80-1) ...
[  103.295856] cloud-init[1250]: Processing triggers for libc-bin (2.27-3ubuntu1) ...
[  103.312484] cloud-init[1250]: Setting up autotools-dev (20180224.1) ...
[  103.328905] cloud-init[1250]: Setting up libssl1.1:amd64 (1.1.1-1ubuntu2.1~18.04.4) ...
[  103.470751] cloud-init[1250]: Checking for services that may need to be restarted...done.
[  103.495452] cloud-init[1250]: Checking for services that may need to be restarted...done.
[  103.502489] cloud-init[1250]: Checking init scripts...

I have found EC2 `UserData` execution hangs on `Checking init scripts...` but the OP of that question wasn't using any similar packages to me, nor was the question answered.

Answer 1

I had the same problem, and changing the AMI solved it temporarily. I didn't realize that this was a bandaid until my ASG replaced an unhealthy instance, only for the new instance to fail to start.

If you look at the bottom of your /var/log/cloud-init-output.log file, you'll likely see a garbled log message (mine was something about a grub disk ID changing). This is likely the source behind an Ubuntu interactive question (one of those colorful, yes/no screens)...so clearly the -y isn't sufficient.

The fix is to update your userdata script to do the updates as follows, and I think the key is the DEBIAN_FRONTEND export:

export DEBIAN_FRONTEND=noninteractive
apt-get update 
apt-get -y upgrade
apt-get autoremove 
apt-get autoclean

More details on my post here: AWS EC2 UserData Script Doesn't Fire on t3 instances

Answer 2

I'm the author of that prior question you mentioned. After a lot of messing around, I am pretty sure this is down to the AMI you may be using. For example I was using ami-0727f3c2d4b0226d5 which I thought was a version of Ubuntu 18.04 LTS eu-west-1, although I can no longer find it in any AMI search. So following this link -

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/finding-an-ami.html

I ran the following to get a "latest" version of Ubuntu 16:04 -

aws ec2 describe-images --owners 099720109477 --filters 'Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-????????' 'Name=state,Values=available' --query 'reverse(sort_by(Images, &CreationDate))[:1].ImageId' --output text

which resulted in ami-0c224e30f7a997d9f; and when I use this new AMI the problem I had magically disappeared :-)

You don't say exactly what AMI you were using, but in my case I suspect there was a problem with the AMI version I was using; and so I would suggest you possibly experiment with different AMI versions. Good luck :-)