cloud-init-output.log being overwritten on EC2 instance

Question

I am running a job on an ec2 instance on startup with user data.

When the job starts I head to var/log and start tailing cloud-init-output.log and it shows me the correct output data. However, after I let the job run over night and my pc goes to sleep, I disconnect from my ssh connection, and when I open the cloud-init-output.log file, none of the data that I was seeing while tailing it is there anymore, instead it is just showing thing like kernel information.

Any ideas why this is happening, or how I can access the logs that I was seeing while tailing the file?

Answer 1

You haven't mentioned the exact operating system but as you are using tail it is a Linux based one. Cloud-init logging configuration by default is described in this document, and unless that specific distribution has made changes to it, it will mostly likely have a line like the one below, which basically sends both stderr and stdout from all init scripts to /var/log/cloud-init-output.log:

output: {all: '| (umask 0026; tee -a /var/log/cloud-init-output.log)'}

Now, there are two main reasons for the behaviour you are seeing:

• first of all, the cloud-init happens only once during the instance initialisation and bootstrap after it has been created.

The only information added to it later on happens after reboots, but it's mostly static. For example on Amazon Linux 2 it looks like this:

Cloud-init v. 19.3-44.amzn2 running 'init-local' at Fri, 24 Sep 2021 11:59:49 +0000. Up 7.03 seconds.
Cloud-init v. 19.3-44.amzn2 running 'init' at Fri, 24 Sep 2021 11:59:51 +0000. Up 9.39 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | Device |  Up  |           Address           |      Mask     | Scope  |     Hw-Address    |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: |  eth0  | True |         172.31.36.56        | 255.255.240.0 | global | 0a:d1:9d:ab:5e:9f |
ci-info: |  eth0  | True | fe80::8d1:9dff:feab:5e9f/64 |       .       |  link  | 0a:d1:9d:ab:5e:9f |
ci-info: |   lo   | True |          127.0.0.1          |   255.0.0.0   |  host  |         .         |
ci-info: |   lo   | True |           ::1/128           |       .       |  host  |         .         |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++
ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
ci-info: | Route |   Destination   |   Gateway   |     Genmask     | Interface | Flags |
ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
ci-info: |   0   |     0.0.0.0     | 172.31.32.1 |     0.0.0.0     |    eth0   |   UG  |
ci-info: |   1   | 169.254.169.254 |   0.0.0.0   | 255.255.255.255 |    eth0   |   UH  |
ci-info: |   2   |   172.31.32.0   |   0.0.0.0   |  255.255.240.0  |    eth0   |   U   |
ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: |   9   |  fe80::/64  |    ::   |    eth0   |   U   |
ci-info: |   11  |    local    |    ::   |    eth0   |   U   |
ci-info: |   12  |   ff00::/8  |    ::   |    eth0   |   U   |
ci-info: +-------+-------------+---------+-----------+-------+
Cloud-init v. 19.3-44.amzn2 running 'modules:config' at Fri, 24 Sep 2021 11:59:52 +0000. Up 10.80 seconds.
Cloud-init v. 19.3-44.amzn2 running 'modules:final' at Fri, 24 Sep 2021 11:59:53 +0000. Up 11.47 seconds.
Cloud-init v. 19.3-44.amzn2 finished at Fri, 24 Sep 2021 11:59:53 +0000. Datasource DataSourceEc2.  Up 11.59 seconds

This means that any script that you add to user-data would run only once during the instance bootstrap, and unless it's set to re-spawn or run in the background it will most likely exit after that. In addition, it won't be executed again upon reboot.

• the second reason is that you are tailing the log, and tail only outputs the last lines of a file. When your ssh connection breaks and you initiate the new one, you are seeing the last lines at that point in time. You should be still able to see the information you were observing the previous day by using less or cat or by grep-ing the log file for it (unless the cloud-init configuration has been altered and it's overwriting the information in the log, but that's unlikely).

That being said, based on the above user-data is not the right place for any jobs or scripts that you want to run and log continuously and you should look into other methods to start those jobs post-bootstrap like for example SSM Run Command and log any information you need in a dedicated log file which you can then send to CloudWatch.