Issue with awslogs service and CloudWatch Logs Agent on Ubuntu 16.04

Question

On one of my AWS ec2 instances running Ubuntu 16.04, I'm getting the following errors filled up in my /var/syslog.

Jul 17 18:11:21 Mysql-Slave systemd[1]: Stopped The CloudWatch Logs agent.
Jul 17 18:11:21 Mysql-Slave systemd[1]: Started The CloudWatch Logs agent.
Jul 17 18:11:26 Mysql-Slave systemd[1]: awslogs.service: Main process exited, code=exited, status=255/n/a
Jul 17 18:11:26 Mysql-Slave systemd[1]: awslogs.service: Unit entered failed state.
Jul 17 18:11:26 Mysql-Slave systemd[1]: awslogs.service: Failed with result 'exit-code'.
Jul 17 18:11:26 Mysql-Slave systemd[1]: awslogs.service: Service hold-off time over, scheduling restart.
Jul 17 18:11:26 Mysql-Slave systemd[1]: Stopped The CloudWatch Logs agent.
Jul 17 18:11:26 Mysql-Slave systemd[1]: Started The CloudWatch Logs agent.
Jul 17 18:11:32 Mysql-Slave systemd[1]: awslogs.service: Main process exited, code=exited, status=255/n/a
Jul 17 18:11:32 Mysql-Slave systemd[1]: awslogs.service: Unit entered failed state.
Jul 17 18:11:32 Mysql-Slave systemd[1]: awslogs.service: Failed with result 'exit-code'.
Jul 17 18:11:32 Mysql-Slave systemd[1]: awslogs.service: Service hold-off time over, scheduling restart.
Jul 17 18:11:32 Mysql-Slave systemd[1]: Stopped The CloudWatch Logs agent.
Jul 17 18:11:32 Mysql-Slave systemd[1]: Started The CloudWatch Logs agent.

The /var/log/awslogs.log contains these messages:

database is locked
2018-07-17 20:59:01,055 - cwlogs.push - INFO - 27074 - MainThread - Missing or invalid value for use_gzip_http_content_encoding config. Defaulting to using gzip encoding.
2018-07-17 20:59:01,055 - cwlogs.push - INFO - 27074 - MainThread - Using default logging configuration.

database is locked
2018-07-17 20:59:06,549 - cwlogs.push - INFO - 27104 - MainThread - Missing or invalid value for use_gzip_http_content_encoding config. Defaulting to using gzip encoding.
2018-07-17 20:59:06,549 - cwlogs.push - INFO - 27104 - MainThread - Using default logging configuration.

database is locked
2018-07-17 20:59:12,054 - cwlogs.push - INFO - 27110 - MainThread - Missing or invalid value for use_gzip_http_content_encoding config. Defaulting to using gzip encoding.
2018-07-17 20:59:12,054 - cwlogs.push - INFO - 27110 - MainThread - Using default logging configuration.

Any pointers in troubleshooting this will be of great help.

Answer 1

A similar issue was posted in the following link - https://forums.aws.amazon.com/thread.jspa?threadID=165134

I did the following:

a) Stopped the awslogs service

$ service awslogs stop  ## Amazon Linux

OR

$ service awslogsd stop  ## Amazon Linux 2

b) Deleted the agent-state file in /var/awslogs/state/ (I renamed it in my case)

$  mv agent-state agent-state.old  ## Amazon Linux

OR

$ cd /var/lib/awslogs; mv agent-stat agent-stat.old  ## Amazon Linux 2

c) Restarted the awslogs service

$ service awslogs start  ## Amazon Linux

OR

$ sudo systemctl start awslogsd  ## Amazon Linux 2

A new agent-state file was created as a result and the errors mentioned my post disappeared after this.

Answer 2

In my case, I found duplicate entries for some properties in /etc/awslogs/awslogs.conf file. (Not all were duplicates, as some of the properties were commented, and I uncommented them to set values.)

It didn't work. Then I scrolled till the bottom of the file. I found following entries. Set the values to these properties and it worked.

[/var/log/messages]
datetime_format = %b %d %H:%M:%S
file = /home/ec2-user/application.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = MyProject

Answer 3

Please try the following commands based on your Linux version

sudo service awslogs start

If you are running Amazon Linux 2, try the below command sudo systemctl start awslogsd took me 2 hours to figure this out