ElasticBeanstalk: Cloudwatch agent stops streaming logs randomely

Question

I have a simple nodejs (express) application deployed through ElasticBeanstalk, I enabled the cloudwatch logs in configuration and logs started to stream. Problem is, it just simply stops randomly. I searched around and came across this answer, following it restart the streaming but after some time it hangs up again and I have to restart it. How do I prevent it happening in the first place?

I even tried providing the configuration manually. This is my .config file placed under .ebextensions directory. No difference after it, took this sample from here, direct file link.

packages:
  yum:
    awslogs: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`

  "/etc/awslogs/awslogs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [general]
      state_file = /var/lib/awslogs/agent-state
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      log_stream_name = {instance_id}

  "/etc/awslogs/config/logs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [/var/log/eb-activity.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/eb-activity.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/eb-activity.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/nodejs/nodejs.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/nodejs/nodejs.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/nodejs/nodejs.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/nginx/error.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/nginx/error.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/nginx/error.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/nginx/access.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/nginx/access.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/nginx/access.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/httpd/error.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/httpd/error.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/httpd/error.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

      [/var/log/httpd/access.log]
      log_group_name = `{"Fn::Join":["/", ["/aws/elasticbeanstalk", { "Ref":"AWSEBEnvironmentName" }, "/var/log/httpd/access.log"]]}`
      log_stream_name = {instance_id}
      file = /var/log/httpd/access.log
      time_zone = UTC
      datetime_format = %Y-%m-%d %H:%M:%S
      multi_line_start_pattern = {datetime_format}

commands:
  "01":
    command: chkconfig awslogs on
  "02":
    command: service awslogs restart

In cloudwatch agent reference's FAQ I see it says:

If the timestamp of log event is more than 2 hours in future, the log event is skipped

In my var/log/awslogs.log I do notice this error that says:

someDateTime - cwlogs.push.batch - WARNING - xxxxx - Thread-4 - Skip event: {'timestamp': xxxxxxxxxx, 'start_position': xxxxxxx, 'end_position': xxxxxxxxx}, reason: timestamp is more than 2 hours in future.

How do I prevent this from happening?

Answer 1

This is a bit late posting question but I had it figured it out with help of AWS support. Our load balancer was configured to ping the server for health check which was producing a specific log something like "token not found in request". Health check was passed because it was configured to check it but constantly logging same string was causing the CW agent to not work properly when creating hash for next cycle.