CODEWITHSUNDEEP
amazon-web-servicesamazon-cloudwatchamazon-eks
user6826691
user6826691

Reputation: 2021

CW agent not getting mem_used_percent from eks cluster

I'm trying to install cw agent on my eks cluster to get instance level metrics, especially mem_used_percent metric, can someone point me where i'm wrong, i've the cwagent server policy attached to the eks nodes.

I'm seeing all container insights metrics but not able to see any metrics in cw agent(aws console).

What am i missing here?

my config_file

apiVersion: v1
data:
  # Configuration is in Json format. No matter what configure change you make,
  # please keep the Json blob valid.
  cwagentconfig.json: |
    {
      "logs": {
        "metrics_collected": {
          "kubernetes": {
            "cluster_name": "xxxx",
            "metrics_collection_interval": 60
          },
          "mem": {
            "measurement": [
              "mem_used_percent"
            ],
            "metrics_collection_interval": 60
          }
        },
        "force_flush_interval": 5
      }
    }
kind: ConfigMap
metadata:
  name: cwagentconfig
  namespace: amazon-cloudwatch

Deployed this : https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cwagent/cwagent-daemonset.yaml

Image: amazon/cloudwatch-agent:1.247348.0b251302 CI_VERSION: "k8s/1.3.7"

My_logs:

2021/07/02 23:49:06 I! I! Detected the instance is EC2
2021/07/02 23:49:03 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
2021/07/02 23:49:03 Reading json config file path: /etc/cwagentconfig/..2021_07_02_23_48_59.690305669/cwagentconfig.json ...
2021/07/02 23:49:03 Find symbolic link /etc/cwagentconfig/..data
2021/07/02 23:49:03 Find symbolic link /etc/cwagentconfig/cwagentconfig.json
2021/07/02 23:49:03 Reading json config file path: /etc/cwagentconfig/cwagentconfig.json ...
Valid Json input schema.
2021/07/02 23:49:03 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
2021/07/02 23:49:04 W! retry [0/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/07/02 23:49:05 W! retry [1/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/07/02 23:49:06 W! retry [2/3], unable to get http response from http://169.254.170.2/v2/metadata, error: unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2021/07/02 23:49:06 I! access ECS task metadata fail with response unable to get response from http://169.254.170.2/v2/metadata, error: Get "http://169.254.170.2/v2/metadata": context deadline exceeded (Client.Timeout exceeded while awaiting headers), assuming I'm not running in ECS.
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded

2021/07/02 23:49:06 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2021-07-02T23:49:06Z I! Starting AmazonCloudWatchAgent 1.247348.0
2021-07-02T23:49:06Z I! Loaded inputs: cadvisor k8sapiserver
2021-07-02T23:49:06Z I! Loaded aggregators:
2021-07-02T23:49:06Z I! Loaded processors: ec2tagger k8sdecorator
2021-07-02T23:49:06Z I! Loaded outputs: cloudwatchlogs
2021-07-02T23:49:06Z I! Tags enabled:
2021-07-02T23:49:06Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-10-182-7-7.ec2.internal", Flush Interval:1s
2021-07-02T23:49:06Z I! [logagent] starting
2021-07-02T23:49:06Z I! [logagent] found plugin cloudwatchlogs is a log backend

Upvotes: 1

Views: 837

Answers (1)

Hayden Moulds
Hayden Moulds

Reputation: 19

Metrics collection such as memory usage needs to be put in the metrics section of the json file instead of in the logs section. Explained in the aws doco. You just need to rename the first section, and I think you have an extra } at the end.

The following should work:

"metrics": {
  "metrics_collected": {
    "kubernetes": {
      "cluster_name": "xxxx",
      "metrics_collection_interval": 60
    },
    "mem": {
      "measurement": [
        "mem_used_percent"
      ],
      "metrics_collection_interval": 60
    }
  },
  "force_flush_interval": 5
}

Upvotes: 0

Related Questions