CODEWITHSUNDEEP
amazon-web-servicesamazon-cloudwatch
RyanBesari
RyanBesari

Reputation: 171

How to get metrics data from aws cloudwatch to csv

i have been working on project, and i want to export metrics data from cloudwatch like CPU Utilization and Network Out data, is there any way to get that data? and convert it to csv?

Upvotes: 15

Views: 31229

Answers (5)

Kethan Srinivas
Kethan Srinivas

Reputation: 99

You can now download the metrics directly from AWS Metrics dashboard in the .csv file format. As shown in the image in order to download the metrics from AWS, you need to go the CloudWatch dashboard and click on the Metrics in the left navigation bar and then select the metric you want to download. Now click on the Actions drop-down button and select Download as .csv

Screen shot of AWS CloudWatch dashboard

Upvotes: 2

vicla
vicla

Reputation: 11

If the csv exporting is a routine work, you can try categraf cloudwatch plugin . It can easily export cloud-watch metrics to your Prometheus then you can visualize data using grafana and export to csv.

And further more, you can also try nightingale project which can be an alternative of prometheus&grafana.

Upvotes: 1

bhaskar das
bhaskar das

Reputation: 195

You can also try out official Prometheus cloudwatch_exporter

Upvotes: 0

mmansoor
mmansoor

Reputation: 620

Export CloudWatch Metrics

You can't do it directly but following is the step by step instructions:

Pre Reqs

  1. AWS CLI (Command Line Interface) from here
  2. jq is a lightweight and flexible command-line JSON processor from here

How to export

Use the following CLI:

aws cloudwatch get-metric-statistics 
--namespace AWS/EC2 --metric-name CPUUtilization 
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx 
--statistics Average 
--start-time 2020-03-01T00:00:00 
--end-time 2020-03-31T23:59:00 
--period 3600
--region us-east-1

Valid options for --metric-name depend on the --name-space parameter. For AWS/EC2, the full list can be seen by running the following CLI command:

aws cloudwatch list-metrics --namespace "AWS/EC2"

Valid options for --statistics are:

SampleCount
Average
Sum
Minimum
Maximum

--start-time and --end-time specify the range.

--period The granularity, in seconds, of the returned data points.

--region The region where the CloudWatch metric is being meatured (us-east-1, us-west-2 etc.)

Data output will look something like the following:

{
    "Label": "CPUUtilization",
    "Datapoints": []
}

To convert it to CSV we will use jq. For this you have two options:

Option 1

Pipe all the aws cli output to jq:

aws cloudwatch get-metric-statistics 
--namespace AWS/EC2 --metric-name CPUUtilization 
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx 
--statistics Average 
--start-time 2020-03-01T00:00:00 
--end-time 2020-03-31T23:59:00 
--period 3600
--region us-east-1
| jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | @csv'

Option 2

Export the data to JSON:

aws cloudwatch get-metric-statistics 
--namespace AWS/EC2 --metric-name CPUUtilization 
--dimensions Name=InstanceId,Value=i-xxxxxxxxxxxxxxxxx 
--statistics Average 
--start-time 2020-03-01T00:00:00 
--end-time 2020-03-31T23:59:00 
--period 3600
--region us-east-1 >> data.json

Use jq to read the json to csv:

jq -r '.Datapoints[] | [.Timestamp, .Minimum, .Unit] | @csv' data.json

Output

Here is how the output will look like:

"2020-03-24T11:00:00Z",0.327868852454245,"Percent"
"2020-03-11T21:00:00Z",0.327868852454245,"Percent"
"2020-03-15T04:00:00Z",0.322580645156596,"Percent"
"2020-03-27T18:00:00Z",0.327868852478101,"Percent"

Upvotes: 30

John Rotenstein
John Rotenstein

Reputation: 270224

There is no in-built capability to export Amazon CloudWatch metrics to CSV.

There are API calls available to extract metrics, but you would need to write a program to call the API, receive the metrics and store it in a suitable format.

There are projects available that can assist with this, such as:

If you are looking for these types of tools, make sure they refer to CloudWatch metrics, not CloudWatch Logs (which is something different).

Upvotes: 6

Related Questions