CODEWITHSUNDEEP
gometricsdatadog
Arvind
Arvind

Reputation: 33

Custom latency metrics in Datadog

We have been using DD successfully for tracing and monitoring at our company. We have a microservices architecture (services written in Go) where the flow is something like this: Incoming request -> Proxy service -> AWS EventBridge -> Service1 -> Service2 We can see the trace on Datadog but we want to be able to measure the time it takes for the request to make it to Service2 and alert if it is beyond a threshold. When we reached out to DD support, they said they don't yet support the ability to measure the latency with the traces we have. So I was thinking I could emit a custom metric at Service2, measuring the time difference between when the request hits the Proxy Service and Service2. I can't find any pointers on how to do this anywhere. Any help in this regard (preferably in Go)?

Upvotes: 2

Views: 2691

Answers (1)

ksparrow
ksparrow

Reputation: 195

I'd recommend tracking timing information (current timestamp minus the header timestamp) as a histogram. By default it aggregates average, median, max, and 95th percentile values into gauges, and a count of samples into a rate.

Sounds like your latency measurement is the current time on Server2 minus the header timestamp. Using the datadog-go library it looks something like:

statsd.Histogram("latency.proxy_to_server2", float64(your_measurement), []string{"tag:value"}, 1)

Datadog's docs go into more detail about how to submit metrics.

Depending on your Datadog agent configuration, you may already have automatic tagging. If not, and you're running several Service2 tasks/pods/instances, then you might want to add some global tags during statsd client initialization.

Upvotes: 2

Related Questions