CODEWITHSUNDEEP
metricsdatadog
Christopher Will
Christopher Will

Reputation: 3064

Datadog distinct-like custom metrics

Given following scenario:

Right now we monitor a custom error-count metric like myService.errorType. Which gives us an exact number of how many times an error occurred - independent from a specific entity: If an entity can't be processed like 100 times, then the metric value will be 100.

What I'd like to have, though, is a distinct metric based on the UUID. Example:

Then I'd like to have a metric with the value of 2 - because the processes failed for two entities only (and not for 30, as it would be reported right now).

While searching for a solution I found the possibility of using tags. But as the docs point out they are not meant for such a use-case:

Tags shouldn’t originate from unbounded sources, such as epoch timestamps, user IDs, or request IDs. Doing so may infinitely increase the number of metrics for your organization and impact your billing.

So are there any other possibilities to achieve my goals?

Upvotes: 0

Views: 3979

Answers (2)

XYZ123
XYZ123

Reputation: 456

To make sure things are clear, you have a metric called myService.errorType with a tag entity. This metric is a counter that will increase every time an entity is in error. You will then use this metric query:

sum:myService.errorType{*} by {entity}

When you speak about UUID, it seems that the cardinality is small (here you show 3). Which means that every hour you will have small amount of UUID available. In that case, adding UUID to the metric tags is not as critical as user ID, timestamp, etc. which have a limitless number of options.

I would invite you to add this uuid tag, and check the cardinality in the metric summary page to ensure it works.

Then to get the number of UUID concerned by errors, you can use something like:

count_not_null(sum:myService.errorType{*} by {uuid})

Finally, as an alternative, if the cardinality of UUID can go through the roof, I would invite you to work with logs or work with Christopher's solution which seems to limit the cardinality increase as well.

Upvotes: 0

Christopher Will
Christopher Will

Reputation: 3064

I've solved it now by verifying the status via code and by adding tags to the metrics:

  • occurrence:first
  • subsequent

This way I can filter in my dashboard for occurrence:first only.

Upvotes: 0

Related Questions