CODEWITHSUNDEEP
pythondockerkubernetesdatadog
IAmCoder
IAmCoder

Reputation: 3442

Datadog count metric dropping with multiple containers

I am incrementing a Datadog counter in python:

from datadog import initialize
from datadog import ThreadStats
stats.increment('api.request_count', tags=['environment:' + environment])

And have set the metric type to "count" and the unit to "requests per none" in the metadata for the metric.

The code runs in a docker container on a kubernetes node in a Container Engine in Google Cloud... I have docker-dd-agent (https://github.com/DataDog/docker-dd-agent) running on each node.

I can move the container to any node and it logs around 200 requests per minute. But as soon as I scale it up and launch a second container, it only logs around 100 requests per minute. If I scale down to one container again, it spikes to 200 rpm again:

enter image description here

What could be causing the requests to drop or get overwritten from other pods?

Upvotes: 1

Views: 2828

Answers (2)

IAmCoder
IAmCoder

Reputation: 3442

In the python script I was initializing with an api key:

from datadog import api
from datadog import initialize
from datadog import statsd
options = {
    'api_key':'#######'
}

initialize(**options)

And sending some events

api.Event.create(title=title, text=text, tags=tags)

When I changed it to initialize like this, it started working with the dd-agent:

initialize(statsd_use_default_route=True)

I didn't need the link command (--link dogstatsd:dogstastd).

With that setup it now works in the staging environment, but not in production. :/

Upvotes: 0

stephenlechner
stephenlechner

Reputation: 2279

Why not use dogstatsd instead of threadstats? If you're already running the dd-agent on your node in a way that's reachable by your containers, you can use the datadog.statsd.increment() method instead to send the metric over statsd to the agent, and from there it would get forwarded to your datadog account.

Dogstatsd has the benefits of being more straight-forward and of being somewhat easier to troublehsoot, at least with debug-level logging. Threadstats sometimes has the benefit of not requiring a dd-agent, but it does very little (if any) error logging, so makes it difficult to troubleshoot cases like these.

If you went the dogstatsd route, you'd use the following code:

from datadog import initialize
from datadog import statsd
statsd.increment('api.request_count', tags=['environment:' + environment])

And from there you'd find your metric metadata with the "rate" type and with an interval of "10", and you could use the "as_count" function to translate the values to counts.

Upvotes: 2

Related Questions