How to send Confluent cloud metrics to Datadog?

Question

I want to get Confluent cloud metrics into Datadog so I followed the this instruction. Instead of using CCLOUD_USER: ${CCLOUD_USER} and CCLOUD_PASSWORD: ${CCLOUD_PASSWORD} I used CCLOUD_API_KEY and CCLOUD_API_SECRET as environment variables for the exporter container.

I get a Failed to establish a new connection: [Errno 111] Connection refused error:

2021-02-12 12:29:08 UTC | CORE | ERROR | (pkg/collector/runner/runner.go:292 in work) | Error running check openmetrics: [{"message": "HTTPConnectionPool(host='localhost', port=2112): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f227f294b80>: Failed to establish a new connection: [Errno 111] Connection refused'))", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connection.py\", line 159, in _new_conn\n    conn = connection.create_connection(\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/connection.py\", line 84, in create_connection\n    raise err\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/connection.py\", line 74, in create_connection\n    sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 670, in urlopen\n    httplib_response = self._make_request(\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 392, in _make_request\n    conn.request(method, url, **httplib_request_kw)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/http/client.py\", line 1255, in request\n    self._send_request(method, url, body, headers, encode_chunked)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/http/client.py\", line 1301, in _send_request\n    self.endheaders(body, encode_chunked=encode_chunked)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/http/client.py\", line 1250, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/http/client.py\", line 1010, in _send_output\n    self.send(msg)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/http/client.py\", line 950, in send\n    self.connect()\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connection.py\", line 187, in connect\n    conn = self._new_conn()\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connection.py\", line 171, in _new_conn\n    raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f227f294b80>: Failed to establish a new connection: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/adapters.py\", line 439, in send\n    resp = conn.urlopen(\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/connectionpool.py\", line 726, in urlopen\n    retries = retries.increment(\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/urllib3/util/retry.py\", line 446, in increment\n    raise MaxRetryError(_pool, url, error or ResponseError(cause))\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=2112): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f227f294b80>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 876, in run\n    self.check(instance)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/base_check.py\", line 112, in check\n    self.process(scraper_config)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 521, in process\n    for metric in self.scrape_metrics(scraper_config):\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 458, in scrape_metrics\n    response = self.poll(scraper_config)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 764, in poll\n    response = self.send_request(endpoint, scraper_config, headers)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/mixins.py\", line 790, in send_request\n    return http_handler.get(endpoint, stream=True, **kwargs)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py\", line 298, in get\n    return self._request('get', url, options)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/utils/http.py\", line 363, in _request\n    response = request_method(url, **new_options)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py\", line 75, in get\n    return request('get', url, params=params, **kwargs)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/api.py\", line 60, in request\n    return session.request(method=method, url=url, **kwargs)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/requests/adapters.py\", line 516, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=2112): Max retries exceeded with url: /metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f227f294b80>: Failed to establish a new connection: [Errno 111] Connection refused'))\n"}]

When I tried to curl http://ccloudexporter_ccloud_exporter_1:2112/metrics I got no reply but I did with a curl to http://localhost:2112/metrics. So I adjusted the openmetrics.yml to use prometheus url http://localhost:2112/metrics. Still same error in the DD container. When I go to http://localhost:2112/metrics in my browser I see metrics.

No clue on why DD cannot connect to /metrics.

Answer 1

In the instructions it refers to http://ccloudexporter_ccloud_exporter_1:2112/metrics in the open metrics file, but in my setup docker-compose gave the ccloud exporter container the name ccloud_ccloud_exporter_1. To prevent this I added "container_name: ccloud_ccloud_exporter_1" in the docker compose file and used "http://ccloud_ccloud_exporter_1:2112/metrics" as prometheus url in the openmetrics file.