How to debug why Telegraf is not sending data from MQTT to influxdb

Question

I am trying to setup TIG stack as follows:

Sensor data -> MQTT (Eclipse-Mosquitto) -> Telegraf -> InfluxDB -> Grafana

I haven't setup the source of the sensor data yet but at its current state, everything compiles and runs without obvious errors and it all seems to be connected but the data published to MQTT topic is not ending up in my Influxdb database.

$ sudo docker-compose up
Creating network "tig-samson-feature-setup-docker-compose_default" with the default driver
Creating mosquitto ... done
Creating influxdb  ... done
Creating grafana   ... done
Creating telegraf  ... done
Attaching to influxdb, mosquitto, grafana, telegraf
mosquitto    | 1583407265: mosquitto version 1.6.9 starting
mosquitto    | 1583407265: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto    | 1583407265: Opening ipv4 listen socket on port 1883.
mosquitto    | 1583407265: Opening ipv6 listen socket on port 1883.
mosquitto    | 1583407266: New connection from 192.168.32.1 on port 1883.
mosquitto    | 1583407266: New client connected from 192.168.32.1 as mosqsub|5779-jackbridge (p1, c1, k60).
grafana      | t=2020-03-05T11:21:05+0000 lvl=info msg="Starting Grafana" logger=server version=6.6.2 commit=3fa63cfc34 branch=HEAD compiled=2020-02-20T12:03:49+0000
grafana      | t=2020-03-05T11:21:05+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
grafana      | t=2020-03-05T11:21:05+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
grafana      | t=2020-03-05T11:21:05+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"

...continues indefinitely.. can add more detail if it's helpfuL!

Below is some of my code and some of the things I've tried

File architecture running on an ubuntu VM (running in virtualbox on a mac)

docker-compose.yml
telegraf.conf
influxdb (folder)
-----data (folder)
-----------data (locked folder auto generated)
-----------meta (locked folder auto generated)
-----------wal (locked folder auto generated)
data 
----mosquitto (empty folder)

docker-compose

# # docker-compose.yml
version: '3'

services: 
  influxdb:
    container_name: influxdb
    image: influxdb
    ports:
        - "8086:8086"
    environment:
      - INFLUXDB_DB=sensors
      - INFLUXDB_ADMIN_USER=admin
      - INFLUXDB_ADMIN_PASSWORD=telegraf
    volumes:
      - ./influxdb/data:/var/lib/influxdb
    restart: always

  telegraf: 
    container_name: telegraf
    image: telegraf
    restart: always
    depends_on: 
      - mosquitto
      - influxdb
    volumes:
    - ./telegraf.conf:/etc/telegraf/telegraf.conf:ro   


  grafana: 
    container_name: grafana
    image: grafana/grafana
    links:
      - influxdb
    hostname: grafana
    ports:
        - "3000:3000"

  mosquitto:
    container_name: mosquitto
    image: eclipse-mosquitto
    ports:
      - "1883:1883"
      - "9001:9001"
    restart: always
    volumes:
      - ./mosquitto.conf:/mqtt/config:ro
#      - /data/mosquitto:mosquitto/data

(I start the program with docker-compose up)

telegraf.conf

 [[outputs.file]]
#   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout"]

# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
  servers = ["tcp://localhost:1883"]
  topics = [
    "sensors"
  ]
  data_format = "influx"
#   ## MQTT broker URLs to be used. The format should be scheme://host:port,
#   ## schema can be tcp, ssl, or ws.
#   servers = ["tcp://127.0.0.1:1883"]


[[outputs.influxdb]]
urls = ["http://influxdb:8086"]
database = "sensors"
skip_database_creation = true
username = "telegraf"
password = "telegraf"

I can access telegraf docker container and ping influxdb so it seems to be connected

root@fc1d9af8f0c6:/# curl -sl -I http://influxdb:8086/ping
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 12b4923a-5ec7-11ea-8006-0242c0a81002
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.10
X-Request-Id: 12b4923a-5ec7-11ea-8006-0242c0a81002
Date: Thu, 05 Mar 2020 09:52:49 GMT

I can publish and subscribe to MQTT in other terminals:

$ mosquitto_pub -t 'sensors' -m 'strength, site=room1 value=HELLO tony'

This is received in another terminal subscribing to sensors

$ mosquitto_sub -t 'sensors' -v # note this is executed before mosquitto_pub
sensors strength, site=room1 value=HELLO tony

When I do run mosquitto_sub there is no activity in the terminal running docker-compose.

When I run mosquitto_pub the client is detected as connecting but the publish event doesn't seem to register

mosquitto    | 1583407850: New connection from 192.168.32.1 on port 1883.
mosquitto    | 1583407850: New client connected from 192.168.32.1 as mosqpub|7039-jackbridge (p1, c1, k60).
mosquitto    | 1583407850: Client mosqpub|7039-jackbridge disconnected.

Telegraf seems to be sending metrics to Influxdb.. In telegraf [[outputs.file]] I am sending files =['stdout'] which you above. It's not clear the metrics are arriving in influxdb as I haven't been able to locate them.

Sample of output in terminal

telegraf     | disk,device=sda1,fstype=ext4,host=foobar,mode=rw,path=/etc/resolv.conf inodes_used=189723i,total=10499674112i,free=3154145280i,used=6791987200i,used_percent=68.2877210177679,inodes_total=655360i,inodes_free=465637i 1583406223000000000
telegraf     | cpu,cpu=cpu-total,host=foobar usage_steal=0,usage_system=7.14285714285916,usage_softirq=0.8281573498966311,usage_nice=0,usage_iowait=0.10351966873709728,usage_irq=0,usage_guest=0,usage_guest_nice=0,usage_user=56.41821946171202,usage_idle=35.507246376802996 1583406223000000000
telegraf     | cpu,cpu=cpu0,host=foobar usage_user=56.41821946171202,usage_irq=0,usage_steal=0,usage_system=7.14285714285916,usage_idle=35.507246376802996,usage_nice=0,usage_iowait=0.10351966873709728,usage_softirq=0.8281573498966311,usage_guest=0,usage_guest_nice=0 1583406223000000000
telegraf     | processes,host=foobar sleeping=2i,paging=0i,idle=0i,dead=0i,total_threads=11i,blocked=0i,zombies=0i,stopped=0i,running=0i,total=2i,unknown=0i 1583406223000000000
telegraf     | 2020-03-05T11:03:53Z D! [outputs.file] Wrote batch of 26 metrics in 1.676587ms
telegraf     | 2020-03-05T11:03:53Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf     | 2020-03-05T11:03:53Z D! [outputs.influxdb] Wrote batch of 26 metrics in 8.885827ms
telegraf     | 2020-03-05T11:03:53Z D! [outputs.influxdb] Buffer fullness: 23 / 10000 metrics

In Influxdb my sensors database is created

compose$ sudo docker exec -it influxdb influx
[sudo] password for jackbridger:
Connected to http://localhost:8086 version 1.7.10
InfluxDB shell version: 1.7.10
> show databases
name: databases
name
----
sensors
_internal
> use _internal


> show stats
name: runtime
Alloc    Frees    HeapAlloc HeapIdle HeapInUse HeapObjects HeapReleased HeapSys  Lookups Mallocs  NumGC NumGoroutine PauseTotalNs Sys      TotalAlloc
-----    -----    --------- -------- --------- ----------- ------------ -------  ------- -------  ----- ------------ ------------ ---      ----------
16426120 26535115 16426120  44695552 21823488  225331      28131328     66519040 0       26760446 277   26           6132353      72483064 1337594448

name: queryExecutor
queriesActive queriesExecuted queriesFinished queryDurationNs recoveredPanics
------------- --------------- --------------- --------------- ---------------
1             3               2               3462237         0

name: database
tags: database=_internal
numMeasurements numSeries
--------------- ---------
12              467


...continues for a while.. can add more detail if it's helpfuL!

I tried to setup volume for mosquitto but couldn't get it to work - I'm thinking this could be an issue

Would anyone be able to provide any guidance on how I can debug this to figure out why the messages added to MQTT are not being sent by Telegraf to the sensors database in Influxdb?

Thanks!

Answer 1

The below was provided very helpfully by a friend and it fixed the issue

# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
  servers = ["tcp://localhost:1883"]
  topics = [
    "sensors"
  ]
  data_format = "influx"
#   ## MQTT broker URLs to be used. The format should be scheme://host:port,
#   ## schema can be tcp, ssl, or ws.
#   servers = ["tcp://127.0.0.1:1883"]

localhost:1883 would resolve the host to the telegraf container. If you want to refer to another container running in the same docker compose setup you can use the service name as the host name, so in this case try tcp://mosquitto:1883. Have a read here: https://docs.docker.com/compose/networking/

Each container can now look up the hostname web or db and get back the appropriate container’s IP address. For example, web’s application code could connect to the URL postgres://db:5432 and start using the Postgres database.