CODEWITHSUNDEEP
influxdbtelegraf
Jônathan Dambros
Jônathan Dambros

Reputation: 11

Avoid reading the same file multiple times using Telegraf and file input plugin

I need to read csv files inside a folder. New csv files are generated every time a user submits a form. I'm using the "file" input plugin to read the data and send it to Influxdb. These steps are working fine.

The problem is that the same file is read multiple times every data collection interval. I was thinking of a solution where I could move the file that was read to a different folder, but I couldn't do that with Telegraf's "exec" output plug.

ps: I can't change the way csv files are generated.

Any ideas on how to avoid reading the same csv file multiple times?

Upvotes: 1

Views: 1860

Answers (1)

powersj
powersj

Reputation: 379

As you discovered file input plugin is used to read entire files at each collection interval.

My suggestion is for you to instead use the directory monitor input plugin. This will read files in a directory, monitor the directory for new files, and parse the ones that have not already been picked up yet. There are some configuration settings in that plugin that make it easier to time when new files are read as well.

Another option is to use the tail input plugin which will tail a file and only read new updates to that file as things come. However, I think the directory monitor is more likely something you are after for your scenario. Thanks!

Upvotes: 1

Related Questions