CODEWITHSUNDEEP
inputlogstash
Yogesh_D
Yogesh_D

Reputation: 18764

Logstash processing the same file with logstash restarts

I am playing around with Logstash, and in this I am trying to process the same time multiple times by killing (Control+C) and restarting the process.

It seems logstash remembers the file and the position I was at, and will only process the files in case of:

  1. Either updates to the file with modifications to existing rows.
  2. Or updates to the file with new lines.

I am not able to find when 1 or 2 works. Also the output of 1 or 2 is also random, i.e., for one file it picks up on the latest added statement (Even after process restarts) and for some it picks up everything. I am using this file input config:

input {
  file {
    path => "/work/log4j_sample.log"
    start_position => beginning
  }
}

I know a sincedb file is created in the user home directory. But can someone point me to the details of how this works? Is the sincedb file per input file or is it per process?

Thanks!

Upvotes: 0

Views: 315

Answers (1)

Brad
Brad

Reputation: 163234

What I do when I want the sincedb to do nothing is point it at /dev/null. Then, the file is effectively new every time.

This sincedb is per-file.

Upvotes: 1

Related Questions