CODEWITHSUNDEEP
elasticsearchgologstashelastic-stackfilebeat
droidlabour
droidlabour

Reputation: 697

How filebeat checks for new content in a file?

Does filebeat uses tail -f to check for new contents in a file and then flushes it over to the desired output? Or is there any other way it checks for new contents in a file?

Upvotes: 0

Views: 1129

Answers (1)

Will Barnwell
Will Barnwell

Reputation: 4089

Since filebeat is open source, you can always go look yourself

Here's the go code from the above linked file which checks if a file has been updated.

I've heavily abridged this code, anywhere you see ... is a code block that was not exactly relevant, I encourage anybody reading this too go look at the entire file, its some pretty well written go.

 // Scan starts a scanGlob for each provided path/glob
func (p *ProspectorLog) scan() {

    newlastscan := time.Now()

    // Now let's do one quick scan to pick up new files
    for _, path := range p.config.Paths {
        p.scanGlob(path)
    }
    p.lastscan = newlastscan
}

The above function gets called every n-length time block where n is specified in the configuration. ScanGlob gets called, and is shown below.

// Scans the specific path which can be a glob (/**/**/*.log)
// For all found files it is checked if a harvester should be started
func (p *ProspectorLog) scanGlob(glob string) {

    ...

    // Evaluate the path as a wildcards/shell glob
    matches, err := filepath.Glob(glob)
    ...

    // Check any matched files to see if we need to start a harvester
    for _, file := range matches {
        ...

For all files which matched the glob, check the statistics on the file using the OS specific call, for linux this would be stat <file> 

        // Stat the file, following any symlinks.
        fileinfo, err := os.Stat(file)
        ...

Based on the stat call, it is decided if a harvester, the part of this go application which reads the files, needs to be started.

        // Conditions for starting a new harvester:
        // - file path hasn't been seen before
        // - the file's inode or device changed
        if !isKnown {
            p.checkNewFile(h)
        } else {
            h.Stat.Continue(&lastinfo)
            p.checkExistingFile(h, &newFile, &oldFile)
        }

        // Track the stat data for this file for later comparison to check for
        // rotation/etc
        p.prospectorList[h.Path] = *h.Stat
    }
}

TL;DR: Filebeat used the files statistics reported by the OS to see if the file has been updated since the last time it harvested the file.

Upvotes: 1

Related Questions