CODEWITHSUNDEEP
linuxfilego
Chaos
Chaos

Reputation: 471

Wait until a file is finished copying when using a directory watcher in Go

I'm using the golang watcher package to watch a directory of files. https://github.com/radovskyb/watcher

The only issue is that the Create event is triggered when a file begins being copied/moved, not when it's actually ready. I was wondering if there was a neat way in Go to sort of wait after the create event until the file is no longer being written to before continuing on.

I suppose this waiting would most likely have to be done in a goroutine to not block any other file events.

Simple example:

package main

import (
        "fmt"
        "log"
        "time"

        "github.com/radovskyb/watcher"
)

func main() {
        w := watcher.New()

        w.FilterOps(watcher.Create, watcher.Write)

        go func() {
                for {
                        select {
                        case event := <-w.Event:
                                fmt.Println(event) // Print the event's info.
                                if event.Op == watcher.Create {
                                    // HERE WE DO STUFF
                                    doStuff(event)
                                }
                        case err := <-w.Error:
                                log.Fatalln(err)
                        case <-w.Closed:
                                return
                        }
                }
        }()

        // Watch this folder for changes.
        if err := w.Add("./files/"); err != nil {
                log.Fatalln(err)
        }


        // Start the watching process - it'll check for changes every 100ms.
        if err := w.Start(time.Millisecond * 100); err != nil {
                log.Fatalln(err)
        }
}

Thank you!

Upvotes: 2

Views: 4033

Answers (3)

Piero
Piero

Reputation: 1716

Another way can be debouncing the fsnotify events so that a function would be called after the write events for a specific item did not happen in a timeframe.

example :

package main

import (
    "log"
    "sync"
    "time"

    "github.com/fsnotify/fsnotify"
    "github.com/romdo/go-debounce"
)

type debouncerMap struct {
    mu sync.RWMutex
    m  map[string]func()
}

func newDebouncerMap() *debouncerMap {
    return &debouncerMap{
        m: make(map[string]func()),
    }
}

func (dm *debouncerMap) getDebouncer(filename string) func() {
    dm.mu.RLock()
    debouncer, ok := dm.m[filename]
    dm.mu.RUnlock()
    if !ok {
        dm.mu.Lock()
        debouncer, ok = dm.m[filename]
        if !ok {
            debouncer, _ = debounce.New(100*time.Millisecond, func() {
        dm.mu.Lock()
                delete(dm.m, filename)
                dm.mu.Unlock()
                log.Println("modified file:", filename)
            })
            dm.m[filename] = debouncer
        }
        dm.mu.Unlock()
    }
    return debouncer
}

func main() {

    // Create new watcher.
    watcher, err := fsnotify.NewWatcher()
    
    if err != nil {
        log.Fatal(err)
    }
    defer watcher.Close()

    debouncerMap := newDebouncerMap()

    // Start listening for events.
    go func() {
        for {
            select {
            case event, ok := <-watcher.Events:
                if !ok {
                    return
                }
                //log.Println("event:", event)
                if event.Has(fsnotify.Write) {
                    //log.Println("modified file:", event.Name)
                    debouncer := debouncerMap.getDebouncer(event.Name)
                              debouncer()

               
                }
            case err, ok := <-watcher.Errors:
                if !ok {
                    return
                }
                log.Println("error:", err)
            }
        }
    }()

    // Add a path.
    err = watcher.Add("C:\\temp\\items")
    if err != nil {
        log.Fatal(err)
    }

    // Block main goroutine forever.
    <-make(chan struct{})
}

Upvotes: 0

seaguest
seaguest

Reputation: 2630

fkocik has committed a PR which implements IN_CLOSE_WRITE, and it does work for me on Linux.

ADDED: CloseWrite event support

NB: the code is not present in the master branch yet, you nee retrieve it manually.

Upvotes: 1

John Zwinck
John Zwinck

Reputation: 249143

The library you are using cannot do what you are asking. The docs say:

watcher is a Go package for watching for files or directory changes (recursively or non recursively) without using filesystem events, which allows it to work cross platform consistently.

When I first saw this question, I thought it would be easily solved by waiting for the "file close" event, which will occur when a file copy is complete. But because the library you're using refuses to utilize platform-specific APIs, it cannot know this information.

You can solve your problem more efficiently and accurately by using inotify. There is https://github.com/fsnotify/fsnotify but it does not support the IN_CLOSE_WRITE event (see https://github.com/golang/go/issues/15406). Perhaps you can use it as inspiration to write your own inotify wrapper which does support IN_CLOSE_WRITE (and probably little else).

One notable limitation of inotify is that it will not work with NFS, but there are limitations with watcher too (and you wouldn't want to use its polling mechanism on a network share, at least not rapidly).

Upvotes: 2

Related Questions