Deadlock error in golang

Question

I recently looked at go and got hooked, it looks so interesting! After completing the tutorial I wanted to build something by myself: I want to list all of my songs from my music library. I think I can profit from go's concurrency here. While on routine is walking down the directory tree it pushes music files (path to those files) into a channel which are then picked up by another routine that reads the ID3 tags, so I don't have to wait until every file has been found.

This is my simple and naive approach:

package main

import (
    "fmt"
    "os"
    "path/filepath"
    "strings"
    "sync"
)

const searchPath = "/Users/luma/Music/test" // 5GB of music.

func main() {
    files := make(chan string)

    var wg sync.WaitGroup
    wg.Add(2)

    go printHashes(files, &wg)
    go searchFiles(searchPath, files, &wg)

    wg.Wait()
}

func searchFiles(searchPath string, files chan<- string, wg *sync.WaitGroup) {
    visit := func(path string, f os.FileInfo, err error) error {
        if !f.IsDir() && strings.Contains(".mp4.mp3.flac", filepath.Ext(f.Name())) {
            files <- path
        }
        return err
    }

    if err := filepath.Walk(searchPath, visit); err != nil {
        fmt.Println(err)
    }

    wg.Done()
}

func printHashes(files <-chan string, wg *sync.WaitGroup) {
    for range files {
        fmt.Println(<-files)
    }

    wg.Done()
}

This program doesn't read the tags, yet. Instead it just prints the file path. This works, it lists all music files extremely fast! But I see this error after the program finishes:

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc42007205c)
    /usr/local/Cellar/go/1.7.4_2/libexec/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc420072050)
    /usr/local/Cellar/go/1.7.4_2/libexec/src/sync/waitgroup.go:131 +0x97
main.main()
    /Users/luma/Code/Go/src/github.com/LuMa/test/main.go:22 +0xfa

goroutine 17 [chan receive]:
main.printHashes(0xc42008e000, 0xc420072050)
    /Users/luma/Code/Go/src/github.com/LuMa/test/main.go:42 +0xb4
created by main.main
    /Users/luma/Code/Go/src/github.com/LuMa/test/main.go:19 +0xab
exit status 2

What is causing the deadlock?

Answer 1

Within searchFiles, you want to close(files) when done sending. This convention is called sender-closes (receivers never close). Also, remove the call to wg.Done() as you are not done... There could still be items on the channel.

The close(files) will signal the for range files to close and exit the loop, which will call your wg.Done() to signal the main function that everything is done.

(Untested on mobile)

package main

import (
    "fmt"
    "os"
    "path/filepath"
    "strings"
    "sync"
)

const searchPath = "/Users/luma/Music/test" // 5GB of music.

func main() {
    files := make(chan string)

    var wg sync.WaitGroup
    wg.Add(1)

    go printHashes(files)
    go searchFiles(searchPath, files, &wg)

    wg.Wait()
}

func searchFiles(searchPath string, files chan<- string) {
    visit := func(path string, f os.FileInfo, err error) error {
        if !f.IsDir() && strings.Contains(".mp4.mp3.flac", filepath.Ext(f.Name())) {
            files <- path
        }
        return err
    }

    if err := filepath.Walk(searchPath, visit); err != nil {
        fmt.Println(err)
    }
    close(files)
}

func printHashes(files <-chan string, wg *sync.WaitGroup) {
    defer wg.Done()
    for range files {
        fmt.Println(<-files)
    }
}

Note that while this may seem fast, using a single goroutine is fine and unblocks the main goroutine too. But, you may not gain any advantage if you try to read multiple files for id3 tags in multiple goroutines - they will all share the same file i/o lock at the syscall level. The only way that would advantageous would be if the processing of data far out weighs the file i/o locking (e.g. something big in computation, because processing is far faster than syscall locks).

PS, welcome to the Go community!

Answer 2

Because you need close files channel. In your case, you don't close it, so

for range files { fmt.Println(<-files) } will wait get value from files channel. so wg.Done() will never done in printHashes.

func searchFiles(searchPath string, files chan<- string, wg *sync.WaitGroup) {
    visit := func(path string, f os.FileInfo, err error) error {
        if !f.IsDir() && strings.Contains(".mp4.mp3.flac", filepath.Ext(f.Name())) {
            files <- path
        }
        return err
    }

    if err := filepath.Walk(searchPath, visit); err != nil {
        fmt.Println(err)
    }

    wg.Done()
    close(files) // close the chanel, because you don't put thing into the channel anymore.
}