CODEWITHSUNDEEP
goconcurrencygoroutine
Jano
Jano

Reputation: 455

Deadlock on simple program, can't identify the reason

I'm facing a deadlock error in the following code, but I can't identify why. The goal of the code is to get the sha256 hash of all files in a given folder, recursively. (End goal is to remove duplicated files based on hash, as a learning project)

For that it spaws a goroutine that spawns another goroutine for each file to get the hash. The hash is stores in a channel and read at the end of the code in a simple loop.

Based on the prints, all files are checked, and the wg.Add and Done seem to be on the right place.

Code:

package main

import (
    "crypto/sha256"
    "fmt"
    "io"
    "io/fs"
    "log"
    "os"
    "path/filepath"
    "sync"
    "time"
)

func get_fsha256sum(filepath string) string {
    f, err := os.Open(filepath)
    if err != nil {
        log.Fatal(err)
    }
    defer f.Close()
    return get_sha256sum(f)
}

func get_sha256sum(r io.Reader) string {
    h := sha256.New()
    if _, err := io.Copy(h, r); err != nil {
        log.Fatal(err)
    }
    return fmt.Sprintf("%x", h.Sum(nil))
}

func main() {
    start := time.Now()
    ch := make(chan string, 10000)
    var wg sync.WaitGroup

    var get_fsha256sum_wrap = func(filepath string) {
        start := time.Now()
        ch <- get_fsha256sum(filepath)
        fmt.Printf("%f-%s\n", filepath, time.Since(start))
        wg.Done()
    }

    var walk_func = func(filepath string, info fs.DirEntry, err error) error {
        wg.Add(1)
        if err != nil {
            return err
        }
        if info.IsDir() {
            return nil
        }

        go get_fsha256sum_wrap(filepath)
        return nil

    }

    var over_wrap = func(root string) {
        filepath.WalkDir(root, walk_func)
        wg.Wait()
        close(ch)
    }
    go over_wrap("/run/media/user/Red/pictures_master/2021/01")
    for v := range ch {
        fmt.Println(v)
    }
    fmt.Printf("%s", time.Since(start))
}

Error Message:

fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan receive]:
main.main()
    /home/user/go_phockup/main.go:66 +0x273

goroutine 18 [semacquire]:
sync.runtime_Semacquire(0xc0000b8018)
    /usr/local/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc0000b8010)
    /usr/local/go/src/sync/waitgroup.go:130 +0x65
main.main.func3(0x4e3d3f, 0x2b)
    /home/go_phockup/main.go:62 +0x5e
created by main.main
    /home/user/go_phockup/main.go:65 +0x1bd
exit status 2

Any idea what's wrong?

Upvotes: 0

Views: 59

Answers (1)

Burak Serdar
Burak Serdar

Reputation: 51632

Your waitgroup is waiting for more goroutines than you created. The walk_func adds to the wg, but may return without creating a new goroutine. Replace with:

var walk_func = func(filepath string, info fs.DirEntry, err error) error{
        if err != nil {
            return err
        }
        if info.IsDir() {
            return nil
        }
        
        wg.Add(1)
        go get_fsha256sum_wrap(filepath)
        return nil

        }

Upvotes: 2

Related Questions