CODEWITHSUNDEEP
go
Valentyn Shybanov
Valentyn Shybanov

Reputation: 19401

Concurrent limited consumers using goroutins and channels

I've tried to reproduce "approach that manages resources well is to start a fixed number of handle goroutines all reading from the request channel." from Effective Go and found 

fatal error: all goroutines are asleep - deadlock!

The idea is simple: have 1 queue and 1 results channels and several limited number of "workers".

My code is in Go Playground

queue := make(chan *Request)
result := make(chan int)
quit := make(chan bool)
go Serve(queue, quit)
for i := 0; i < 10; i++ {
    req := Request{i, result}
    queue <- &req
}
close(queue)
for i := 0; i < 10; i++ {
    fmt.Printf("Finished %d\n", <-result)
}
fmt.Printf("All finished\n")
quit <- true

Function Serve:

func handle(queue chan *Request) {
    for r := range queue {
        //process(r)
        fmt.Printf("Processing %d\n", r.i)
        r.r <- r.i

    }
}

func Serve(clientRequests chan *Request, quit chan bool) {
    MaxOutstanding := 2
    // Start handlers
    for i := 0; i < MaxOutstanding; i++ {
        go handle(clientRequests)
    }
    <-quit // Wait to be told to exit.
}

What is wrong? Or may be there is simplier solution for implementing limited number of workers that process requests?

Upvotes: 0

Views: 148

Answers (1)

twotwotwo
twotwotwo

Reputation: 30097

The result channel isn't buffered, so each send blocks the goroutine until something receives the result. So after the first two items are processed, the two handler routines are waiting for their results to be received. But the code to receive work items isn't running yet, so they're stuck. Then main() tries to send the next work item, and the handlers aren't ready to receive it so main() is now stuck too.

One way to fix it is to start a goroutine to receive results in the background before you queue the work items. Here's a version of your code that receives the results in the background, plus makes main() wait until all results have been received and handled: http://play.golang.org/p/_CKn3CxQFc.

In your example, you can know when it's safe to quit just by counting results received. But if a situation comes up where you need to figure out when you're done and counting is not enough, then 1) each worker can signal when it's done, 2) Serve can close the result channel once all workers signal done, 3) your result-handling goroutine can send a quit to main() when all the results have been handled. There are lots of variations on that; you can see code at http://play.golang.org/p/12wZbm0rxa

Upvotes: 1

Related Questions