Goroutines, Channels, WaitGroups, and select (just trying to understand)

Question

I am not necessarily trying to accomplish something specific, more just understand how goroutines, channels, waitgroups, and select (on channels) plays together. I am writing a simple program that loops through an slice of URLs, fetches the URL, then basically just ends. The simple idea is that I want all of the fetches to occur and return, send their data over channels, and then end once all fetches have occurred. I am almost there, and I know I am missing something in my select that will end the loop, something to say "hey the waitgroup is empty now", but I am unsure how to best do that. Mind taking a look and clearing it up for me? Right now everything runs just fine, it just doesn't terminate, so clearly I am missing something and/or not understanding how some of these components should work together.

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "sync"
)

var urls = []string{
    "https://www.google.com1",
    "https://www.gentoo.org",
}

var wg sync.WaitGroup

// simple struct to store fetching
type urlObject struct {
    url     string
    success bool
    body    string
}

func getPage(url string, channelMain chan urlObject, channelError chan error) {

    // increment waitgroup, defer decrementing
    wg.Add(1)
    defer wg.Done()
    fmt.Println("fetching " + url)

    // create a urlObject
    uO := urlObject{
        url:     url,
        success: false,
    }

    // get URL
    response, getError := http.Get(url)

    // close response later on
    if response != nil {
        defer response.Body.Close()
    }

    // send error over error channel if one occurs
    if getError != nil {
        channelError <- getError
        return
    }

    // convert body to []byte
    body, conversionError := ioutil.ReadAll(response.Body)
    // convert []byte to string
    bodyString := string(body)

    // if a conversion error happens send it over the error channel
    if conversionError != nil {
        channelError <- conversionError
    } else {
        // if not send a urlObject over the main channel
        uO.success = true
        uO.body = bodyString
        channelMain <- uO
    }

}

func main() {

    var channelMain = make(chan urlObject)
    var channelError = make(chan error)

    for _, v := range urls {
        go getPage(v, channelMain, channelError)
    }

    // wait on goroutines to finish
    wg.Wait()

    for {
        select {
        case uO := <-channelMain:
            fmt.Println("completed " + uO.url)
        case err := <-channelError:
            fmt.Println("error: " + err.Error())
        }
    }

}

Answer 1

You need to make the following changes:

• As people have mentioned, you probably want to call wg.Add(1) in the main function, before calling your goroutine. That way you KNOW it occurs before the defer wg.Done() call.
• Your channel reads will block, unless you can figure out a way to either close the channels in your goroutines, or make them buffered. Probably the easiest way is to make them buffered, e.g., var channelMain = make(chan urlObject, len(urls))
• The break in your select statement is going to only exit the select, not the containing for loop. You can label the for loop and break to that, or use some sort of conditional variable.

Playground link to working version: https://play.golang.org/p/WH1fm2MhP-L

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "sync"
)

var urls = []string{
    "https://www.google.com1",
    "https://www.gentoo.org",
}

var wg sync.WaitGroup

// simple struct to store fetching
type urlObject struct {
    url     string
    success bool
    body    string
}

func getPage(url string, channelMain chan urlObject, channelError chan error) {

    // increment waitgroup, defer decrementing
    defer wg.Done()
    fmt.Println("fetching " + url)

    // create a urlObject
    uO := urlObject{
        url:     url,
        success: false,
    }

    // get URL
    response, getError := http.Get(url)

    // close response later on
    if response != nil {
        defer response.Body.Close()
    }

    // send error over error channel if one occurs
    if getError != nil {
        channelError <- getError
        return
    }

    // convert body to []byte
    body, conversionError := ioutil.ReadAll(response.Body)
    // convert []byte to string
    bodyString := string(body)

    // if a conversion error happens send it over the error channel
    if conversionError != nil {
        channelError <- conversionError
    } else {
        // if not send a urlObject over the main channel
        uO.success = true
        uO.body = bodyString
        channelMain <- uO
    }

}

func main() {

    var channelMain = make(chan urlObject, len(urls))
    var channelError = make(chan error, len(urls))

    for _, v := range urls {
        wg.Add(1)
        go getPage(v, channelMain, channelError)
    }

    // wait on goroutines to finish
    wg.Wait()

    for done := false; !done; {
        select {
        case uO := <-channelMain:
            fmt.Println("completed " + uO.url)
        case err := <-channelError:
            fmt.Println("error: " + err.Error())
        default:
            done = true
        }
    }

}