CODEWITHSUNDEEP
go
mrosales
mrosales

Reputation: 1568

Golang pipe output of subcommand real time

I am trying to pipe the output of a command, but no data seems to be read from the pipe until the write end is closed. Eventually I want this to connect to a websocket that streams the status of a command while it is being executed. The problem is that while this code prints the messages line by line, it does not print anything until the program has finished executing.

cmd := exec.Command(MY_SCRIPT_LOCATION, args)

// create a pipe for the output of the script
// TODO pipe stderr too
cmdReader, err := cmd.StdoutPipe()
if err != nil {
    fmt.Fprintln(os.Stderr, "Error creating StdoutPipe for Cmd", err)
    return
}

scanner := bufio.NewScanner(cmdReader)
go func() {
    for scanner.Scan() {
        fmt.Printf("\t > %s\n", scanner.Text())
    }
}()

err = cmd.Start()
if err != nil {
    fmt.Fprintln(os.Stderr, "Error starting Cmd", err)
    return
}

err = cmd.Wait()
if err != nil {
    fmt.Fprintln(os.Stderr, "Error waiting for Cmd", err)
    return
}

Is there any way I can do something similar and have a scanner actually read line by line as it is written to the pipe instead of after everything has been written? The program takes about 20 seconds to run, and there is a steady stream of updates, so it is annoying to have them all go through at once.

Upvotes: 4

Views: 3801

Answers (2)

KEINOS
KEINOS

Reputation: 1224

F.Y.I

read line by line as it is written to the pipe instead of after everything has been written?

I found the package go-pipeline very useful for those who came to this topic by googling.

The below is equivalent to git log --online | grep first import | wc -l

package sample

import (
    "fmt"
    "log"

    "github.com/mattn/go-pipeline"
)

func ExampleCommandPipeLine() {
    out, err := pipeline.Output(
        []string{"git", "log", "--oneline"},
        []string{"grep", "first import"},
        []string{"wc", "-l"},
    )
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(string(out))
    // Output:
    // 1
}

Upvotes: 0

mrosales
mrosales

Reputation: 1568

Turns out that the issue was not in the code I posted above. That works as expected. The problem was that the C program that was being executed was not properly flushing stdout. When running it interactively, it worked as expected, but when stdout was piped, it would not actually get written until I called flush. After manually adding some flush statements to the c program, the go code worked as expected.

Upvotes: 6

Related Questions