CODEWITHSUNDEEP
gnu-parallel
FGV
FGV

Reputation: 165

GNU parallel sorted stdout and stderr

I've been using GNU parallel and I want to keep output order (--kepp-order), grouped by jobs (--grouped) but also with sorted stdout and stderr. Right now, the grouped options first print stdout and only after does it print stderr.

As an example, any way that these two commands give the same output?

seq 4 | parallel -j0 'sleep {}; echo -n start{}>&2; sleep {}; echo {}end'

seq 4 | parallel -j0 'sleep {}; echo -n start{}   ; sleep {}; echo {}end'

thanks,

Upvotes: 1

Views: 3234

Answers (3)

methuselah-0
methuselah-0

Reputation: 124

Assuming that you don't have to use gnu parallel, and the main requirements are parallel execution with maintained ordered output of both stderr and stdout; we can create a solution that allows for the following example usage(plus providing return code), where you will have the results of the executions in a list, where each list element is in return a list of 3 strings: indexed as 0=stdout, 1=stderr and 2=return code.

source mapfork.sh
ArgsMap=("-Pn" "-p" "{}" "{}")
Args=("80" "google.com" "25" "tutanota.com" "80" "apa bepa")
declare -a Results=$(mapfork nmap "(${ArgsMap[*]@Q})" "(${Args[*]@Q})")

So, in order to print for example the stderr results, of the third destination ("apa bepa"), you can do:

declare -a res3="${Results[2]}"
declare -p res3
# declare -a res3=([0]=$'Starting Nmap 7.70 ( https://nmap.org ) at 2019-06-21 18:55 CEST\nNmap done: 0 IP addresses (0 hosts up) scanned in 0.09 seconds' [1]=$'Failed to resolve "apa bepa".\nWARNING: No targets were specified, so 0 hosts scanned.' [2]="0")
printf '%b\n' "${res3[1]}"

mapfork.sh is shown below. It is a bit complicated but it's parts have been explained in other answers so I won't provide the details here as well:

Capture both stdout and stderr in Bash [duplicate]

How can I make an array of lists (or similar) in bash?

#!/bin/bash
# reference: https://stackoverflow.com/questions/13806626/capture-both-stdout-and-stderr-in-bash
nullWrap(){
    local -i i; i="$1"
    local myCommand="$2"
    local -a myCommandArgs="$3"
    local myfifo="$4"
    local stderr
    local stdout
    local stdret
    . <(\
    { stderr=$({ stdout=$(eval "$myCommand ${myCommandArgs[*]@Q}"); stdret=$?; } 2>&1 ;\
               declare -p stdout >&2 ;\
           declare -p stdret >&2) ;\
      declare -p stderr;\
    } 2>&1)
    local -a Arr=("$stdout" "$stderr" "$stdret")
    printf "${i}:%s\u0000" "(${Arr[*]@Q})" > "$myfifo"
}
mapfork(){
    local command
    command="$1"
    local -a CommandArgs="$2"
    local -a Args="$3"
    local -a PipedArr
    local -i i
    local myfifo=$(mktemp /tmp/temp.XXXXXXXX)
    rm "$myfifo"
    mkfifo "$myfifo"

    local -a placeHolders=()
    for ((i=0;i<${#CommandArgs[@]};i++)); do
    [[ "${CommandArgs[$i]}" =~ ^\{\}$ ]] && placeHolders+=("$i") ;done

    for ((i=0;i<${#Args[@]};i+=0)); do
    # if we have placeholders in CommandArgs we need to take args
    # from Args to replace.
    if [[ ${#placeHolders[@]} -gt 0 ]]; then
        for ii in "${placeHolders[@]}"; do
        CommandArgs["$ii"]="${Args[$i]}"
        i+=1; done; fi
    nullWrap "$i" "$command" "(${CommandArgs[*]@Q})" "$myfifo" &
    done
    for ((i=0;i<${#Args[@]};i+=$(("${#placeHolders[@]}")))) ; do
    local res
    res=$(read -d $'\u0000' -r temp <"$myfifo" && printf '%b' "$temp")
    local -i resI
    resI="${res%%:*}"
    PipedArr[$resI]="${res#*:}"
    done
    # reference: https://stackoverflow.com/questions/41966140/how-can-i-make-an-array-of-lists-or-similar-in-bash
    printf '%s' "(${PipedArr[*]@Q})"
}

Upvotes: 0

cnst
cnst

Reputation: 27238

As per the comment to the other answer, to keep the output ordered, simply have parallel's bash invocation redirect stderr to stdout:

parallel myfunc '2>&1'

E.g.,

parallel -j8 eval \{1} -w1 \{2} '2>&1' ::: "traceroute -a -f9" traceroute6 ::: ordns.he.net one.one.one.one google-public-dns-a.google.com

Upvotes: 2

Ole Tange
Ole Tange

Reputation: 33740

You cannot do that if you still want stderr and stdout to be separated.

The reason for this is that stderr and stdout are buffered to 2 different files using buffered output.

But maybe you can explain a bit more on what you need this for. In that case there might be a solution.

Upvotes: 1

Related Questions