CODEWITHSUNDEEP
jsonbashjqaws-clitee
Gismo Ranas
Gismo Ranas

Reputation: 6442

Bash - Pipe first line into variable and display the rest without using a temp file

I receive some json that I process until it becomes just text lines. In the first line there's a value that I would like to keep in a variable and all the rest after the first line should be displayed with less or other utils.

Can I do this without using a temporary file?

The context is this: 

aws logs get-log-events --log-group-name "$logGroup" --log-stream-name "$logStreamName" --limit "$logSize" |
 jq '{message:.nextForwardToken}, .events[] | .message' |
 sed 's/^"//g' | sed 's/"$//g'

In the first line there's the nextForwardToken that I want to put in the variable and all the rest is log messages.

The json looks like this:

{
"events": [
    {
        "timestamp": 1518081460955,
        "ingestionTime": 1518081462998,
        "message": "08.02.2018 09:17:40.955 [SimpleAsyncTaskExecutor-138] INFO  o.s.b.c.l.support.SimpleJobLauncher - Job: [SimpleJob: [name=price-update]] launched with the following parameters: [{time=1518081460875, sku=N-W7ZLH9U737B|N-XIBH22XQE87|N-3EXIRFNYNW0|N-U19C031D640|N-6TQ1847FQE6|N-NF0XCNG0029|N-UJ3H0OZROCQ|N-W2JKJD4S6YP|N-VEMA4QVV3X1|N-F40J6P2VM01|N-VIT7YEAVYL2|N-PKLKX1PAUXC|N-VPAK74C75DP|N-C5BLYC5HQRI|N-GEIGFIBG6X2|N-R0V88ZYS10W|N-GQAF3DK7Y5Z|N-9EZ4FDDSQLC|N-U15C031D668|N-B8ELYSSFAVH}]"
    },
    {
        "timestamp": 1518081461095,
        "ingestionTime": 1518081462998,
        "message": "08.02.2018 09:17:41.095 [SimpleAsyncTaskExecutor-138] INFO  o.s.batch.core.job.SimpleStepHandler - Executing step: [index salesprices]"
    },
    {
        "timestamp": 1518082421586,
        "ingestionTime": 1518082423001,
        "message": "08.02.2018 09:33:41.586 [upriceUpdateTaskExecutor-3] DEBUG e.u.d.a.j.d.b.StoredMasterDataReader - Reading page 1621"
    }
],
"nextBackwardToken": "b/33854347851370569899844322814554152895248902123886870536",
"nextForwardToken": "f/33854369274157730709515363051725446974398055862891970561"
}

I need to put in a variable this:

f/33854369274157730709515363051725446974398055862891970561

and display (or put in an other variable) the messages:

08.02.2018 09:17:40.955 [SimpleAsyncTaskExecutor-138] INFO  o.s.b.c.l.support.SimpleJobLauncher - Job: [SimpleJob: [name=price-update]] launched with the following parameters: [{time=1518081460875, sku=N-W7ZLH9U737B|N-XIBH22XQE87|N-3EXIRFNYNW0|N-U19C031D640|N-6TQ1847FQE6|N-NF0XCNG0029|N-UJ3H0OZROCQ|N-W2JKJD4S6YP|N-VEMA4QVV3X1|N-F40J6P2VM01|N-VIT7YEAVYL2|N-PKLKX1PAUXC|N-VPAK74C75DP|N-C5BLYC5HQRI|N-GEIGFIBG6X2|N-R0V88ZYS10W|N-GQAF3DK7Y5Z|N-9EZ4FDDSQLC|N-U15C031D668|N-B8ELYSSFAVH}]
08.02.2018 09:17:41.095 [SimpleAsyncTaskExecutor-138] INFO  o.s.batch.core.job.SimpleStepHandler - Executing step: [index salesprices]
08.02.2018 09:33:41.586 [upriceUpdateTaskExecutor-3] DEBUG e.u.d.a.j.d.b.StoredMasterDataReader - Reading page 1621

Thanks in advance for your help.

Upvotes: 0

Views: 1690

Answers (4)

peak
peak

Reputation: 116870

I believe the following meets the stated requirements, assuming a bash-like environment:

x=$(aws ... |
    tee >(jq -r '.events[] | .message' >&2) |
    jq .nextForwardToken) 2>&1

This makes the item of interest available as the shell variable $x.

Notice that the string manipulation using sed can be avoided by using the -r command-line option of jq.

Calling jq just once

x=$(aws ... |
 jq -r '.nextForwardToken, (.events[] | .message)' |
 tee >(tail -n +2 >&2) |
 head -n 1) 2>&1

echo "x=$x"

Upvotes: 1

RomanPerekhrest
RomanPerekhrest

Reputation: 92854

Straightforwardly:

aws logs get-log-events --log-group-name "$logGroup" \
--log-stream-name "$logStreamName" --limit "$logSize" > /tmp/log_data

-- set nextForwardToken variable:

nextForwardToken=$(jq -r '.nextForwardToken' /tmp/log_data)
echo $nextForwardToken 
f/33854369274157730709515363051725446974398055862891970561

-- print all message items:

jq -r '.events[].message' /tmp/log_data

08.02.2018 09:17:40.955 [SimpleAsyncTaskExecutor-138] INFO  o.s.b.c.l.support.SimpleJobLauncher - Job: [SimpleJob: [name=price-update]] launched with the following parameters: [{time=1518081460875, sku=N-W7ZLH9U737B|N-XIBH22XQE87|N-3EXIRFNYNW0|N-U19C031D640|N-6TQ1847FQE6|N-NF0XCNG0029|N-UJ3H0OZROCQ|N-W2JKJD4S6YP|N-VEMA4QVV3X1|N-F40J6P2VM01|N-VIT7YEAVYL2|N-PKLKX1PAUXC|N-VPAK74C75DP|N-C5BLYC5HQRI|N-GEIGFIBG6X2|N-R0V88ZYS10W|N-GQAF3DK7Y5Z|N-9EZ4FDDSQLC|N-U15C031D668|N-B8ELYSSFAVH}]    
08.02.2018 09:17:41.095 [SimpleAsyncTaskExecutor-138] INFO  o.s.batch.core.job.SimpleStepHandler - Executing step: [index salesprices]
08.02.2018 09:33:41.586 [upriceUpdateTaskExecutor-3] DEBUG e.u.d.a.j.d.b.StoredMasterDataReader - Reading page 1621

Upvotes: 1

Inian
Inian

Reputation: 85780

If you are interested in storing the contents to variables, use mapfile or read on older bash versions.

Just using read to get the first line do. I've added -r flag to jq print output without quotes

read -r token < <(aws logs get-log-events --log-group-name "$logGroup" --log-stream-name "$logStreamName" --limit "$logSize" | jq -r '{message:.nextForwardToken}, .events[] | .message')
printf '%s\n' "$token"

Or using mapfile

mapfile -t output < <(aws logs get-log-events --log-group-name "$logGroup" --log-stream-name "$logStreamName" --limit "$logSize" | jq -r '{message:.nextForwardToken}, .events[] | .message')

and loop through the array. The first element will always contain the token-id you want.

printf '%s\n' "${output[0]}"

Rest of the elements can be iterated over,

for ((i=1; i<${#output[@]}; i++)); do
    printf '%s\n' "${output[i]}"
done

Upvotes: 1

silel
silel

Reputation: 587

You might consider it a bit of trick, but you can use tee to pipe all the output to stderr and fetch the one line you want for your variable with head:

var="$(command | tee /dev/stderr | head -n 1)"

Or you can solve this with a bit of scripting:

first=true
while read -r line; do
    if $first; then
        first=false
        var="$line"
    fi
    echo "$line"
done < <(command)

Upvotes: 2

Related Questions