CODEWITHSUNDEEP
bashcurl
InvalidSyntax
InvalidSyntax

Reputation: 9495

How to fetch next URL from JSON using Bash

I have created a simple script to store response from a third party API

The request is like this..

https://externalservice.io/orders?key=password&records=50&offset=0

The response is as follows:

{
  "results": [
    {
      "engagement": {
        "id": 29090716,
        "portalId": 62515,
        "active": true,
        "createdAt": 1444223400781,
        "lastUpdated": 1444223400781,
        "createdBy": 215482,
        "modifiedBy": 215482,
        "ownerId": 70,
        "type": "NOTE",
        "timestamp": 1444223400781
      },
    },
  ],
  "hasMore": true,
  "offset": 4623406
}

If there is a hasMore attribute, I need to read the offset value to get the next set of records.

Right now I've created a script that simply loops over the estimated number of records (I believed there is) and thought incrementing the offset would work but this is not the case as the offset is not incremental.

#!/bin/bash

for i in {1..100}; do
    curl -s "https://externalservice.io/orders?key=password&records=50&offset=$i" >>outfile.txt 2>&1
done

Can someone explain how I can read continue the script reading the offset value until hasMore=false?

Upvotes: 0

Views: 615

Answers (2)

pawrog
pawrog

Reputation: 118

You can use jq utility to extract specific attribute from you json response:

$ jq -r ".hasMore" outfile

true

jq expects perfectly valid json input, otherwise it will print error. Most common mistake is to echo json stored in variable. A mistake, because echo will interpret all escaped newlines in your json and will send unescaped newlines within values causing jq to throw parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line message.

To avoid modification to json (and avoiding an error) we need to use printf instead of echo.

Here is the solution without breaking json content:

more="true"
offset=0
while [ $more = "true" ]; do
    echo $offset
    response=$(curl -s "https://example.com/orders?offset=$offset")
    more=$(printf "%s\n" "$response" | jq -r ".hasMore")
    offset=$(printf "%s\n" "$response" | jq -r ".offset")
done

Upvotes: 0

rpld
rpld

Reputation: 86

You can read a value from a json using the jq utility:

$ jq -r ".hasMore" outfile

true

Here is what you could use:

more="true"
offset=0
while [ $more = "true" ]; do
    echo $offset
    response=$(curl -s "https://example.com/orders?offset=$offset")
    more=$(echo $response | tr '\r\n' ' ' | jq -r ".hasMore")
    offset=$(echo $response | tr '\r\n' ' ' | jq -r ".offset")
done

Upvotes: 1

Related Questions