jq - Parsing fields with hyphen - Invalid Numeric Literal

Question

I'm trying to pull a list of product categories from an API using jq and some nested for-loops. I have to pull the category ID first, then I'm able to pull product details. Some of the category IDs have hypens and jq seems to be treating them like math instead of a string, and I've tried every manner of quoting but I'm still running into this error. In Powershell, I'm able to pull the list just fine, but I really need this to work in bash.

Here's the expected list:

aprons
backpacks
beanies
bracelet
coaster
cutting-board
dress-shirts
duffel-bags
earring
full-brim-hats
generic-dropoff
hats
etc...

And trying to recreate the same script in Bash, here's the output:

aprons
backpacks
beanies
bracelet
coaster
parse error: Invalid numeric literal at line 1, column 6
parse error: Invalid numeric literal at line 1, column 7
parse error: Invalid numeric literal at line 1, column 5
earring
parse error: Invalid numeric literal at line 2, column 0
parse error: Invalid numeric literal at line 1, column 5
parse error: Invalid numeric literal at line 1, column 8
hats
etc...

You can see that it's running into this error with all values that contain hyphens. Here's my current script:

#!/bin/bash

CATEGORIES=$(curl -s https://api.scalablepress.com/v2/categories)
IFS=$' \t\n'


for CATEGORY in $(echo $CATEGORIES | jq -rc '.[]')
do
CATEGORY_IDS=$(echo $CATEGORY | jq -rc '."categoryId"')
for CATEGORY_ID in $(echo $CATEGORY_IDS)
do
echo $CATEGORY_ID
PRODUCT_IDS=$(curl -s https://api.scalablepress.com/v2/categories/$CATEGORY_ID | jq -rc '.products[].id')
#for PRODUCT_ID in $(echo $PRODUCT_IDS)
#do
#echo $PRODUCT_ID
#done
done
done

This is a publicly available API so you should be able to copy this script and produce the same results. All of the guides I've seen have said to put double quotes around the field you're trying to parse if it contains hyphens, but I'm having no luck trying that.

Answer 1

You can select the key categoryId for each object in the array by applying the selector: curl -s https://api.scalablepress.com/v2/categories | jq 'map(.categoryId)'

This will give you a JSON array with only the values you're interested in. Then you can use the antislurp filter .[] to turn the array into individual results. jq can then output raw strings with the -r switch.

Combining everything, you can achieve what you're looking for with a one-liner:

curl -s https://api.scalablepress.com/v2/categories | jq -r 'map(.categoryId) | .[]'

Even better, you can antislurp first, and then select the key you're looking for: curl -s https://api.scalablepress.com/v2/categories | jq -r '.[] | .categoryId'

Answer 2

you can loop over categories ids right away, without doing all the "echos" that break the json. the two loops can be rewritten as:

#!/bin/bash

CATURL="https://api.scalablepress.com/v2/categories"

curl -s "$CATURL" | jq -rc '.[] | .categoryId' | while read catid; do
        echo "$catid"
        curl -s "$CATURL/$catid" | jq -rc '.products[].id'
done

this will print category id followed by all products ids which from you code seems like your end result:

$ ./pullcat.sh
aprons
port-authority-port-authority-â-medium-length-apron-with-pouch-pockets
port-authority-port-authority-â-full-length-apron-with-pockets
port-authority-easy-care-reversible-waist-apron-with-stain-release
port-authority-easy-care-waist-apron-with-stain-release
backpacks
port-authority-â-wheeled-backpack
nike-performance-backpack
port-authority-â-value-backpack
port-authority-â-basic-backpack
port-authority-â-cyber-backpack
port-authority-â-commuter-backpack
port-authority-â-contrast-honeycomb-backpack
port-authority-â-camo-xtreme-backpack
port-authority-â-xtreme-backpack
port-authority-â-xcapeâ-computer-backpack
port-authority-â-nailhead-backpack
nike-elite-backpack
port-authority-â-urban-backpack
eddie-bauer-eddie-bauer-â-ripstop-backpack
the-north-face-aurora-ii-backpack
the-north-face-fall-line-backpack
the-north-face-groundwork-backpack
the-north-face-connector-backpack
beanies
rabbit-skins-infant-baby-rib-cap
yupoong-adult-cuffed-knit-cap
ultra-club-adult-knit-beanie-with-cuff
ultra-club-adult-knit-beanie
ultra-club-adult-two-tone-knit-beanie
ultra-club-adult-knit-beanie-with-lid
ultra-club-adult-waffle-beanie
ultra-club-adult-knit-pom-pom-beanie-with-cuff
bayside-beanie
...

if you want just the categories ids, you can of course "drop" while loop:

#!/bin/bash

CATURL="https://api.scalablepress.com/v2/categories"

curl -s "$CATURL" | jq -rc '.[] | .categoryId'

---

$ ./pullcat.sh
aprons
backpacks
beanies
bracelet
coaster
cutting-board
dress-shirts
duffel-bags
earring
full-brim-hats
generic-dropoff
hats
hoodies
infant-shirts
ladies-dress-shirts
ladies-dresses
ladies-long-sleeve
ladies-pants
ladies-performance-shirts
ladies-polos
ladies-short-sleeve
ladies-tank-tops
large-bags
...