CODEWITHSUNDEEP
javascriptnode.jsjsonfetch-api
Max888
Max888

Reputation: 3780

node fetching json from url

I am trying to fetch and parse the JSON returned by the below, in Nodejs. It seems the problem is that the node code is trying to parse the JSON string which contains newline characters, but I am not sure how to avoid this, and why fetch works in chrome devtools. I'm guessing the problem is my simplistic knowledge of https requests, so I would really appreciate it if somebody can explain what is going wrong.

const url = "https://beta.charitycommission.gov.uk/umbraco/api/charityApi/getSearchResults?searchText=&pageNumber=1&contextId=1126&onlyShow=&&&&&&&"

I can successfully run this in the chrome devtools open on the domain. Also, the JSON appears to automatically get parsed to an object, which I didn't expect.

fetch(url).then(res => res.json()).then(json => console.log(json.pageItems))

I have tried the following in node, but none work.

const fetch = require("node-fetch");
await fetch(url).then(res => res.json()); // Unexpected token  in JSON at position 0

const rp = require('request-promise-native');
const json = await rp({uri: url}) // returns JSON string with newline characters
JSON.parse(json) // Unexpected token  in JSON at position 0

Upvotes: 3

Views: 9671

Answers (2)

FZs
FZs

Reputation: 18619

The problem is that your JSON file is saved in UTF-8 BOM format.

What does it mean?

Your file begins with a so-called Byte Order Mark character, U+FEFF (Zero Width No-break Space).

That's invisible (has zero width), but still there, and the JSON interpreter fails to parse it.

How to solve?

  1. Save the JSON data without BOM, or

  2. Use res.text() and then JSON.parse(text.slice(1)) to remove the leading character:

    fetch(url)
.then(res => res.text())
.then(text => JSON.parse(text.slice(1)))
.then(json => console.log(json.pageItems))

Why does it work in Chrome?

Chrome automatically removes the BOM character from any fetched/downloaded file, to avoid similar issues.

Upvotes: 3

Brent Washburne
Brent Washburne

Reputation: 13158

Try using the default fetch:

const json = await fetch(url).then(res => res.json())

res.json() parses the JSON data and returns an object. Use await to return the data into the variable json.

Upvotes: 1

Related Questions