CODEWITHSUNDEEP
jsoncurl
Aeradriel
Aeradriel

Reputation: 1232

Curl returning different JSON than web browser

I am having an issue with my curl request. I am getting (through a GET request) a JSON from a web server and it works really great in my web browser. However when I get it from curl (or my ruby code), some value of my JSON are empty (they were not in my browser). That makes me lose a lot of data which is not acceptable. Does anyone have an idea of what makes this happen ? I already tried to change the user agent (see below) and still it gives the same result.

curl "http://..../path/to/json"

or

curl -H "User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.89 Safari/537.36" "http://..../path/to/json"

gives the same result.

EDIT : I tried copying curl request from Safari (see below) and it gives the same missing values. Chrome however display the same result as curl.

curl 'https://path/to/json' -XGET -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Cache-Control: max-age=0' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.4'

Upvotes: 2

Views: 1606

Answers (2)

Aeradriel
Aeradriel

Reputation: 1232

It seems that the problem came from the fact that the response is customize depending on the language of the browser. So in French the json is full and in English some fields are missing...

Upvotes: 0

hanshenrik
hanshenrik

Reputation: 21665

i guess its possible that your browser ignore the content-length header, and just keeps reading until the connection is closed. curl isn't like that by default (probably for performance reasons, because it can be much slower to read until the connection is dropped), and stops reading once content-length bytes has been read. and that your target webserver is bugged, and sends the wrong length. you can tell curl to ignore the header and read everything, with the --ignore-content-length switch.

another possibility is that the webpage you're reading the curl from, fills in the blanks with javascript somehow?

another possibility is that the full json is not displayed from the target server without having a http session cookie first?

Upvotes: 1

Related Questions