CODEWITHSUNDEEP
jsoncurlchunked
VadaVad
VadaVad

Reputation: 33

how to handle chunked response with curl while expecting json

I am making GET request and receiving json or "binary" data. Based on my testing the chance for json response is not far from a coin flip. When I receive binary data I also get either "Content-Length: xxx" or "Transfer-Encoding: chunked" response header (this is also more or less 50-50 chance). I have not noticed getting both headers at the same time (if I add -i option for curl in below snippet). However I do get "Content-Length" header for json response. Usually the json response is 280kB in size while chunked response is about 40kB in size.

curl -o output.json\
  -H "Content-Type: application/json;charset=UTF-8"\
  "www.example.com/data"\
  && ./process-output.sh

I would like to find a solution where I can be sure that the whole response is in "output.json" before I execute the next script.

--EDIT--

My current solution is to check the output with file output.json | grep -c "UTF-8 Unicode" and retry (max 5 times). Most people would say that this is an ugly workaround and some might even stop talking to me. I hope that I should not need to use a "solution" like this.

Upvotes: 2

Views: 12265

Answers (1)

DanUber
DanUber

Reputation: 141

does curl -N (no-buffer) fix your issue?

I saw this previously with piping curl to a json formatter - where curl catches SIGPIPE and returns success (so your shell moves on to the process step) before the last chunked response part is sent to STDOUT.

Upvotes: 11

Related Questions