CODEWITHSUNDEEP
curlchunked-encoding
ensonic
ensonic

Reputation: 3440

how to specify the chunk size in a chunked post using curl

I am trying to send audio to a webservice that want the audio as a chunked post. Sending pre-recorded files works fine, but running the below code does not send anything to the server as it seems. Is curl trying to send everything in one chunk? I can't find any option to e.g. ask it to chunk every xxx bytes.

gst-launch-1.0 pulsesrc ! "audio/x-raw,rate=16000,channels=1,format=S16LE" ! fdsink sync=false | \
curl --header "Transfer-Encoding: chunked" --data-binary @- http://...

Upvotes: 7

Views: 7587

Answers (2)

JonathanH-UK
JonathanH-UK

Reputation: 51

In case this helps anyone in the future - I was trying to use curl to reproduce a problem I was seeing with an application based on an old version Apache CXF 3.x to call a SOAP API: With the same headers and payload, I observed the server silently dopping the connection from CXF after the very first POST (no server reply received), whereas the same thing with curl would succeed. The HTTP headers were identical including Transfer-Encoding: chunked and did not have the Content-Length header set. One key difference was that curl was sending the whole POST as a single chunk of about 13K whereas CXF was splitting it into multiple 4K chunks. The receiving server is an old ASP.NET server of some kind, possibly using IIS 6.x, which seems to have chunking disabled by default and/or poor support for it.

I found that there is not an option in curl for controlling the chunk size, but once I started buiding my own binary from source (it's really easy - lots of docs on how to do it), the limits which seemed obvious did not have any effect on chunking.

Setting CURLOPT_BUFFERSIZE from an environment variable only affects the Read buffer size but we're looking at writing data So I tried shinking CURL_MAX_WRITE_SIZE in include/curl/curl.h down to 4096 first and when that made no differece then I tried shrining UPLOADBUFFER_DEFAULT and UPLOADBUFFER_MAX to 4096 in lib/urldata.h But it seems that these buffer sizes have no effect on chunking. Eventually I found that you can set the chunk size by changing MAX_INITIAL_POST_SIZE from (64*1024) to (4*1024) in lib/http.h and then it broke my POST payload into ~4k chunks and reproduced the same issue as I was seeing with CXF's HttpClient.

So in summary: If you need to force curl to break up POST data into chunks smaller than the 64K default, then you'll need to compile your own version with MAX_INITIAL_POST_SIZE set accordingly.

Upvotes: 2

ensonic
ensonic

Reputation: 3440

Actually there is a curlhttpsink in gst-plugins-bad and this works now: gst-launch-1.0 pulsesrc ! "audio/x-raw,rate=16000,channels=1,format=S16LE" ! curlhttpsink location=http://...

This way you don't need to manually specify any chunk-size, since this is handled by the sink.

Upvotes: 1

Related Questions