R: postForm from the RCurl package and issues API call

Question

Does anyone have experience with the limitations of postForm from the RCurl package?

I am pulling data off an server and almost out of nowhere I got the error message * HTTP 1.0, assume close after body and then a 500 Internal Server Error. I tested the configurations and everything seemed to be fine. I've created a clean database and re-uploaded my database 20/30 cases at the time while repeatedly pulling the data with an API/postForm call from R. It all works as it should until I get to around 150 cases then the error message appears. Regardless of the order I upload the cases in the error appears around 150/160 cases and a total files size around 11 to 12 MB. In other words, the error doesn't seem to be dependent on specific cases as it is not the same cases that breaks it

Any advice would be appreciated.

I've attached a screenshot to spice up this rather boring post a bit and to make up for not having a working example,

Update 2013-08-24 19:33:18Z

Here my curlVersion()$version and sessionInfo() information,

> curlVersion()$version
[1] "7.22.0"
> sessionInfo()
R version 3.0.1 (2013-05-16)
Platform: i686-pc-linux-gnu (32-bit)

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=C                 LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] RCurl_1.95-4.1 bitops_1.0-6

Update 2013-08-26 05:39:26Z

As suggested in hadley's comment I've added the verbose RCurl output from the call that works and from the call that fails, see below

The call that works with less then 150 cases in the database

> R.object.API <- postForm(R.object.URL, token=R.object.token, content="record", type="flat", format="csv", rawOrLabel="Label", .opts=curlOptions(ssl.verifypeer=TRUE, cainfo=R.object.crt, verbose=TRUE))
* About to connect() to research.org port 443 (#0)
*   Trying xx.xx.xxx.xxx... * connected
* successfully set certificate verify locations:
*   CAfile: /home/dir/research.cert
  CApath: /etc/ssl/certs
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*      subject: C=XX; postalCode=XXXXX-XXXX; ST=XX; L=XXXXXX; street=XXX; street=XX XXXXXX XX; O=XXXX, XXX; OU=XXX; CN=research.org
*      start date: 2013-02-04 00:00:00 GMT
*      expire date: 2016-02-04 23:59:59 GMT
*      subjectAltName: research.org matched
*      issuer: C=US; O=XXXXXX; OU=XXXXXX; CN=XXXXXX Server XX
*      SSL certificate verify ok.
> POST /api/ HTTP/1.1
Host: research.org
Accept: */*
Content-Length: 573
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------XXXXXXXXXXXX

< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Date: Mon, 26 Aug 2013 05:16:44 GMT
< Server: Apache/2.2.15 (Red Hat)
< X-Powered-By: PHP/5.3.3
< Expires: 0
< cache-control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=utf-8
< 
* Closing connection #0
> 

The call fails with more then 150 cases in the database

> R.object.API <- postForm(R.object.URL, token=R.object.token, content="record", type="flat", format="csv", rawOrLabel="Label", .opts=curlOptions(ssl.verifypeer=TRUE, cainfo=R.object.crt, verbose=TRUE))
* About to connect() to research.org port 443 (#0)
*   Trying xx.xx.xxx.xxx... * connected
* successfully set certificate verify locations:
*   CAfile: /home/dir/research.cert
  CApath: /etc/ssl/certs
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
*      subject: C=XX; postalCode=XXXXX-XXXX; ST=XX; L=XXXXXX; street=XXX; street=XX XXXXXX XX; O=XXXX, XXX; OU=XXX; CN=research.org
*      start date: 2013-02-04 00:00:00 GMT
*      expire date: 2016-02-04 23:59:59 GMT
*      subjectAltName: research.org matched
*      issuer: C=US; O=XXXXXX; OU=XXXXXX; CN=XXXXXX Server XX
*      SSL certificate verify ok.
> POST /api/ HTTP/1.1
Host: research.org
Accept: */*
Content-Length: 573
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----------------------------XXXXXXXXXXXX

< HTTP/1.1 100 Continue
* HTTP 1.0, assume close after body
< HTTP/1.0 500 Internal Server Error
< Date: Mon, 26 Aug 2013 05:15:05 GMT
< Server: Apache/2.2.15 (Red Hat)
< X-Powered-By: PHP/5.3.3
< Expires: 0
< cache-control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Content-Length: 276
< Connection: close
< Content-Type: text/html; charset=UTF-8
< 
* Closing connection #0
Error: Internal Server Error

Answer 1

Putting this in an answer as it's easier to format. My suggestion is to save the following in a file, fill in the form address (i.e change WHEREVER_YOU_ARE_TRYING_TO_POST_DATA to the appropriate address), name it something like test.html, and open it in your browser.

I've filled in most of the values based upon your above examples, but I have no idea what should go in the token field - this depends upon your specific problem.

<!DOCTYPE html>
<html>
<head></head>
<body>   
<div class="content">
    <form action="WHEREVER_YOU_ARE_TRYING_TO_POST_DATA" method="post">
        token: <input name="token" type="text" size="100" /><br />
        content: <input name="content" type="text" size="100" value="record" /><br />
        type: <input name="type" type="text" size="100" value="flat"/><br />
        format: <input name="format" type="text" size="100" value="csv"/><br />
        rawOrLabel: <input name="rawOrLabel" type="text" size="100" value="Label"/><br />
        <input name="Submit" type="submit" value="submit" />
    </form>
</div>
</body>
</html>

If submitting this form works, but your postForm code doesn't, then something weird is going on with your R code. If they both fail then you have a much deeper problem that probably has nothing to do with R.

Answer 2

Doesnt answer your question but relates to options and keepalive:

RCurl uses the libcurl library. This is different to the CURL command line tool. You need to look at the libcurl options here. CURLOPT_TCP_KEEPALIVE maybe what you want. In RCurl this would be listed as tcp.keepalive if it were present in listCurlOptions().

From the easyopt man page this was added in 7.25.0. You can check what version of libcurl RCurl is using by running

> curlVersion()$version
[1] "7.22.0"

Unfortunately the version of libcurl RCurl is using doesnt handle the keep alives yet.