sftp protocol with RCurl - last step, how to change R path to libcurl from usr/lib to usr/local/lib

Question

First off - I know this isn't a specific code problem, so feel free to vote to close the question, but I've spent the better part of half of my day today struggling with this problem and could use some help. I also think this thread can help anybody trying to get sftp protocol working in R, since I'll share what I've done today.

I have been attempting to update RCurl so it supports sftp protocol. In R, my protocols look as such (with version and host as well):

> library(RCurl)
> curlVersion()$protocols
 [1] "dict"   "file"   "ftp"    "ftps"   "gopher" "http"   "https"  "imap"   "imaps"  "ldap"   "ldaps"  "pop3"   "pop3s"  "rtsp"   "smb"    "smbs"  
[17] "smtp"   "smtps"  "telnet" "tftp" 

> curlVersion()$version
[1] "7.43.0"

> curlVersion()$host
[1] "x86_64-apple-darwin15.0"

Not great - no sftp option...

I followed this thread - http://andrewberls.com/blog/post/adding-sftp-support-to-curl - to update curl on my machine, and was partially successful in doing so. The success part is reflected in when I run the following in the command line:

curl -V 
curl 7.55.1 (x86_64-apple-darwin15.6.0) libcurl/7.55.1 zlib/1.2.5 
libssh2/1.8.0
Release-Date: 2017-08-14
Protocols: dict file ftp gopher http imap ldap ldaps pop3 rtsp scp sftp smtp telnet tftp 
Features: AsynchDNS IPv6 Largefile libz UnixSockets 

This is great, because I've got curl updated. However, when I load RCurl and run curlVersion() in R, it's not updated. I've examined the lib folders of my /usr/ directory and have the following problem. In both my /usr/lib AND /usr/local/lib, there exists libcurl files. Specifically:

• libcurl.3.dylib, libcurl.4.dylib, and libcurl.dylib in /usr/lib
• libcurl.4.dylib, libcurl.dylib, libcurl.a and libcurl.la in /usr/local/lib

The files in my /usr/local/lib directory are the new files that I'd like used, however here is where my major headache began. I copied the four files from /usr/local/lib into /usr/lib and I BROKE MY COMPUTER. Many of my applications stopped loading, and I had to reinstall my OS X on account of being afraid of messing anything else up. When I reinstalled my OS X, it put my files back into their respective folders (how they look in the bullets above).

If I had to guess, I probably shouldn't have moved the .a or .la files... I'm not sure.

One last thing - when i run $ $PATH in console, I get:

$PATH
-bash: /Users/Home/.rvm/gems/ruby-2.3.3/bin:/Users/Home/.rvm/gems/ruby-2.3.3@global/bin:/Users/Home/.rvm/rubies/ruby-2.3.3/bin:/Users/Home/anaconda2/bin:/usr/local/git/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/TeX/texbin:/Users/Home/.rvm/bin: No such file or directory

/usr/local/bin is before /usr/bin. but I don't see anything with /usr/local/lib or /usr/lib - could I add these (/usr/local/lib) to my path to fix this? .libPaths() in R gives me this:

.libPaths()
"/Library/Frameworks/R.framework/Versions/3.4/Resources/library"

...if it helps at all

EDIT - I should note that the root of all problems, ofcourse, is that since 'sftp' is not listed in curlVersion()$protocols in R, that when i run RCurl::getURL(), or RCurl::ftpUpload() (im hoping to use ftpUpload), for both I receive the error:

> RCurl::ftpUpload(what = 'myfile.txt', to = 'sftp://userid:userpw@ip/'myfile.txt')
Error in function (type, msg, asError = TRUE)  : 
  Protocol "sftp" not supported or disabled in libcurl

Thanks!

Answer 1

In practice i've found it more dangerous to try updating an existing curl and openssl installation, than installing an entirely new version and updating PATH envars where they'll be used

In my case my newer curl and openssl with sftp support where installed into

/usr/local/opt/curl-openssl/bin
/usr/local/opt/openssl/lib/

Consequently I needed to set the following envars

PATH="/usr/local/opt/curl-openssl/bin:${PATH}"
LIBRARY_PATH="/usr/local/opt/openssl/lib/:${LIBRARY_PATH}"

before executing

install.packages("RCurl", type="source")

Answer 2

Phew! Finally got this working in an Ubuntu 14.04 docker container with rstudio-server. I hit every issue! But I had to spend more hours than I care to count finding all this stuff, so trying to put it all in one place for anyone else having this problem Here's what I did...

In the linux terminal:

apt-get install libcurl4-openssl-dev

wget libssh2 and then compile with ordinary

./configure

make

make install

wget curl and then compile

./configure --with-sshlib2=/usr/local

make

make install

If you need curl to work in the terminal with sftp, it still may not at this point. Verify its using the new libcurl.so.4 that you just installed. The curl you just installed will install into /usr/local/bin, so use ldd to verify its not using an old library. Mine was using an old libcurl so I linked the old libcurl to the new libcurl in /usr/local/lib and then recompiled curl.

If you want RCurl to work, copy all of the new libs from /usr/local/lib (libcurl & libssh2) to /lib, then move to the R console.

In Rconsole:

install.packages("RCurl", type = "source")

verify sftp protocol is enabled

libcurlVersion() RCurl::curlVersion()

These should show you enabled protocols for libcurl and RCurl respectively. Cross your fingers for sftp!

Just in case, I also did these things that probably didn't do much but I did when I was hitting a wall and just kept in place.

upgraded R and R studio-server

removed the R version of curl and RCurl prior to reinstalling RCurl. Also in newer rstudio am not sure the type="source" is needed.

Also as Canovice noted, be careful moving around libcurl and libssh libraries. I broke networking a couple of times, fortunately since it was a docker container I just restarted. I found that moving everything at once worked best,

cp /usr/local/lib/libssh* /lib && cp /usr/local/lib/libcurl* /lib

Should also note I'm a total R novice so there may be cleaner ways to do this than moving libraries all over the place. I tried utilizing .libaths() in rstudio, but gave up in the end.

Answer 3

Installing RCurl from source did the trick for me !

install.packages("RCurl", type="source")

I found the solution here - http://www.omegahat.net/RCurl/FAQ.html - specifically, the following lines.

"I can't use scp or sftp within RCurl but the documentation for curl seems to suggest that it can. So why does RCurl not support it?

RCurl does support it, but the likely problem is that the version of libcurl you have installed does not support it. You can check what protocols your libcurl and hence RCurl supports via the command curlVersion. If scp and sftp are not there, reinstall libcurl but with support using libssh2. You will need to have the libssh2 development libraries and headers installed before installing libcurl. On some OSes, you will need to rebuild RCurl from source."

In general, this is a great summary of my issue from start to finish. If your version of RCurl doesn't support sftp, you need to install (at the command line) libssh2 first, then reinstall libcurl, then rebuild from source for some reason.