How to follow a URL in R

Question

Sorry for the bad title but I don't know how else to phrase "follow".

I'm looking to remotely download a csv file from a website. I could do this by clicking the download button using RSelenium, but I've found that there's a direct link that will initiate the download for me. I.e. I could go to https://www.fake-website-url.com and click the download button, or I could just enter https://www.fake-website-url.com/exportcsv into my browser and it would automatically download.

I try not to use RSelenium whenever I can help it since it's clunky, but I'm not sure how to just initiate the download. Nothing from rvest stands out since I'm not actually reading html.

Basically, I'm looking for an R function like gotoURL('https://www.website.com/exportfullcsv) that will download the file just like it would if I entered the URL into my browser.

Answer 1

Since you said that you were able to find a direct URL, then the issue is not that the download failed, it's that you aren't accessing the content correctly.

I uploaded a small zip file to a personal website and ran this code:

dl <- httr::GET("https://.../sessions_tracker.zip")
dl
# Response [https://.../sessions_tracker.zip]
#   Date: 2020-04-08 20:59
#   Status: 200
#   Content-Type: application/zip
#   Size: 19.2 kB
# <BINARY BODY>

length(httr::content(dl))
# [1] 19184
19184 / 1000
# [1] 19.184     ### confirmation of download, this rounds to 19.2kB

head(httr::content(dl), n=80)
#  [1] 50 4b 03 04 14 00 00 00 08 00 60 7e 7b 50 1e c3 ed e8 32 4a 00 00 fa b7 01 00 14 00 1c 00
# [31] 73 65 73 73 69 6f 6e 73 5f 74 72 61 63 6b 65 72 2e 63 73 76 55 54 09 00 03 53 83 7e 5e 5e
# [61] 01 85 5e 75 78 0b 00 01 04 d3 c6 2d 00 04 64 00 00 00 b4 5d

writeBin(httr::content(dl), "sessions_tracker.zip")