CODEWITHSUNDEEP
bashcurlweb-scraping
ulima2_
ulima2_

Reputation: 1336

curl 1020 error when trying to scrape page using bash script

I'm trying to write a bash script to access a journal overview page on SSRN.

I'm trying to use curl for this, which works for me on other webpages, but it returns error code: 1020 for me if I try to run the following codes:

curl https://papers.ssrn.com/sol3/papers.cfm?abstract_id=1925128

I thought it might have to do with the question mark in the URL, but I got it to work with other pages that contained question marks.

It probably has something to do with what the page's allows to do. However, I can also access the page using R's rvest package, so I think it should work in general also using bash.

Upvotes: 4

Views: 9143

Answers (1)

Raman Sailopal
Raman Sailopal

Reputation: 12877

Looks like the site has blocked access via curl. Change the user agent and it should work fine i.e.

curl --user-agent 'Chrome/79' "https://papers.ssrn.com/sol3/papersstract_id=1925128"

Upvotes: 8

Related Questions