CODEWITHSUNDEEP
javacsv
Shatr
Shatr

Reputation: 31

How to pull a CSV file from a website using a Java application?

So I have a problem. I want to design a program which pulls a CSV file from a certain website automatically. The thing is, I have no idea how to do so. It seems that the button which is used to download the CSV file doesn't actually point that file, but rather points to a function that does its thing and returns a CSV file. How can I pull this CSV file from the website? Apparently this has something to do with a servlet but I'm not quite sure. Thanks

Edit: It seems that there isn't really a link like bla.com/.../myfile.csv but rather bla.com/.../getReport.do and regardless of what I input into the text boxes which are provided on the website, the link stays as bla.com/.../getReport.do. The problem I have is that I can't just use the standard IO libraries in Java to download the csv file because I have no idea how to actually get the file without manually downloading it.

Upvotes: 0

Views: 412

Answers (1)

SQL Hacks
SQL Hacks

Reputation: 1332

  1. Use Chrome.
  2. Press F12 to get the developer window - look at the network tab.
  3. Download your csv and watch for the file that you want in the panel.
  4. Right-click on your CSV file in the network tab you should be able to copy as cURL enter image description here
  5. you can paste that command into a CMD prompt (on windows) or a bash terminal (in Linux or Mac).
  6. On Linux set up a cron job to fire off that command regularly.

To send the output to a file... add >file.txt to the end of the command.

curl 'http://app.toronto.ca/tmmis/getAdminReport.do' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/;q=0.8' -H 'Referer: http://app.toronto.ca/tmmis/getAdminReport.do' -H 'Origin: http://app.toronto.ca' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded' --data 'function=getMemberVoteReport&download=csv&page=0&itemsPerPage=50&sortBy=&sortOrder=&exportPublishReportId=2&termId=6&memberId=2&decisionBodyId=0&fromDate=2014-12-31&toDate=2016-05-10' --compressed > file1.txt

Upvotes: 1

Related Questions