CODEWITHSUNDEEP
pythoncsvpython-3.xurllib
sandrosil
sandrosil

Reputation: 553

Downloading CSV file from website/server with Python 3.X

Programming beginner here. So for my very first project I was able to make a quick python script that downloaded files from this website: http://www.wesm.ph/inner.php/downloads/market_prices_&_schedules

I noticed that the link address of the to-be-downloaded file followed a pattern.
(http://wesm.ph/admin/downloads/download.php?download=../csv/mpas/XXXXX/XXXX.csv)

With some string concatenation and using the datetime module, I was able to create the HTML string of the csv file. After which, I just would use the: 

urllib.request.urlopen(HTMLlink).read()

and save it with something like:

with open('output.csv', "w", newline='') as f:
    writer = csv.writer(f)
    writer.writerows(fullList)

It used to work - now it doesn't. I noticed however whenever I clicked the 'Generate Report' button and THEN run the script, the script would generate the output file. I'm not sure why this works. Is there a way to send a request to their server to generate the actual file? Which module, or commands should I use?

Upvotes: 0

Views: 1020

Answers (1)

Danny_ds
Danny_ds

Reputation: 11406

Most likely those files are only temporarily stored on that webserver after you click 'Generate Report'.

In order to grenerate new ones, there might even be a check (in JavaScript or using Cookies, Session-ID), to see if the generation of the new link/file is asked from a human, or a bot.

You might also want to check the HTTP return code (or even the full returned headers to see what exactly the server is answering).

Upvotes: 1

Related Questions