CODEWITHSUNDEEP
pythonweb-scraping
John
John

Reputation: 468

How to get dynamic HTML code from webpage without opening browser?

This site has an "export as csv" button next to the "Power and Energy" text. This is the link of that button which can be found on browser's developer tools (Note that the link changes every 15 minutes)

My question is, is there any way to get that link using python without opening the browser? (I.e. without using selenium because it would take time if I did this for multiple sites.)

My goal is to automate the process of downloading that CSV file, every 1 hour.

Upvotes: 3

Views: 869

Answers (2)

Robert Co
Robert Co

Reputation: 1715

That url works in curl. Therefore the data doesn't depend on javascript and you do not need selenuim. The st and et are unix time:

datetime.datetime.utcfromtimestamp(1567296000000/1000)

The timeUnit=4 is daily, so making that request hourly is wasting the site's resource and yours. If you really need hourly, change time timeUnit=3 and pn0=Power. Energy=Power x Hr

Lastly, I assume that since you are planning to run hourly, you are planning to save the data somewhere. If so, consider adjusting your st and et to only that range you need to further reduce your request size.

Upvotes: 6

psidex
psidex

Reputation: 465

If you open your browsers debugging tools and go to the "Network" tab, you can see web requests as they are being made.

For example, the request for download from the site you linked is:

https://monitoringpublic.solaredge.com/solaredge-web/p/charts/274560/chartExport?st=1567296000000&et=1569887999999&fid=274560&timeUnit=4&pn0=Energy&id0=0&t0=0&hasMeters=false

The various URL parameters mean that that link will probably always point to that specific file (rather than one that is updated over time).

You can use this method to get the links for the other sites as well, but like this one, the URLs will probably point to a link for that specific time.

Upvotes: 5

Related Questions