CODEWITHSUNDEEP
pythoncsvweb-scrapingbeautifulsouppython-requests
Thomas R
Thomas R

Reputation: 1217

perform download via download button when request_url in browser-inspect does not work

I have almost the question already posted and answered here:

Perform Download via download button in Python

I also followed the instructions, in the answer of the above link.

In my case I want to download the data from the following page:

https://www.smard.de/home/downloadcenter/download-marktdaten#!?downloadAttributes=%7B%22selectedCategory%22:1,%22selectedSubCategory%22:1,%22selectedRegion%22:%22DE%22,%22from%22:1658872800000,%22to%22:1659563999999,%22selectedFileType%22:%22CSV%22%7D

The download-button is the one called "Datei herunterladen" in the lower right corner.

download button in the lower right corner

I went into the inspect mode downloaded the file and got the following output in the inspection

inspection results

But the resulting

Request URL: 'https://www.smard.de/nip-download-manager/nip/download/market-data'

does not help getting the csv-file. Opened in the browser I get: 'The requested URL was rejected.' On the other hand, it does not even contain the parameters anymore, so it can't be the right download url.

May anyone help to automate this download?

edit Now I also tried

url = 'https://www.smard.de/nip-download-manager/nip/download/market-data'
json_body = {'format': "CSV",
'language': "de",
'moduleIds': [1001224, 1004066, 1004067, 1004068, 1001223, 1004069, 1004071, 1004070, 1001226, 1001228, 1001227,1001225],
'region': "DE",
'timestamp_from': 1659304800000,
'timestamp_to': 1659391199999,
'type': "discrete"}

x = requests.post(url, json = json_body)
x.content

> b'Exception when parsing incoming request to JSON object.'

So how do I get the csv-file based on this method?

Upvotes: 1

Views: 676

Answers (2)

Thomas R
Thomas R

Reputation: 1217

  1. The json_body of the request-payload was not correct.
  2. And I found an easy way to get the correct one with less effort and less probability to make it wrong.
  • In the inspect mode after the file is downloaded:
  • Click on the "Raw"-Button in the upper right corner of the "request"-tab, than you can get the correct payload per copy and paste.

enter image description here

The final solution is:

url = 'https://www.smard.de/nip-download-manager/nip/download/market-data'
payload = {"request_form":[{"format":"CSV","moduleIds":[1001224,1004066,1004067,1004068,1001223,1004069,1004071,1004070,1001226,1001228,1001227,1001225],"region":"DE","timestamp_from":1658872800000,"timestamp_to":1659563999999,"type":"discrete","language":"de"}]}
x = requests.post(url, json = payload)
with open(f'energy_ger.csv', 'wb') as f:
    f.write(x.content)

Upvotes: 0

neiii
neiii

Reputation: 160

the reason that you're getting "URL Rejected" is because the link specified in the request is not file-specific. The link that the POST request is being made to, contains several files, which are given out based on the body that the request is sent with. If you look into the developer tools, you can see that there is a request payload attached to the request (image attached) request payload. If you want to learn more about sending JSON body data with requests, please take a look at w3 and this thread. I hope that helps!

Upvotes: 1

Related Questions