CODEWITHSUNDEEP
pythonpython-requestsget
Datenkralle
Datenkralle

Reputation: 45

HTTP GET In Python

I have a bash script that downloads a lot of files from a URL and saves them to my Computer like:

curl 'http://website.com/shop.php?store_id=[66-900]&page=[7-444]&PHPSESSID=phpsessid' -O

(and a bunch of headers set, the typical request you get when you copy a request as curl in firefox)

Tried to rewrite the script in python but turns out it is very hard (complete newbie) like:

import requests

cookies = {
    'PHPSESSID': '<phpsessid here>',
}

headers = {
    '<bunch of header stuff here>'
}

params = (
    ('store_id', '[66-900]'),
    ('page', '[7-444]'),
    ('PHPSESSID', '<phpsessid here>'),
)

response = requests.get('http://website.com/shop.php', headers=headers, params=params, cookies=cookies)

But it just doesn't do anything (curl command downloading everything as commanded) I realised I may need to write my downloaded stuff to a file but am completly lost since I am not handling one file but hundreds of files and as seen from the curl command above.

Completly lost what I am missing or need to do to replicate a curl request like that in python, ANY help appreciated!

Upvotes: 0

Views: 344

Answers (2)

furas
furas

Reputation: 143197

You have to use for-loops instead of [66-900], [7-444]

for store_id in range(66, 990+1):
    for page in range(7, 444+1):

        params = {
           'store_id': store_id,
           'page': page,
           'PHPSESSID': '<phpsessid here>',
        }

        response = requests.get(...)

        with open(f"{store_id}-{page}", "wb") as fh:
             fh.write(response.content)

Upvotes: 1

balrundev
balrundev

Reputation: 381

You can't use a range as you can do it when using curl.

As @furas noted, you can use a loop, because curl sends requests with all parameter combinations from the ranges.

Function response.get returns an object of class requests.models.Response, so you should use json() method or properties content (returns bytes) and text (returns string).

Upvotes: 1

Related Questions