CODEWITHSUNDEEP
pythonloopspython-requests
try_to_code
try_to_code

Reputation: 25

How to iterate list of urls using request.get?

I have such code:

url = "https://www.reformagkh.ru/opendata/export/"
regions = ["150", "101"]
csv_files = []
for region in regions:
    result = requests.get(url, params={"t":region})
    zf = ZipFile(BytesIO(result.content))
    for filename in zf.namelist():
        if filename.endswith(".csv"):
            file = zf.open(filename)
            csv_files.append(file)
            if len(csv_files) == 1:
                reader = csv.reader(TextIOWrapper(file, 'utf-8'))
                for row in reader:
                    print(row)
            else:
                print("Error")

I have 2 links, where located some unzip csv files and I should open them and read. The main question is how work with list of urls and open them step by step? When I am trying to debug and fix it, I have 400 error and problem with loop. Could somebody give me advise how to handle it?

I should open and handle such links: ['https://www.reformagkh.ru/opendata/export/150', 'https://www.reformagkh.ru/opendata/export/101']

Upvotes: 1

Views: 1584

Answers (2)

Krishna Chaurasia
Krishna Chaurasia

Reputation: 9552

You need to prepare the url in the loop instead of passing region as params.

Use f-strings to prepare the url as for Python 3.6+:

for region in regions:
    url_cur = f"{url}{region}"
    result = requests.get(url_cur)

Use format() if you are using python version less than 3.6:

for region in regions:
    url_cur = "{}{}".format(url, region)
    result = requests.get(url_cur)

You also need to create the csv_files list newly for each url.

The complete code would be:

url = "https://www.reformagkh.ru/opendata/export/"
regions = ["150", "101"]
for region in regions:
    cur_url = f"{url}{region}"
    result = requests.get(cur_url)
    zf = ZipFile(BytesIO(result.content))
    csv_files = []  # create a new list everytime
    for filename in zf.namelist():
        if filename.endswith(".csv"):
            file = zf.open(filename)
            csv_files.append(file)
            if len(csv_files) == 1:
                reader = csv.reader(TextIOWrapper(file, 'utf-8'))
                for row in reader:
                    print(row)
            else:
                print("Error")

Upvotes: 1

Amirhossein
Amirhossein

Reputation: 92

regions = ["150", "101"]
csv_files = []
for region in regions:
    url = "https://www.reformagkh.ru/opendata/export/%s" % region
    result = requests.get(url)
    zf = ZipFile(BytesIO(result.content))
    for filename in zf.namelist():
        if filename.endswith(".csv"):
            file = zf.open(filename)
            csv_files.append(file)
            if len(csv_files) == 1:
                reader = csv.reader(TextIOWrapper(file, 'utf-8'))
                for row in reader:
                    print(row)
            else:
                print("Error")

I think it is much easier with %s. I often use the same method.

Upvotes: 0

Related Questions