CODEWITHSUNDEEP
pythonpandas
Marcelo Soares
Marcelo Soares

Reputation: 187

How to loop over queries and page results?

I'm trying to scrape the results of three different queries from a fixed URL where I iterate on the result number (25 per page). For each query, I want the iteration to stop as it reaches a certain number.

So far, I got the code to stop at the desired result number, but it resumes at the same query instead of jumping to the next. I tried "break", "pass" and "continue". What am I doing wrong?

This is my code so far:


for query in queries:
  while n<580:
    url = urlbase+query+period+str(n)
    print(url)
    r = requests.get(url)
    html = parser.fromstring(r.text)
    print(html)
    if n > 575:
      n=1
      continue
    else: 
      n=n+25

Upvotes: 0

Views: 139

Answers (1)

Felix Ekl&#246;f
Felix Ekl&#246;f

Reputation: 3720

for query in queries:
    for n in range(0, 580, 25): # iterate from 0 to 580 and step 25 each iteration.
        url = urlbase+query+period+str(n)
        print(url)
        r = requests.get(url)
        html = parser.fromstring(r.text)
        print(html)

Upvotes: 1

Related Questions