CODEWITHSUNDEEP
pythonweb-scrapingdropdown
ClaraJ
ClaraJ

Reputation: 67

Python - Scraping and looping through a dropdown list

I am trying to scrape the following website https://predictioncenter.org/casp14/results.cgi?view=tables&target=T1024&model=1&groups_id= and in particular I would like to loop through all the values of "Target" in the upper left, so I can then scrape the table associated with target value.

I am able to scrape the table once I am on a page of a specific target, but I am having trouble iterating and looping through all the target values.

So far, I have retrieved all the possible target values with this code:

options = soup.find("select",{"name":"target"}).findAll("option")
list_prot = []
for i in options:
    name = i.text
    list_prot.append(name)

Could anyone help me to use that in order to click and open the page corresponding to each of the target value?

Thanks a lot!

Upvotes: 1

Views: 101

Answers (1)

anarchy
anarchy

Reputation: 5204

Once you have the list of targets you can just do this to iterate over the websites. So let’s assume your list variable is list_prot.

for target in list_prot:
    link = f"https://predictioncenter.org/casp14/results.cgi?view=tables&target={link}&model=1&groups_id="
    requests.get(link)…………
    ………
    ………

Now for each loop, link will be the new website based on your targets. You can access them using requests and scrape the table using beautiful soup.

I used an f string that replaces the link variable in the curly braces with your target variable.

Upvotes: 1

Related Questions