CODEWITHSUNDEEP
pythonbeautifulsoup
stke
stke

Reputation: 25

Can't retrieve link from webpage

I am using bs4 to run through a bunch of websites and grab a specific link off each page but I am having an issue grabbing that link.

I have tried getting all the links using.

 soup = BeautifulSoup(browser.page_source,"lxml")
 print(soup.find_all('a'))

I have tried many other ways including telling it the exact address of one site.

but every time seems to return everything but the link I want.

For context my code goes to pages of this site
https://ce.naco.org/?find=true

These are two of many pages that I am searching for the link in
https://ce.naco.org/?county_info=06019 https://ce.naco.org/?county_info=08045

Under "COUNTY CONTACT" there is a link in most of these pages and that is the link I want to grab but I just can't find a way to make it return only that link it just seems to be invisible to bs4.

I think it has something to do with how the page loads data based on what the user clicks and since bs4 isn't interacting with the site it doesn't load the data??? but this is just a guess.

Upvotes: 2

Views: 80

Answers (1)

baduker
baduker

Reputation: 20022

Instead of scraping the page, just use this endpoint to grab the data:

https://ce.naco.org/get/county?fips=06019

Here's how:

import requests

data = requests.get("https://ce.naco.org/get/county?fips=06019").json()
print(f'{data["county"]["Full_Address"]}\n{data["county"]["County_Website"]}')

Output:

2281 Tulare St<br>Hall Of Records<br>Fresno, CA 93721-2105
http://www.co.fresno.ca.us

This works for both county codes:

import requests

county_codes = ["06019", "08045"]

with requests.Session() as s:
    for county_code in county_codes:
        data = requests.get(f"https://ce.naco.org/get/county?fips={county_code}").json()
        print(f'{data["county"]["Full_Address"]}\n{data["county"]["County_Website"]}')

Output:

2281 Tulare St<br>Hall Of Records<br>Fresno, CA 93721-2105
http://www.co.fresno.ca.us
108 8Th St<br>Glenwood Springs, CO 81601-3355
http://www.garfield-county.com/

Upvotes: 1

Related Questions