CODEWITHSUNDEEP
pythonbeautifulsoup
Kamikaze_goldfish
Kamikaze_goldfish

Reputation: 861

Now trying to extract just `href` from class with BeautifulSoup and Python 3.

I can't seem to get this to work. I have my script going to a site and scraping the data into my info variable but when I am trying to pull out the href from a specific class I am getting None or it just isn't working when I try all kinds of different combos. Where am I screwing up? When I scrape it into my info variable, there is a class='business-name' and the href inside of it. 

import requests
from bs4 import BeautifulSoup

count = 0
search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com/search"
q = {'search_terms': search_terms, 'geo_location_terms': location}
page = requests.get(url, params=q)
url_link = page.url
page_num = str(count)
searched_page = url_link + '&page=' + str(count)
page = requests.get(searched_page)
soup = BeautifulSoup(page.text, 'html.parser')
info = soup.findAll('div', {'class': 'info'})
for each_business in info:
    # This is the spot that is broken. I can't make it work! 
    yp_bus_url = each_business.get('class_','business-name')['href']
    print(yp_bus_url)

Upvotes: 5

Views: 4945

Answers (3)

Dodge
Dodge

Reputation: 3309

You can also do this:

import requests
from bs4 import BeautifulSoup

count = 0
search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com/search"
q = {'search_terms': search_terms, 'geo_location_terms': location}
page = requests.get(url, params=q)
url_link = page.url
page_num = str(count)
searched_page = url_link + '&page=' + str(count)
page = requests.get(searched_page)
soup = BeautifulSoup(page.text, 'html.parser')

With the change here (be sure to assign the list to whatever you want):

#info = soup.findAll('div', {'class': 'info'})
info = soup.select("[class~=business-name]")
[i.get('href') for i in info]

Returns:

['/new-orleans-la/mip/upperline-restaurant-526381149?lid=1001797484770',
 '/new-orleans-la/mip/brunos-tavern-451091659?lid=451091659',
 '/new-orleans-la/mip/lafittes-blacksmith-shop-bar-19195002?lid=19195002',
 '/new-orleans-la/mip/johnny-whites-pub-grill-5198728?lid=5198728',
 '/new-orleans-la/mip/chart-room-6924442?lid=6924442',
 '/new-orleans-la/mip/golden-lantern-8517918?lid=8517918',
 '/new-orleans-la/mip/ryans-irish-pub-inc-851820?lid=851820',
 '/new-orleans-la/mip/d-b-a-2084747?lid=2084747',
 '/new-orleans-la/mip/parlays-13663513?lid=13663513',
 '/new-orleans-la/mip/apple-barrel-18379645?lid=18379645',
 '/new-orleans-la/mip/snake-jakes-xmas-club-lounge-4531421?lid=4531421',
 '/new-orleans-la/mip/port-of-call-394043?lid=394043',
 '/new-orleans-la/mip/coops-place-14511722?lid=14511722',
 '/new-orleans-la/mip/twi-ro-pa-466224645?lid=466224645',
 '/new-orleans-la/mip/krazy-korner-11594425?lid=11594425',
 '/new-orleans-la/mip/bourbon-o-480103567?lid=480103567',
 '/new-orleans-la/mip/hi-ho-lounge-458821090?lid=458821090',.....]

Upvotes: 3

von Oak
von Oak

Reputation: 833

I think this is, what you need:

for each_business in info:
    yp_bus_url = each_business.find('a', {'class': 'business-name'}).get('href')
    print(yp_bus_url)

Upvotes: 2

Ajay Bisht
Ajay Bisht

Reputation: 585

The below code should work for you:

import requests
from bs4 import BeautifulSoup

count = 0
search_terms = "Bars"
location = "New Orleans, LA"
url = "https://www.yellowpages.com/search"
q = {'search_terms': search_terms, 'geo_location_terms': location}
page = requests.get(url, params=q)
url_link = page.url
page_num = str(count)
searched_page = url_link + '&page=' + str(count)
page = requests.get(searched_page)
soup = BeautifulSoup(page.text, 'html.parser')
info = soup.findAll('div', {'class': 'info'})
for each_business in info:
   # Your Fix here
   for a in each_business.find_all('a', href=True):
      print("Found the URL:", a['href'])

Upvotes: 2

Related Questions