CODEWITHSUNDEEP
pythonweb-scraping
Fouad Elmahdy
Fouad Elmahdy

Reputation: 25

Scraping returning None

I am trying to scrape yellow pages everything working fine except scraping the phone numbers! it's a div class = 'popover-phones' but having an a tag with href = the phone number can anyone assist me please. yellow pages inspection

import item as item
import requests
from bs4 import BeautifulSoup
import json
from csv import writer

url = 'https://yellowpages.com.eg/en/category/charcoal'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'}
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.content, 'html.parser')

articles = soup.find_all('div', class_= 'col-xs-12 item-details')


for item in articles:
    address = item.find('a',class_= 'address-text').text
    company = item.find('a',class_= 'item-title').text
    telephone = item.find('div', class_='popover-phones')enter code here

    print(company,address,telephone)

Upvotes: 1

Views: 59

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195613

The phone numbers you see are loaded from external URL. To get all phone numbers from the page you can use next example:

import requests
from bs4 import BeautifulSoup

url = "https://yellowpages.com.eg/en/category/charcoal"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

for p in soup.select("[data-tooltip-phones]"):
    phone_url = "https://yellowpages.com.eg" + p["data-tooltip-phones"]
    title = p.find_previous(class_="item-title").text
    phones = requests.get(phone_url).json()
    print(title, *[b for a in phones for b in a])

Prints:

2 Bacco 02-3390-8764
3 A Group International 0120-3530-005 057-2428-449
3 A Group International 0120-3833-500 0120-3530-005
Abdel Karim 0122-3507-461
Abdel Sabour Zidan 03-4864-641
Abou Aoday 0111-9226-536 0100-3958-351
Abou Eid For Charcoal Trading 0110-0494-770
Abou Fares For Charcoal Trade 0128-3380-916
Abou Karim Store 0100-6406-939
Adel Sons 0112-1034-398 0115-0980-776
Afandina 0121-2414-087
Ahmed El Fahham 02-2656-0815
Al Baraka For Charcoal 0114-6157-799 0109-3325-720
Al Ghader For Import & Export 03-5919-355 0111-0162-602 0120-6868-434
Al Mashd For Coal 0101-0013-743 0101-0013-743
Al Zahraa Co. For Exporting Charcoal & Agriculture Products 040-3271-056 0100-0005-174 040-3271-056
Alex Carbon Group 03-3935-902
Alwaha Charcoal Trade Est. 0100-4472-554 0110-1010-810 0100-9210-812
Aly Abdel Rahman For Charcoal Trade 03-4804-440 0122-8220-661
Amy Deluxe Egypt 0112-5444-410

Upvotes: 2

Related Questions