CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
James Huang
James Huang

Reputation: 61

Web Scraping tag of html in Python

I would like to scrape all the links that end with .php I have written a regrex to select the target url such as samsung-phones-f-9-0-r1-p1.php

I am wondering if there's something wrong with my regrex or the tag is not correct.

Thank you so much in advance for answering

from bs4 import BeautifulSoup
import urllib.request as urlopen
import ssl 
import re

base_url = 'https://www.gsmarena.com/samsung-phones-9.php'
webrequest = request.Request(url, headers = {
    "User-Agent" : 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.101 Safari/537.36'})
    
    
# open the url
html = request.urlopen(base_url).read().decode('utf-8')
soup = BeautifulSoup(html, features = 'lxml')
# scraping sub urls
sub_urls = soup.find_all('a', {"href": re.compile("(samsung).+(.php)")})
# https:\/\/www\.gsmarena\.com\/samsung.+(.php)
print(sub_urls)

The wrong result is demonstrated in bellow

Upvotes: 0

Views: 74

Answers (2)

BrKo14
BrKo14

Reputation: 41

Your regular expression is not exactly wrong, since it will capture the URLs you are aiming for. However, two points to consider: (1) the parenthesis are unnecessary and (2) you should escape the . character in .php, or it'll be compiled as a quantifier. A better solution might look like this: samsung.+\.php. Of course this is will only capture the .php file itself and not the whole URL. If wanted the whole thing you'd have to use .*samsung.+\.php.
You can always use an online tool to test your regular expressions.

Either way, you haven't outlined any concrete issues, therefore I'm not sure if I've satisfied your doubts.

Upvotes: 0

idar
idar

Reputation: 610

You are doing it right but you are not extracting the actual href property from the tags.
Modify this line:

sub_urls = soup.find_all('a', {"href": re.compile("(samsung).+(.php)")})

to this:

sub_urls = [x.get('href') for x in soup.find_all('a', {"href": re.compile("(samsung).+(.php)")})]

Upvotes: 1

Related Questions