CODEWITHSUNDEEP
pythonbeautifulsoup
lture
lture

Reputation: 33

Scraping href links with Beautifulsoup

I have a code like this

onexurl = "https://1xbet.com/en/live/Football/"
reply = requests.get(onexurl)
soup = BeautifulSoup(reply.content, "html.parser")
links = soup.find_all("a", {"class": "c-events__name"})
print(links)
urls = []
for matchlink in links:
    urls.append("https://1xbet.com/en/"+(matchlink.get("href")))
print(urls)

to get links from a page.

which's one of the results is like below:

https://1xbet.com/en/live/Football/24581-AFC-Champions-League/207140194--/

But original source code is this:

<a href="live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/" class="c-events__name"><span title="Kashima Antlers — Guangzhou Evergrande  " class="c-events__teams"><div class="c-events-scoreboard__team-wrap"><div class="c-events__team">Kashima Antlers</div> <!----> <!----></div> <div class="c-events-scoreboard__team-wrap"><div class="c-events__team"> Guangzhou Evergrande</div> <!----> <!----></div> <!----> <!----> <!----></span></a>

Why doesn't (matchlink.get("href") get the whole text of the link?

Upvotes: 3

Views: 202

Answers (2)

Mrugesh Kadia
Mrugesh Kadia

Reputation: 555

import requests
from bs4 import BeautifulSoup

onexurl = "https://1xbet.com/en/live/Football/"
reply = requests.get(onexurl)
soup = BeautifulSoup(reply.content, "html.parser")
links = soup.find_all("a", {"class": "c-events__name"})

urls = []
for matchlink in links:
    url = "https://1xbet.com/en/"+(matchlink["href"]).replace('--/', '')
    teams = matchlink.text
    remaining_url = ( teams.strip().replace('\n', '-').replace('(', '-').replace(')', '-').replace(' ', '-').replace('--', '-'))
    final_url = url + '-' + remaining_url
    urls.append(final_url.lower())
print(urls)

Which gives you list of URLs:

['https://1xbet.com/en/live/football/1999982-5h5-dragon-league-league-b/207278079-manchester-city-team-manchester-united-team', 'https://1xbet.com/en/live/football/1471313-indonesia-liga-1/207271440-badak-lampung-kalteng-putra', 'https://1xbet.com/en/live/football/1471313-indonesia-liga-1/207271451-psm-makassar-ps-tira', ]

Upvotes: 2

Nathan
Nathan

Reputation: 3190

There is something else going on here. Let's examine the behavior of the parser:

html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/" class="c-events__name"><span title="Kashima Antlers — Guangzhou Evergrande  " class="c-events__teams"><div class="c-events-scoreboard__team-wrap"><div class="c-events__team">Kashima Antlers</div> <!----> <!----></div> <div class="c-events-scoreboard__team-wrap"><div class="c-events__team"> Guangzhou Evergrande</div> <!----> <!----></div> <!----> <!----> <!----></span></a>,
<a href="live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/" class="c-events__name"><span title="Kashima Antlers — Guangzhou Evergrande  " class="c-events__teams"><div class="c-events-scoreboard__team-wrap"><div class="c-events__team">Kashima Antlers</div> <!----> <!----></div> <div class="c-events-scoreboard__team-wrap"><div class="c-events__team"> Guangzhou Evergrande</div> <!----> <!----></div> <!----> <!----> <!----></span></a>
<a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>"""


soup = BeautifulSoup(html_doc, 'html.parser')
for link in soup.find_all('a', {"class": "c-events__name"}):
    print(link.get('href'))

Returns the full value of the href tag you provided. 

live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/
live/Football/24581-AFC-Champions-League/207140194-Kashima-Antlers-Guangzhou-Evergrande/

The next step is to check how the links are appended to the list you created to hold them. We can simplify this expression:

my_list = []
for ml in links:
    my_list.append("http://url.com/" + ml.get("href"))

to a list comprehension:

my_list = ["http://url.com/" + ml.get("href") for ml in links]

And the href values should be stored in a nice list. If that is broken, make sure your BeautifulSoup filtering is returning what you think it is.

Upvotes: 0

Related Questions