CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
lord g
lord g

Reputation: 191

How to scrape specific elements using beautifulsoup in Python?

I got PHP file which contains repetitive code, which I'm interesting in. Here's example

<a href="/browse.php?cat=298"><img src="/Static/icons/cat_black_mirror.jpeg" alt="Черное зеркало" title="Черное зеркало" align="left" class="category_icon" border="0" /></a>
    <span class="torrent_title"><b>Заткнись и танцуй (Shut Up and Dance)</b></span><br />
        Дата: <b>01.01.2017 20:51</b><br />Звук: <b>Многоголосый закадровый (LostFilm.TV)</b>
    </span>

What I'm interesting is torrent title and the link. However, tried to go for span with class. And look for link after. Here is example

url = 'http://www.lostfilm.tv/browse.php?'
lost_f = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
lost_soup = BeautifulSoup(lost_f.text,'html.parser',from_encoding="windows-1251")
for item in lost_soup.findAll('span', {'class': 'torrent_title'}):
print item.text
print item.previous_sibling.previous_sibling['href']

Which brings result:name + incorrect link. How could I get torrent name and related link?

Upvotes: 0

Views: 400

Answers (2)

宏杰李
宏杰李

Reputation: 12168

<a href="/browse.php?cat=298"><img src="/Static/icons/cat_black_mirror.jpeg" alt="Черное зеркало" title="Черное зеркало" align="left" class="category_icon" border="0" /></a>

This a tag contain the url href="/browse.php?cat=298" and the title="Черное зеркало", you can get all information in this tag.

import requests, bs4
url = 'http://www.lostfilm.tv/browse.php?'
lost_f = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
lost_soup = bs4.BeautifulSoup(lost_f.text,'lxml',from_encoding="windows-1251")

content_body = lost_soup.find('div', class_="content_body")
a_tags = content_body.select('a[href^="/browse.php?cat="]')
titles = [a.img.get('title') for a in a_tags]
links = [a.get('href') for a in a_tags]

for link, title in zip(links, titles):
    print(link, title)

out:

/browse.php?cat=130 Шерлок
/browse.php?cat=298 Черное зеркало
/browse.php?cat=296 Третий глаз
/browse.php?cat=297 Твин Пикс
/browse.php?cat=298 Черное зеркало
/browse.php?cat=297 Твин Пикс
/browse.php?cat=298 Черное зеркало
/browse.php?cat=219 Викинги
/browse.php?cat=295 Корпорация
/browse.php?cat=298 Черное зеркало
/browse.php?cat=297 Твин Пикс
/browse.php?cat=298 Черное зеркало
/browse.php?cat=294 Стрелок
/browse.php?cat=267 Человек в высоком замке
/browse.php?cat=297 Твин Пикс

Upvotes: 1

Mohammad Yusuf
Mohammad Yusuf

Reputation: 17074

Something like this?

import re

url = 'http://www.lostfilm.tv/browse.php?'
lost_f = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})
lost_soup = BeautifulSoup(lost_f.text,'html.parser', from_encoding="windows-1251")
for a in lost_soup.find_all('a',{'href': re.compile('/browse\.php\?cat=\d+')}):
    print "HREF=", a['href'], "TITLE =", a.text

Upvotes: 2

Related Questions