CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Faheem Akhtar
Faheem Akhtar

Reputation: 113

How to extract text from <br> tags in Python?

Trying to extract text from br tags. How do we do that?

So far I have tried different combinations but no success. Complete beginner.

from requests import get
from bs4 import BeautifulSoup


article = "https://www.readlightnovel.org/martial-god-asura/chapter-4095"
r = get(article, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"})
soup = BeautifulSoup(r.content, "lxml")
title = soup.find(class_="block-title")
print(title.text.strip())
full_article = soup.find("div", {"class": "desc"})
for br_tags in full_article:
    desc = br_tags.findAll("br")
    print(desc.text.strip())

Upvotes: 0

Views: 589

Answers (3)

Andrej Kesely
Andrej Kesely

Reputation: 195613

This code prints text of the novel. It does basic preprocessing (removing ads etc.): 

from requests import get
from bs4 import BeautifulSoup

article = "https://www.readlightnovel.org/martial-god-asura/chapter-4095"
r = get(article, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"})
soup = BeautifulSoup(r.content, "lxml")
title = soup.find(class_="block-title")
print(title.text.strip())

full_article = soup.find("div", {"class": "desc"})

# remove ads inside the text:
for ads in full_article.select('center, small, a, .hidden'):
    ads.extract()

print( full_article.get_text(strip=True, separator='\n') )

Prints:

Martial God Asura - Chapter 4095
Chapter 4095
Hundreds of millions of lightning bolts struck down from the sky and continued to bombard Chu Feng’s location .
Being bombarded in such a manner, everyone felt that Chu Feng had long been killed .
Because of that, they were feeling confused .
Since Chu Feng was already dead, why was the lightning still attacking his location nonstop?
Furthermore, the attacks were growing fiercer and fiercer .
Confused, the crowd’s eyes turned to Chu Feng’s location, their gazes fixed onto that place .
Because of that, no one noticed that the humanoid body of light in the clouds had actually lowered its head . It was as if it were looking at Chu Feng’s location .
In fact, that was exactly what was happening .

... and so on.

Upvotes: 2

Umar Aftab
Umar Aftab

Reputation: 535

You just need to look at next siblings :

 for br_tags in full_article:
    desc = br_tags.findAll("br")
    print(''.join(desc.next_siblings))

This might work to get the next br tag ! I just took the snippet from your code !

Upvotes: 0

Zesty Dragon
Zesty Dragon

Reputation: 561

If you just want any text which is between two
tags, you could do something like the following:

from BeautifulSoup import BeautifulSoup, NavigableString, Tag

input = '''<br />
Important Text 1
<br />
<br />
Not Important Text
<br />
Important Text 2
<br />
Important Text 3
<br />
<br />
Non Important Text
<br />
Important Text 4
<br />'''

soup = BeautifulSoup(input)

for br in soup.findAll('br'):
    next_s = br.nextSibling
    if not (next_s and isinstance(next_s,NavigableString)):
        continue
    next2_s = next_s.nextSibling
    if next2_s and isinstance(next2_s,Tag) and next2_s.name == 'br':
        text = str(next_s).strip()
        if text:
            print "Found:", next_s

Upvotes: 0

Related Questions