CODEWITHSUNDEEP
pythonbeautifulsoup
realblockhead
realblockhead

Reputation: 47

How to scrape spacing along with paragraphs

How do I add the spacing?

Right now the output is clustered I want to scrape the spacing along with the paragraphs.

I've seen other persons use get_text separator, but I am not using it.

from urllib.request import urlopen
from bs4 import BeautifulSoup

# specify the url
url = "https://www.bbc.com/sport/football/50944416"

# Connect to the website and return the html to the variable ‘page’
try:
    page = urlopen(url)
except:
    print("Error opening the URL")

# parse the html using beautiful soup and store in variable `soup`
soup = BeautifulSoup(page, 'html.parser')

# Take out the <div> of name and get its value
content = soup.find('div', {"class": "story-body sp-story-body gel-body-copy"})

article = ''
for i in content.findAll('p'):
    article = article + ' ' +  i.text
print(article)

Upvotes: 2

Views: 200

Answers (2)

PaxPrz
PaxPrz

Reputation: 1928

string.strip() command might help you

article = ''
for i in content.findAll('p'):
    article = article + '\t' +  i.text.strip() + '\n'
print(article)

Upvotes: 0

Marsilinou Zaky
Marsilinou Zaky

Reputation: 1047

You can textwrap from the standard library to specify the length of each line and add two empty lines as seperator every each paragraph p

from urllib.request import urlopen
from bs4 import BeautifulSoup
import textwrap

article = ''
line_size = 75

for i in content.findAll('p'):
    w = textwrap.TextWrapper(width=line_size,break_long_words=False,replace_whitespace=False)
    body = '\n'.join(w.wrap(i.text))
    article += body+"\n\n"
print(article)

You can do it using loops, but i would recommend using the textwrap better as it handles breaking words and much simpler, anyways basic way to do it manually would be something like:

article = ''
for i in content.findAll('p'):
    text = i.text.strip()
    for n in range(len(text)):
        if n % line_size != 0 and i!=0:
            article += text[n]
        else:
            article += "\n" + text[n]
    article+="\n\n"
print(article)

Upvotes: 1

Related Questions