CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Varun S
Varun S

Reputation: 25

Problems in geting article content while scraping news website using beautiful soup

I am trying to scrape news article from rss feed along with the details like title, description, URL and date. I am not getting the entire article content in the description column as expected. Below is my code.

import requests
from bs4 import BeautifulSoup as bs

url='https://www.business-standard.com/rss/economy-policy-102.rss'
resp= requests.get(url)
soup = bs(resp.content,features='xml')
items= soup.findAll('item')
news_items = []

for item in items:
    news_item = {}
    news_item['title'] = item.title.text
    news_item['description'] = item.description.text
    news_item['link'] = item.link.text
    news_item['pubDate'] = item.pubDate.text
    news_items.append(news_item)

import pandas as pd
df = pd.DataFrame(news_items,columns=['title','description','link','pubDate'])
df['description'][0]

Output obtained - 'The re-import in the extended period would be without payment of basic customs duty and integrated goods and services tax'

As seen above I am not getting the full article content. What changes should be made?

Upvotes: 0

Views: 100

Answers (1)

Andrej Kesely
Andrej Kesely

Reputation: 195448

The RSS feed doesn't contain full text of the articles, you have to open the link and grab the article from there.

For example:

import requests
from bs4 import BeautifulSoup


url='https://www.business-standard.com/rss/economy-policy-102.rss'
soup = BeautifulSoup(requests.get(url).content, 'xml')

news_items = []
for item in soup.findAll('item'):
    news_item = {}
    news_item['title'] = item.title.text
    news_item['excerpt'] = item.description.text

    print(item.link.text)
    s = BeautifulSoup(requests.get(item.link.text).content, 'html.parser')

    news_item['text'] = s.select_one('.p-content').get_text(strip=True, separator=' ')
    news_item['link'] = item.link.text
    news_item['pubDate'] = item.pubDate.text
    news_items.append(news_item)

import pandas as pd
df = pd.DataFrame(news_items)
df.to_csv('data.csv')

Creates data.csv (screenshot from LibreOffice):

enter image description here

Upvotes: 1

Related Questions