CODEWITHSUNDEEP
web-scrapingbeautifulsouppython-requestsweb-crawler
Quintis
Quintis

Reputation: 27

Scroll to next page and extract data

i'm trying to extract all body texts in Latest updates from 'https://www.bbc.com/news/coronavirus'

i have successfully extracted body texts from the first page (1 out of 50).

I would like to scroll to the next page and do this process again.

This is the code that i have written.

from bs4 import BeautifulSoup as soup
import requests

links = []
header = []
body_text = []

r = requests.get('https://www.bbc.com/news/coronavirus')
b = soup(r.content,'lxml')

# Selecting Latest update selection
latest = b.find(class_="gel-layout__item gel-3/5@l")
# Getting title
for news in latest.findAll('h3'):
    header.append(news.text)
    #print(news.text)

# Getting sub-links
for news in latest.findAll('h3',{'class':'lx-stream-post__header-title gel-great-primer-bold qa-post-title gs-u-mt0 gs-u-mb-'}):
    links.append('https://www.bbc.com' + news.a['href'])

# Entering sub-links and extracting texts
for link in links:
    page = requests.get(link)
    bsobj = soup(page.content,'lxml')

    for news in bsobj.findAll('div',{'class':'ssrcss-18snukc-RichTextContainer e5tfeyi1'}):
        body_text.append(news.text.strip())
        #print(news.text.strip())

How should i scroll to the next page ?

Upvotes: 0

Views: 186

Answers (1)

chitown88
chitown88

Reputation: 28565

Not sure what text you are exactly after, but you can go through the api.

import requests

url = 'https://push.api.bbci.co.uk/batch'
headers = {'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.106 Mobile Safari/537.36'}

for page in range(1,51):
    payload = '?t=%2Fdata%2Fbbc-morph-lx-commentary-data-paged%2Fabout%2F63b2bbc8-6bea-4a82-9f6b-6ecc470d0c45%2FisUk%2Ffalse%2Flimit%2F20%2FnitroKey%2Flx-nitro%2FpageNumber%2F{page}%2Fversion%2F1.5.4?timeout=5'.format(page=page)

    jsonData = requests.get(url+payload, headers=headers).json()

    results = jsonData['payload'][0]['body']['results']
    for result in results:
        print(result['title'])
        print('\t',result['summary'],'\n')

Upvotes: 1

Related Questions