CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsouppython-requests
Alex K
Alex K

Reputation: 139

Beautiful soup: Need to click a button to expand the content

I am trying to web scrape this site my link. The problem I have is that I need to expand the content by clicking on the button on the bottom of the page. When I click it, the url updates from www.placera[...]&sida=1 to www.placera[...]&sida=2. So I tried to web scrape using this following code but I get the same content despite I update the page:

import requests
from bs4 import BeautifulSoup

for i in range(1,11):
    URL = f"https://www.placera.se/placera/sok/bolag.html/ALL?query=17747&sida={i}"

    page = requests.get(URL)

    soup = BeautifulSoup(page.content, "html.parser")

    soup.find_all("p", class_="marginTop5px publishedBy")
    print(soup)

Any suggestion how to solve this problem?

Upvotes: 2

Views: 1169

Answers (1)

HedgeHog
HedgeHog

Reputation: 25073

Use the url the content is loaded from and rendered dynamically by JavaScript:

f"https://www.placera.se/placera/sok/bolag/_jcr_content/site-content/cq-column-1.more.html/ALL?query=17747&offset={i*30}"

Check your browsers dev tools in network section on xhr tab, for more infos.

Example

import requests
from bs4 import BeautifulSoup

data = []

for i in range(1,11):
    URL = f"https://www.placera.se/placera/sok/bolag/_jcr_content/site-content/cq-column-1.more.html/ALL?query=17747&offset={i*30}"
    page = requests.get(URL)
    soup = BeautifulSoup(page.content)
    
    for e in soup.select('.searchItem'):
        data.append({
            'title':e.h2.text,
            'content':e.p.text.strip(),
            'date':' '.join(e.p.find_next('p').text.split())
        })
data

Output

[{'title': 'Deutsche Bank höjer riktkursen för Evolution till 1 361 kronor (1 230), upprepar köp',
  'content': 'Evolution: Deutsche Bank höjer riktkursen för Evolution till 1 361 kronor (1 230), upprepar köp [email protected]ån Finwire',
  'date': 'Publicerad: 2023-02-03 07:11 av Finwire'},
 {'title': 'Morgan Stanley höjer riktkursen för Evolution till 1 730 kronor (1 680), upprepar övervikt - BN',
  'content': 'Evolution: Morgan Stanley höjer riktkursen för Evolution till 1 730 kronor (1 680), upprepar övervikt - BN [email protected]ån Finwire',
  'date': 'Publicerad: 2023-02-03 06:50 av Finwire'},
 {'title': 'Carnegie höjer riktkursen för Evolution till 1 460 kronor (1 400), upprepar köp - BN',
  'content': 'Evolution: Carnegie höjer riktkursen för Evolution till 1 460 kronor (1 400), upprepar köp - BN [email protected]ån Finwire',
  'date': 'Publicerad: 2023-02-03 06:50 av Finwire'},...]

Upvotes: 2

Related Questions