CODEWITHSUNDEEP
pythonhtmlpython-3.xweb-scrapingbeautifulsoup
sombra2
sombra2

Reputation: 35

Data scraper: the contents of the div tag is empty (??)

I am data scraping a website to get a number. This number changes dynamically every split second, but upon inspection, the number is shown. I just need to capture that number but the div wrapper that contains it, it returns no value. What am I missing? (please go easy on me as I am quite new to Python and data scraping).

I have some code that works and returns the piece of html that supposedly contains the data I want, but no joy, the div wrapper returns no value.

import requests
from bs4 import BeautifulSoup


r = requests.get('https://deuda-publica-espana.com')

deuda = BeautifulSoup(r.text, 'html.parser')
deuda = deuda.findAll('div', {'id': 'contador_PDEH'})

print(deuda)

I don't receive any errors, I am just getting [<div class="contador_xl contador_verde" id="contador_PDEH"></div>] with no value!

Upvotes: 1

Views: 471

Answers (1)

QHarr
QHarr

Reputation: 84465

Indeed it is easy with selenium. I suspect there is a js script running a counter supplying the number which is why you can't find it with your method (as mentioned in comments)

from selenium import webdriver

d = webdriver.Chrome(r'C:\Users\User\Documents\chromedriver.exe')
d.get('https://deuda-publica-espana.com/')
print(d.find_element_by_id('contador_PDEH').text)
d.quit()

Upvotes: 1

Related Questions