CODEWITHSUNDEEP
pythonxmlbeautifulsoup
royskatt
royskatt

Reputation: 1210

Extracting from HTML with beautifulsoup

I try to extract lotto numbers from https://www.lotto.de/de/ergebnisse/lotto-6aus49/archiv.html (I know there is an easier way, but it's rather for learning).

These are the numbers I want to extract:

Tried with Python, beautifulsoup the following:

from BeautifulSoup import BeautifulSoup
import urllib2

url="https://www.lotto.de/de/ergebnisse/lotto-6aus49/archiv.html"
page=urllib2.urlopen(url)
soup = BeautifulSoup(page.read())
numbers=soup.findAll('li',{'class':'winning_numbers.boxRow.clearfix'})

for number in numbers:
    print number['li']+","+number.string

Returns nothing, which I actually expected. I read the tutorial, but still didn't understand the parsing totally. Could someone give me a hint?

Thank you!

Upvotes: 2

Views: 257

Answers (1)

Anzel
Anzel

Reputation: 20543

As the data content is dynamically generated, one of the EASIER solutions you may use Selenium or alike to simulate the action as a browser (I use PhantomJS as webdriver), like so:

from selenium import webdriver

url="https://www.lotto.de/de/ergebnisse/lotto-6aus49/archiv.html"
# I'm using PhantomJS, you may use your own...
driver = webdriver.PhantomJS(executable_path='/usr/local/bin/phantomjs')
driver.get(url)
soup = BeautifulSoup(driver.page_source)
# I just simply go through the div class and grab all number texts
# without special number, like in the Sample
for ul in soup.findAll('div', {'class': 'winning_numbers'}):
    n = ','.join(li for li in ul.text.split() if li.isdigit())
    if n:
        print 'number: {}'.format(n)

number: 6,25,26,27,28,47

To also grab the special number:

for ul in soup.findAll('div', {'class': 'winning_numbers'}):
    # grab only numeric chars, you may apply your own logic here
    n = ','.join(''.join(_ for _ in li if _.isdigit()) for li in ul.text.split())
    if n:
        print 'number: {}'.format(n)

number: 6,25,26,27,28,47,5 # with special number

Hope this helps.

Upvotes: 3

Related Questions