CODEWITHSUNDEEP
pythonpython-3.xweb-scrapinghtml-tablebeautifulsoup
Randomb
Randomb

Reputation: 43

scraping a table from wikipedia with python : can't get a column

I am trying to scrape a table from Wikipedia

<tr>
  <td>1</td>
  <td><span class="nowrap"><span class="datasortkey" data-sort-value="Etats unis"><span class="flagicon"><a class="image" href="/wiki/Fichier:Flag_of_the_United_States.svg" title="Drapeau des États-Unis"><img alt="Drapeau des États-Unis" class="noviewer thumbborder" data-file-height="650" data-file-width="1235" height="11" src="//upload.wikimedia.org/wikipedia/commons/thumb/a/a4/Flag_of_the_United_States.svg/20px-Flag_of_the_United_States.svg.png" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/a/a4/Flag_of_the_United_States.svg/30px-Flag_of_the_United_States.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/a/a4/Flag_of_the_United_States.svg/40px-Flag_of_the_United_States.svg.png 2x" width="20" /></a> </span><a href="/wiki/%C3%89tats-Unis" title="États-Unis">États-Unis</a></span></span></td>
  <td>19 390,60 </td>
</tr>

as you have noticed there are 3 columns, and here is the code i'm using

A = []
B = []
C = []

for row in DataFondMonetaireInt.findAll("tr"):
    cells = row.findAll("td")
    if len(cells) == 3:
        A.append(cells[0].find(text=True))
        B.append(cells[1].find(text=True))
        C.append(cells[2].find(text=True))

It works well for A and C but not for B, i can't get the country name (in the example : Etats Unis)

why doesn't it work ?

thank you in advance,

Upvotes: 2

Views: 521

Answers (3)

Bertrand Martel
Bertrand Martel

Reputation: 45382

You can also use Wikipedia API to get the WikiText data :

import requests
import wikitextparser as wtp
import re

r = requests.get(
    'https://fr.wikipedia.org/w/api.php',
    params = {
        'action': 'parse',
        'page': 'Liste_des_pays_par_PIB_nominal',
        'contentmodel': 'wikitext',
        'prop': 'wikitext',
        'format': 'json'
    }
)

data = wtp.parse(r.json()['parse']['wikitext']['*'])

f = re.compile(r'[0-9]+[.[0-9]+]?')

for i in range(1, 4):
    print([
        (t[0], wtp.parse(t[1]).templates[0].name, float(f.findall(t[2])[0]))
        for t in data.tables[i].data()
        if len(wtp.parse(t[1]).templates) > 0
    ])

The above will give you data from the 3 tables using WikiTextParser library

Upvotes: 3

QHarr
QHarr

Reputation: 84465

You could do the following to get each table

import pandas as pd
tables = pd.read_html("https://fr.wikipedia.org/wiki/Liste_des_pays_par_PIB_nominal")
[tables[i] for i in range(3)]

Upvotes: 2

ewwink
ewwink

Reputation: 19154

use .text instead of .find(text=True) 

DataFondMonetaireInt = BeautifulSoup(html_text, "html.parser")

A = []
B = []
C = []

for row in DataFondMonetaireInt.findAll("tr"):
    cells = row.findAll("td")
    if len(cells) == 3:
        A.append(cells[0].text)
        B.append(cells[1].text.strip())
        C.append(cells[2].text)

Upvotes: 4

Related Questions