CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Homerdough
Homerdough

Reputation: 173

Web scraping with BeautifulSoup

I want to scrape the country names and country capitals from this link: https://en.wikipedia.org/wiki/List_of_national_capitals_in_alphabetical_order

From the html code, I'm looking for all of these:

from bs4 import BeautifulSoup
import requests

BASE_URL = "https://en.wikipedia.org/wiki/List_of_national_capitals_in_alphabetical_order"

html = requests.get(BASE_URL).text
soup = BeautifulSoup(html, "html.parser")
countries = soup.find_all("td")

print (countries)

But I don't know how to actually get what's in between the tags, especially since there are ones with no information in them.

I feel like it's pretty simple but I can't really understand all the tutorials since they use classes and this wiki page doesn't have classes for its info inside the table.

Upvotes: 0

Views: 1589

Answers (2)

Martin Evans
Martin Evans

Reputation: 46789

You just need to add some code to iterate over the table columns as follows:

from bs4 import BeautifulSoup
import requests

BASE_URL = "https://en.wikipedia.org/wiki/List_of_national_capitals_in_alphabetical_order"

capitals_countries = []

html = requests.get(BASE_URL).text
soup = BeautifulSoup(html, "html.parser")
country_table = soup.find('table', {"class" : "wikitable sortable"})

for row in country_table.find_all('tr'):
    cols = row.find_all('td')

    if len(cols) == 3:
        capitals_countries.append((cols[0].text.strip(), cols[1].text.strip()))

for capital, country in capitals_countries:
    print('{:35} {}'.format(capital, country))

This would display the capital and country pairs starting as follows:

Abu Dhabi                           United Arab Emirates
Abuja                               Nigeria
Accra                               Ghana
Adamstown                           Pitcairn Islands
Addis Ababa                         Ethiopia
Algiers                             Algeria
Alofi                               Niue
Amman                               Jordan

Upvotes: 1

lord63. j
lord63. j

Reputation: 4670

How about this one:

>>> table = soup.find('table', attrs={'class': 'wikitable'})  # find the table
>>> tds = table.find_all('td')   # get all the table data
>>> countries = [tds[i:i+3] for i in range(0, len(tds), 3)]  # get all the countries' data
>>> result = [[item.text for item in country] for country in countries]  # get the final result
>>> print ' /'.join(result[0])
Abu Dhabi / United Arab Emirates /

Upvotes: 0

Related Questions