CODEWITHSUNDEEP
pythonhtmlbeautifulsoup
O Tal Antiquado
O Tal Antiquado

Reputation: 397

Parsing table columns to a list each

I'm working with a public bus website with a specific bus stop (see variable 'url') and I want to parse every column (Bus Line - Depart time - ETA) to a list each, but I'm getting weird results with this code:

import requests                                                     
from bs4 import BeautifulSoup                                                                               

url = 'http://www.stcp.pt/pt/itinerarium/soapclient.php?codigo=AAL1'
r = requests.get(url)                                               
soup = BeautifulSoup(r.content, 'html.parser')                      
buses = []                                                          


for table in soup.find_all('table', attrs={                         
    'id': 'smsBusResults'                                           
}):                                                                 
    for row in table.find_all('tr', attrs={                         
        'class': 'even'                                             
    }):                                                             
        for col in row.find_all('td'):                              
            buses.append(row.get_text().strip())                    
            print(buses)

Note: If you see "a passar", that means "passing by"

Upvotes: 0

Views: 85

Answers (1)

iamklaus
iamklaus

Reputation: 3770

try this

from bs4 import BeautifulSoup
import requests
import pandas as pd

data = requests.get('http://www.stcp.pt/pt/itinerarium/soapclient.php?codigo=AAL1').content
soup = BeautifulSoup(data)

table = soup.find_all('table', {'id':'smsBusResults'})

tr = table[0].find_all('tr')



headers = []
for td in tr[0].find_all('th'):
    headers.append(td.text)
temp_df = pd.DataFrame(columns=headers)


pos = 0
for i in range(1,len(tr)):
    temp_list = []
    for td in tr[i].find_all('td'):
        value = (td.text).replace('\n','')
        value = value.replace('\t','')
        temp_list.append(value)    
    temp_df.loc[pos] = temp_list
    pos+=1

print(temp_df)

Output

                Linha Hora Prevista Tempo de Espera
0    600  AV. ALIADOS         16:29            1min
1   202  AV.ALIADOS -         16:34            6min
2    600  AV. ALIADOS         16:41           12min
3    600  AV. ALIADOS         16:50           21min

Upvotes: 1

Related Questions