How do I extract a table from a webpage using selenium when the table is not constructed with the HTML 'table' tag?

Question

I am trying to extract the tables generated by selecting "Branches", a city and a district from this site: https://www.acb.com.vn/wps/portal/en/atm

So far, I have been able to write the code to parse through each city and district:

from selenium.webdriver.support.ui import Select
from selenium.webdriver import Chrome
import pandas as pd
import time

webdriver = "chromedriver.exe"

driver = Chrome(webdriver)
driver.get('https://www.acb.com.vn/wps/portal/en/atm')
    
branch_selector = driver.find_element_by_xpath('//*[@id="branch"]')
branch_selector.click()

city = Select(driver.find_element_by_id('cityId'))

for i in range(len(city.options)):
    city.select_by_index(i)
    time.sleep(1)
    
    district = Select(driver.find_element_by_id('districtId'))

    for j in range(len(district.options)):
        district.select_by_index(j)
        time.sleep(1)

        try:
            find_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]')
            find_btn.click()
            time.sleep(1)
            
        except:
            close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]')
            close_btn.click()
            time.sleep(1)

Now, I want to extract the table that's displayed in each iteration of the 2 loops. However, if you look at the HTML for the table, it does not make use of the "table" tag:

So, how do I extract the table for each city-district pair?

I tried the following:

    try:
        click_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]')
        click_btn.click()
        time.sleep(1)
        
        table = driver.find_elements_by_class_name('tbody')
        for table_row in table:
            row = table_row.find_elements_by_class_name('row')
            print ([r.text for r in row])
        
    except:
        close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]')
        close_btn.click()
        time.sleep(1)

But it prints a list of blank elements for each city-district pair, the length of the list being as many addresses are present in the table for the corresponding city-district pair:

['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '', '']
['', '', '', '']
['', '']
['', '']

I also tried to access each element in each row of the table individually:

    try:
        find_btn = driver.find_element_by_xpath('//*[@id="frm-filter"]/div[3]/a[1]')
        find_btn.click()
        time.sleep(1)
        
        table = driver.find_elements_by_class_name('tbody')
        for table_row in table:
            row = table_row.find_elements_by_class_name('row')
            
            for element in row:
                time.sleep(1)
                
                Type.append(element.find_element_by_class_name('col type'))
                Address.append(element.find_element_by_class_name('col address'))
                District.append(element.find_element_by_class_name('col district'))
                Tel_Fax.append(element.find_element_by_class_name('col tel-fax'))
                Hours.append(element.find_element_by_class_name('col hours'))
        
    except:
        close_btn = driver.find_element_by_xpath('//*[@id="close-send-email"]/span[2]')
        close_btn.click()
        time.sleep(1)

But this gives the following error:

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
 in 
     39 
---> 40                     Type.append(element.find_element_by_class_name('col type'))
     41                     Address.append(element.find_element_by_class_name('col address'))

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="col type"]"}

Since it says css selector in the error, I tried the following:

element.find_element_by_css_selector('div.col.type').text

This outputs a blank string, ''.

So, how do I do this?

EDIT: The HTML of the table, for one district-city selection, is:

        
            
            
                1
                
                
                PGD Hai Bà Trưng
                56-58-60 Hai Bà Trưng, P. Bến Nghé, Quan 1, Ho Chi Minh
                1
                (028) 6291 3690
(028) 6291 3691
                      07:00-16:30
                Direction
            
            
            
                2
                
                
                PGD Đa Kao
                45 Võ Thị Sáu, P. Đa Kao, Quan 1, Ho Chi Minh
                1
                (028) 6290 5980
(028) 6290 5981
                  07:30 – 16:30
                Direction
            
            
            
                3
                
                
                PGD Nguyễn Công Trứ
                74 - 76 Nguyễn Công Trứ, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh
                1
                (028) 3914 4470 
(028) 3914 4471
                  07:30 – 16:30
                Direction
            
            
            
                4
                
                
                PGD Lê Lợi
                72 Lê Lợi, P. Bến Thành, Quận 1, TP.Hồ Chí Minh
                1
                (028) 3821 4619
(028) 3821 4618
                    07:00-16:30
                Direction
            
            
            
                5
                
                
                CN Sài Gòn
                41 Mạc Đỉnh Chi, P. Đakao, Quan 1, Ho Chi Minh
                1
                (028) 3824 3770
(028) 3824 3946
                  07:30 – 16:30
                Direction
            
            
            
                6
                
                
                PGD Nguyễn Thái Bình
                176 – 178 Ký Con, P. Nguyễn Thái Bình, Quan 1, Ho Chi Minh
                1
                (028) 3915 1310
(028) 3915 1311
                  07:30 – 16:30
                Direction
            
            
            
                7
                
                
                PGD Bến Chương Dương
                328 Võ Văn Kiệt, phường Cô Giang, Quận 1, Tp.HCM
                1
                (028) 3837 0586
(028) 3837 0584
                   7h30-16h30
                Direction
            
            
            
                8
                
                
                PGD Trần Khắc Chân
                48-50 Nguyễn Hữu Cầu, P.Tân Định, Q.1, TP.HCM
                1
                (028) 3820 9990
(028) 3526 7738
                 07:30 -16:30
                Direction
            
            
            
                9
                
                
                PGD Cống Quỳnh
                106  108 Cống Quỳnh, P. Nguyễn Cư Trinh, Q.1
                1
                (028) 38385464
(028) 3925 6645
                 07:30 -16:30
                Direction
            
            
            
                10
                
                
                CN Bến Thành
                96 Lý Tự Trọng, P. Bến Thành, Quan 1, Ho Chi Minh
                1
                (028) 3825 7949
(028) 3825 7950
                 07:30-16:30
                Direction
            
            
            
                11
                
                
                PGD Tân Định   
                261 Trần Quang Khải, Phường Tân Định, Quận 1, TP.HCM
                1
                (028) 3848 0520

                 07:30 - 16:30
                Direction
            
            
            
                12
                
                
                PGD Nguyễn Du
                Tầng hầm 1, tầng trệt, tầng lửng và tầng 2 tòa nhà 480 đường Nguyễn Thị Minh Khai, Phường 2, Quận 3, TP.Hồ Chí Minh
                1
                (028) 35218626
(028) 35218627
                 07:30 -16:30
                Direction

bigbounty · Accepted Answer

On analysing the website, it makes a post request on submitting the form. The function in the website is as follows:

function findMap() {        
        var keyWord =document.getElementById("keyWord").value;
        var cityId =document.getElementById("cityId").value;
        var districtId =document.getElementById("districtId").value;
        var isCheckBranch = document.getElementById("branch").checked;
        var isCheckAtm = document.getElementById("atm").checked;
        var isCheckWestern = document.getElementById("western").checked;
        var isCheckCdm = document.getElementById("cdm").checked;
        var branch="";
        var atm="";
        var western="";
        var cdm="";
        
        var input = document.getElementById ("keyWord");
        var placeholder = input.placeholder;
        if( keyWord == placeholder ){
            keyWord = "";
        }
        
        if((!isCheckBranch) && (!isCheckAtm) && (!isCheckWestern) && (!isCheckCdm)){
            showMessage('Please select Branch or ATM or Western Union or CDM.', 'branch');
            return;
        }
        
        if((!districtId || 0 === districtId.length) && (!keyWord || 0 === keyWord)){
            showMessage('Please select the province or enter the address.', 'keyWord');
            return;
        }
        if(isCheckBranch){
            branch = "branch";
        }
        if(isCheckAtm){
            atm = "atm";
        }
        if(isCheckWestern){
            western = "western";
        }
        if(isCheckCdm){
            cdm = "cdm";
        }
        var url = '/ACBMapPortlet/en/Process.jsp';
        var urlPattern = 'https://www.acb.com.vn:443/ACBMapPortlet/en/MapMobi.jsp';
        $( "#resultSearch" ).load( url, { "params[]": [ "Search", branch, atm, western, cdm, districtId, keyWord, cityId, latlng, urlPattern]} );
        
    }

So, now you can understand what happens when you click the submit button.

The website constructs the values as a form data. I'll explain one such request that contains as in the following screenshot

Scraping in python using the above information.

Here - cityId = 18, DistrictId (populated through ajax call) = 187

import requests
from bs4 import BeautifulSoup
import pandas as pd

res=requests.post("https://www.acb.com.vn/ACBMapPortlet/en/Process.jsp", data={"params[]": ["Search","branch","atm","western","cdm",187,"",18,0,"https://www.acb.com.vn:443/ACBMapPortlet/en/MapMobi.jsp"]})

result = res.text.replace("
","").replace("	","").replace("
","")

soup = BeautifulSoup(result, "lxml")
headers = [i.text.strip() for i in soup.find("div",class_="thead").find_all("div",class_="col")[:-1]]
body = [[j.text.strip() for j in i.find_all("div",class_="col")[:-1]] for i in soup.find("div",class_="tbody").find_all("div",class_="row")]

df = pd.DataFrame(body, columns=headers)

print(df)

df.to_csv("data.csv", index=False)

Output:

Update 1:

In order to get the city Id - city id is hard coded in the website in the select tag value attribute.

In order to district Id: To get this, the website makes an ajax call.

function getDistrict(cityId) {
        var url = '/ACBMapPortlet/en/DistrictSelectBox.jsp';
        $.post( url, { cmd:'DISTRICT', cityId:cityId}, function(data) {
            var content = $( data );
            $("#divDistrict span").empty().append("District");
            $("#iconselect").empty();
            $("#districtId").empty().append(content);               
         });     
    }

How to get all districts given a cityId?

def districts_names(cityid):
    data = {"cmd":"DISTRICT", "cityId": cityid}
    res = res = requests.post("https://www.acb.com.vn/ACBMapPortlet/en/DistrictSelectBox.jsp", data=data)
    soup = BeautifulSoup(res.text, "lxml")
    return [(i["value"].strip(),i.text.strip()) for i in soup.find_all("option")]

Example: districts_names(3) will give the following

[('', 'District'),
 ('234', 'Ba Bể'),
 ('235', 'Bạch Thông'),
 ('236', 'Chợ Đồn'),
 ('237', 'Chợ Mới'),
 ('238', 'Na Rì'),
 ('239', 'Ngân Sơn'),
 ('240', 'Bắc Kạn')]

The output is of the format - (district_id, district_name)

How do I extract a table from a webpage using selenium when the table is not constructed with the HTML 'table' tag?

Answers (1)

Related Questions

How do I extract a table from a webpage using selenium when the table is not constructed with the HTML &#39;table&#39; tag?

Answers (1)

Related Questions

How do I extract a table from a webpage using selenium when the table is not constructed with the HTML 'table' tag?