CODEWITHSUNDEEP
pythonseleniumweb-scrapingphantomjs
Amr Yousef
Amr Yousef

Reputation: 594

Scrape Table using selenium and PhantomJS

I am trying to scrape the following table :

enter image description here

My code is working when I am using the chrome web driver but when using PhantomJS driver the output doesn't seem to get the numbers, it only gets the text.

enter image description here

My Python code is this :

    from selenium import webdriver

path_to_chromedriver = '/Users/amr_f/Desktop/chromedriver' # change path as needed
browser = webdriver.PhantomJS('/home/ubuntu/phantomjs-2.1.1-linux-x86_64/bin/phantomjs')
url = 'http://www.cibeg.com/English/Pages/default.aspx'
browser.get(url)

browser.find_element_by_xpath('//*[@id="sliderHome"]/div[2]/div/ul/li[3]/a').click()

data = []

for tr in browser.find_elements_by_xpath('//*[@id="divCurrTableContainer"]/table'):
    tds = tr.find_elements_by_tag_name('td')
    if tds: 

        data.append([td.text for td in tds])
print(data)

Upvotes: 0

Views: 661

Answers (1)

user2096803
user2096803

Reputation:

By adding, browser.set_window_size(1124, 850), to set the window size for the PhantomJS driver I was able to retrieve the table's data from the page.

This happens, if I'm recalling this correctly, because certain javascript libraries use the window's size "on load". Not having the window size parameter can cause the routine to not correctly load all the elements on the page.

from selenium import webdriver


browser = webdriver.PhantomJS('/home/ubuntu/phantomjs-2.1.1-linux-x86_64/bin/phantomjs')
browser.set_window_size(1124, 850)
url = 'http://www.cibeg.com/English/Pages/default.aspx'
browser.get(url)    
browser.find_element_by_xpath('//*[@id="sliderHome"]/div[2]/div/ul/li[3]/a').click()    
data = []

for tr in browser.find_elements_by_xpath('//*[@id="divCurrTableContainer"]/table'):
    tds = tr.find_elements_by_tag_name('td')
    if tds:     
        data.append([td.text for td in tds])

print(data)

After I added the window size I was able to retrieve:

[['USD', '16.26', '16.75', 'EUR', '17.6696', '18.3563', 'GBP', '20.0895', '20.8621', 'CHF', '16.4571', '17.0536', 'SAR', '4.3297', '4.4663', 'KWD', '53.5202', '55.3353']]

Upvotes: 2

Related Questions