CODEWITHSUNDEEP
pythonselenium-webdriverweb-scrapingphantomjs
Abdur
Abdur

Reputation: 27

Table extraction with Selenium and PhantomJS never finishes

I am trying to extract a table from a site using the following code (courtesy: Padraic). When I execute this code it goes into execution and does not end or return anything unless terminated. 

from selenium import webdriver

import pandas as pd

dr = webdriver.PhantomJS(r'C:\Users\Admin\Anaconda3\phantomjs-2.1.1-windows\bin\phantomjs.exe')

elements=[]

url='http://www.moneycontrol.com/stocks/fno/marketstats/options/active_calls/index.php'


dr.get(url)  
table = dr.find_element_by_css_selector("div.MT15")

for row in table.find_elements_by_xpath(".//tr"):
        elem=(":".join([td.text.replace("\n","") for td in \   
        row.find_elements_by_xpath(".//td")]))
        element= elem.split(":")
        elements.append(element)

print (elements)

Upvotes: 1

Views: 602

Answers (1)

Padraic Cunningham
Padraic Cunningham

Reputation: 180461

If you add a print(row) in the loop, you can see output like:

<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003017")>
<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003018")>
<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003019")>
<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003020")>
<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003021")>
<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003022")>
<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003023")>
<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003024")>
<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003025")>
<selenium.webdriver.remote.webelement.WebElement (session="10676710-2e8c-11e6-b13a-473272d23fd8", element=":wdc:1465509003026")>

There are ~1600 tr tags in the source, most of which are in the div you are searching so that is why it seems to loop for a long time. The code is working, it is just going to take a while to complete.

You might also find this runs in a fraction of the time, it completes in about one second on my laptop:

import requests
from bs4 import BeautifulSoup, Tag
r = requests.get(url)
soup = BeautifulSoup(r.content, "lxml")

table = soup.select_one("table.tblList")
cols = [th.text for th in table.select_one("tr") if isinstance(th, Tag)]
print(cols)

elems = [[td.text for td in row if isinstance(td, Tag)] for row in table.select("tr + tr")]

print(elems)

If we run the code:

In [13]: import requests

In [14]: from bs4 import BeautifulSoup, Tag

In [15]: url = 'http://www.moneycontrol.com/stocks/fno/marketstats/options/active_calls/index.php'

In [16]: r = requests.get(url)

In [17]: soup = BeautifulSoup(r.content, "lxml")

In [18]: table = soup.select_one("table.tblList")

In [19]: cols = [th.text.strip() for th in table.select_one("tr") if isinstance(th, Tag)]

In [20]: print(cols)
[u'Symbol', u'Expiry\n Date', u'Option Type', u'Strike Price', u'LastPrice', u'Change\n \t\t\t\t\t\t\t\tChg%', u'High\n  Low', u'Shares', u'Contracts', u'Value (Rs. Lakh)', u'Open Interest', u'Open Int Chg']

In [21]: elems = [[td.text.strip() for td in row if isinstance(td, Tag)] for row in table.select("tr + tr")]

In [22]: print(elems[0])
[u'IFCI', u'30-Jun-16', u'CE', u'27.50', u'0.50', u'0.25100.00%', u'0.650.20', u'18,760,000', u'938', u'90.05', u'6,000,000', u'2,520,00072.41%']

In [23]: print(elems[-1])
[u'EICHERMOT', u'30-Jun-16', u'CE', u'20,800.00', u'30.00', u'-30.00-50.00%', u'30.0030.00', u'25', u'1', u'0.01', u'225', u'00.00%']

In [24]: len(elems)
Out[24]: 1585

You can see there are 1585 rows in the table. I outputted just the first and last row as there is way too much data to post but it gets the full table for you.

Upvotes: 2

Related Questions