CODEWITHSUNDEEP
pythonparsinglxml
James Hallen
James Hallen

Reputation: 5014

Parsing with lxml xpath

I was trying to implement a lxml, xpath code to parse html from link: https://www.theice.com/productguide/ProductSpec.shtml?specId=251 Specifically, I was trying to parse the <tr class="last"> table at near the end of the page.

I wanted to obtain the text in that sub-table, for example: "New York" and the hours listed next to it (and do the same for London and Singapore) .

I have the following code (which doesn't work properly):

doc = lxml.html.fromstring(page)
tds = doc.xpath('//table[@class="last"]//table[@id"tradingHours"]/tbody/tr/td/text()')

With BeautifulSoup:

table = soup.find('table', attrs={'id':'tradingHours'})
for td in table.findChildren('td'):
    print td.text

What is the best method to achieve this? I want to use lxml not beautifulSoup (just to see the difference).

Upvotes: 3

Views: 2280

Answers (3)

unutbu
unutbu

Reputation: 880707

Your lxml code is very close to working. The main problem is that the table tag is not the one with the class="last" attribute. Rather, it is a tr tag that has that attribute:

</tr><tr class="last"><td>TRADING HOURS</td>&#13;

Thus,

//table[@class="last"]

has no matches. There is also a minor syntax error: @id"tradingHours" should be @id="tradingHours".

You can also omit //table[@class="last"] entirely since table[@id="tradingHours"] is specific enough.

The closest analog to your BeautifulSoup code would be:

import urllib2
import lxml.html as LH

url = 'https://www.theice.com/productguide/ProductSpec.shtml?specId=251'
doc = LH.parse(urllib2.urlopen(url))
for td in doc.xpath('//table[@id="tradingHours"]//td/text()'):
    print(td.strip())

The grouper recipe, zip(*[iterable]*n), is often very useful when parsing tables. It collects the items in iterable into groups of n items. We could use it here like this:

texts = iter(doc.xpath('//table[@id="tradingHours"]//td/text()'))
for group in zip(*[texts]*5):
    row = [item.strip() for item in group]
    print('\n'.join(row))
    print('-'*80)

I'm not terribly good at explaining how the grouper recipe works, but I've made an attempt here.

This page is using JavaScript to reformat the dates. To scrape the page after the JavaScript has altered the contents, you could use selenium:

import urllib2
import lxml.html as LH
import contextlib
import selenium.webdriver as webdriver

url = 'https://www.theice.com/productguide/ProductSpec.shtml?specId=251'
with contextlib.closing(webdriver.PhantomJS('phantomjs')) as driver:
    driver.get(url)
    content = driver.page_source
    doc = LH.fromstring(content)
    texts = iter(doc.xpath('//table[@id="tradingHours"]//td/text()'))
    for group in zip(*[texts]*5):
        row = [item.strip() for item in group]
        print('\n'.join(row))
        print('-'*80)

yields

NEW YORK
8:00 PM-2:15 PM *
20:00-14:15
7:30 PM
19:30
--------------------------------------------------------------------------------
LONDON
1:00 AM-7:15 PM
01:00-19:15
12:30 AM
00:30
--------------------------------------------------------------------------------
SINGAPORE
8:00 AM-2:15 AM *
08:00-02:15
7:30 AM
07:30
--------------------------------------------------------------------------------

Note that in this particular case, if you did not want to use selenium, you could use pytz to parse and convert the times yourself:

import dateutil.parser as parser
import pytz

text = 'Tue Jul 30 20:00:00 EDT 2013'
date = parser.parse(text)
date = date.replace(tzinfo=None)
print(date.strftime('%I:%M %p'))
# 08:00 PM

ny = pytz.timezone('America/New_York')
london = pytz.timezone('Europe/London')
london_date = ny.localize(date).astimezone(london)
print(london_date.strftime('%I:%M %p'))
# 01:00 AM

Upvotes: 5

tdelaney
tdelaney

Reputation: 77407

If the site is proper html, id attributes are unique and you can find the table at doc.get_element_by_id('tradingHours').

import urllib
from lxml import html

url = 'https://www.theice.com/productguide/ProductSpec.shtml?specId=251'

response = urllib.urlopen(url).read()

h = html.document_fromstring(response)

print "BY ID"
tradingHours = h.get_element_by_id('tradingHours')

for tr in tradingHours.xpath('tbody/tr'):
    tds = tr.xpath('td')
    print tds[0].text.strip()
    for td in tds[1:]:
        print ' ', td.text.strip()

Results in

BY ID
NEW YORK
  Tue Jul 30 20:00:00 EDT 2013-Tue Jul 30 14:15:00 EDT 2013 *
  Tue Jul 30 19:30:00 EDT 2013
LONDON
  Tue Jul 30 20:00:00 EDT 2013-Tue Jul 30 14:15:00 EDT 2013
  Tue Jul 30 19:30:00 EDT 2013
SINGAPORE
  Tue Jul 30 20:00:00 EDT 2013-Tue Jul 30 14:15:00 EDT 2013 *
  Tue Jul 30 19:30:00 EDT 2013

Upvotes: 1

Faisal
Faisal

Reputation: 2276

I like css selectors much adaptive on page changes than xpaths:

import urllib
from lxml import html

url = 'https://www.theice.com/productguide/ProductSpec.shtml?specId=251'

response = urllib.urlopen(url).read()

h = html.document_fromstring(response)
for tr in h.cssselect('#tradingHours tbody tr'):
    td = tr.cssselect('td')
    print td[0].text_content(), td[1].text_content()

Upvotes: 1

Related Questions