CODEWITHSUNDEEP
pythonpython-2.7xpathweb-scrapingscrapy
waylonion
waylonion

Reputation: 6976

Scrapy iterate through sibling nodes - xpath yields empty list

I'm iterating over the sibling nodes of an element. However, I'm having trouble using the xpath function to extract the data within. I seem to always obtain an empty list.

# Determine if the cast members are listed in a table or unordered list.
cast_siblings = response.xpath('//h2/span[starts-with(@id,"Cast")]/../following-sibling::*')
for sibling in cast_siblings:
    desc_str = "".join(sibling.xpath('./descendant::text()').extract())
    if "Trailer" in desc_str or "Film Festival" in desc_str or "Comment" in desc_str:
        break

    # Extract the name to get the type of the node.
    node_type = sibling.xpath('name()').extract_first()
    if node_type == 'ul':
        cast_list = sibling.xpath('./ul[1]/li')
        self.parse_list_into_cast_table(cast_list, _id, conn)
    elif node_type == 'table':
        cast_tables = sibling.xpath('./table')
        self.parse_table_into_cast_table(cast_tables, _id, conn)
    else:
        print "Unknown cast element type! " + _id

In the above code, I keep going through the sibling nodes until I hit an element which contains "Trailer" "Film Festival" or "Comments" somewhere in the text.

Otherwise, for each sibling node, I first get its name to find out what kind of node it is - I'm interested in tables and unordered lists.

However, the part that fails is when I call sibling.xpath('./table') and/or sibling.xpath('ul[1]/li').

The resulting object is always an empty list.

I have used scrapy shell and examined the list of selectors that I get from cast_siblings. Each selector seems valid and when I extract them, I can see the inner elements. I just can't get to them using xpath.

Here's an excerpt from my console showing the empty list result:

>>> a = response.xpath('//h2/span[starts-with(@id,"Cas")]/../following-sibling::*')
>>> a
[<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Maruy'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Azumi'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Prese'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p><b>Additional Cast Members:</b>\n</p>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<ul><li> <a href="/Yukari_Ito" title="Yu'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Trail'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<p>\n<script type="text/javascript" src="'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="mediaplayer"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<script type="text/javascript">\n  jwplay'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h2> <span class="mw-headline" id="Comme'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<br>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div id="comment-outer"><hr><div id="com'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<div align="center"></div>'>, <Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table cellpadding="2" style="border:0px'>]
>>> a[0]
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<h3> <span class="mw-headline" id="Sudo_'>
>>> a[1]
<Selector xpath='//h2/span[starts-with(@id,"Cas")]/../following-sibling::*' data=u'<table border="0" cellspacing="3" cellpa'>
>>> a[1].xpath('./table')
[]

This is really strange since a[1] in the above example clearly shows a table within.

Upvotes: 1

Views: 720

Answers (1)

Granitosaurus
Granitosaurus

Reputation: 21446

Seems like you are looking for table children of a node that already is a table: 

node_type = sibling.xpath('name()').extract_first()
if node_type == 'ul':
    ...
elif node_type == 'table':
    cast_tables = sibling.xpath('./table') <---

here you already know that the current node is table and you are trying to look for another table underneath it. So in this case it should just be:

cast_tables = sibling

It's also worth noting that when you test scrapy selectors the results include root node tags, i.e. '//h1' will return <h1>...</h1>

Upvotes: 1

Related Questions