CODEWITHSUNDEEP
pythonpython-2.7xpathweb-scrapingscrapy
waylonion
waylonion

Reputation: 6976

Scrapy get type/info about immediate next sibling

I am extracting information from a site with html that looks like:

<h2> <span class="mw-headline" id="Cast">Cast</span></h2>
<ul><li> <a href="/Shota_Sometani" title="Shota Sometani">Shota Sometani</a> - Shinichi Izumi
</li><li> <a href="/Ai_Hashimoto" title="Ai Hashimoto">Ai Hashimoto</a> - Satomi Murano
</li><li> <a href="/Sadao_Abe" title="Sadao Abe">Sadao Abe</a> - Migi
</li><li> <a href="/Eri_Fukatsu" title="Eri Fukatsu">Eri Fukatsu</a> - Teacher Ryoko Tamiya
</li><li> <a href="/Masahiro_Higashide" title="Masahiro Higashide">Masahiro Higashide</a> - Hideo Shimada (mysterious transfer student)
</li><li> <a href="/Nao_Omori" title="Nao Omori">Nao Omori</a> - Kuramori (freelance journalist)
</li><li> <a href="/Pierre_Taki" title="Pierre Taki">Pierre Taki</a> - Miki
</li><li> <a href="/Hirofumi_Arai" title="Hirofumi Arai">Hirofumi Arai</a> - Uragami (serial killer)
</li><li> <a href="/Kazuki_Kitamura" title="Kazuki Kitamura">Kazuki Kitamura</a> - Takeshi Hirokawa (mysterious mayor candidate)
</li><li> <a href="/Tadanobu_Asano" title="Tadanobu Asano">Tadanobu Asano</a> - Goto (Takeshi Hirokawa’s bodyguard)
</li><li> <a href="/Kimiko_Yo" title="Kimiko Yo">Kimiko Yo</a> - Nobuko Izumi (Shinichi Izumi’s mother)
</li><li> <a href="/Jun_Kunimura" title="Jun Kunimura">Jun Kunimura</a> - Detective Hirama
</li><li> <a href="/Wataru_Ichinose_(1985)" title="Wataru Ichinose (1985)">Wataru Ichinose</a> - gang member
</li><li> <a href="/Roy_Kishima" title="Roy Kishima">Roy Kishima</a> - Roy Kishima
</li></ul>

alternatively, the html could also look like this:

<h2> <span class="mw-headline" id="Cast">Cast</span></h2>
<table border="0" cellspacing="3" cellpadding="0" style="text-align:center">
<tr>
<td width="100pt">
</td>
<td width="100pt">
</td>
<td width="100pt">
</td>
<td width="100pt">
</td>
<td width="100pt">
</td></tr>
<tr>
<td> <a href="/Ryosuke_Yamada" title="Ryosuke Yamada"><img alt="Assassination Classroom-Graduation-Ryosuke Yamada.jpg" src="/images/1/12/Assassination_Classroom-Graduation-Ryosuke_Yamada.jpg" width="90" height="83"/></a>
</td>
<td> <a href="/Masaki_Suda" title="Masaki Suda"><img alt="Assassination Classroom-Graduation-Masaki Suda.jpg" src="/images/4/45/Assassination_Classroom-Graduation-Masaki_Suda.jpg" width="90" height="90"/></a>
</td>
<td> <a href="/Maika_Yamamoto" title="Maika Yamamoto"><img alt="Assassination Classroom-Graduation-Maika Yamamoto.jpg" src="/images/e/e7/Assassination_Classroom-Graduation-Maika_Yamamoto.jpg" width="90" height="90"/></a>
</td>
<td> <a href="/Seika_Taketomi" title="Seika Taketomi"><img alt="Assassination Classroom-Graduation-Seika Taketomi.jpg" src="/images/c/ce/Assassination_Classroom-Graduation-Seika_Taketomi.jpg" width="90" height="90"/></a>
</td>
<td> <a href="/Mio_Yuki" title="Mio Yuki"><img alt="Assassination Classroom-Graduation-Mio Yuki.jpg" src="/images/b/b4/Assassination_Classroom-Graduation-Mio_Yuki.jpg" width="90" height="90"/></a>
</td></tr>
...
</table>

I DO NOT know the type of element that immediately follows h2 as a sibling node. In most cases, the element that follows the h2 element should be a table and this is what I want to check for. For this example, I need to detect that the element following h2 is not a table. To do so, I use

response.xpath('//h2/span[@id="Cast"]/../following-sibling::table')

to grab the following table. I would normally expect an empty list. However, this isn't working since there does happen to be a table, but it's many elements further down with completely unrelated content.

Is there a way to obtain information about the following-sibling that immediately follows a node?

Upvotes: 0

Views: 1088

Answers (2)

Andersson
Andersson

Reputation: 52685

If you want to get type (name) of the immediate following sibling node, you can try

//h2/following-sibling::*[1]/name()

Upvotes: 1

eLRuLL
eLRuLL

Reputation: 18799

just to get the next sibling it would be something like:

(//h2/span[@id="Cast"]/../following-sibling::table)[1]

Upvotes: 1

Related Questions