CODEWITHSUNDEEP
groovyxmlslurperweb-scraping
Stefan Kendall
Stefan Kendall

Reputation: 67892

XmlSlurper never finds node

I'm trying to page scrape some DOM that looks like this:

<span>text</span>

and sometimes looks like this: 

<span><p>text</p></span>

However, I just can't seem to figure out how to get text in the second scenario. I've tried several methods, and here's what I thought should work below:

def html = slurper.parse(reader)
Collection<NodeChild> nodes = html.'**'.findAll { it.name() == 'span' && it.@class == 'style2' }
...
def descriptionNode = html.'**'.find { it.name() == 'span' && it.@class == 'style20' }
def innerNode = descriptionNode.'**'.find { it.name() == 'p' }
def description
if (innerNode?.size() > 0)
{
description = innerNode.text()
}
else
{
description = descriptionNode.text()
}

Any idea how I need to go about using xmlslurper to get the behavior I need?

Upvotes: 0

Views: 1935

Answers (3)

Steven
Steven

Reputation: 3894

have you tried the xpath: //span/text() ? you might need to query twice to account for the p tagged.

Upvotes: 0

Stefan Kendall
Stefan Kendall

Reputation: 67892

As it turns out, the HTML must have been invalid. Tagsoup created

<div>
<span>
</span>
<p></p>
</div>

but Firebug displayed

<div>
<span>
<p></p>
</span>
</div>

What a terrible bug.

Upvotes: 0

ataylor
ataylor

Reputation: 66109

It sounds like you want to check if a given span contains a nested p. You can iterate over the span node's children to check for that case. Example:

def xml = """
<test>
  <span>test1</span>
  <span><p>test2</p></span>
  <other><span>test3</span></other>
  <other><span><p>test4</p></span></other>
</test>
"""

def doc = new XmlSlurper().parseText(xml)
def descriptions = []
doc.'**'.findAll { it.name() == 'span' }.each { node ->
    if (node.children().find { it.name() == 'p' }) {
            descriptions << node.p.text()
    } else {
            descriptions << node.text()
    }
}
assert descriptions == ['test1', 'test2', 'test3', 'test4']

Upvotes: 3

Related Questions