CODEWITHSUNDEEP
javahtmldomxpath
dapperwaterbuffalo
dapperwaterbuffalo

Reputation: 2748

parsing HTML file to DOM Tree for extract (Java)

so I am trying to parse a HTML file into the DOM Tree and extract nodes via an XPath expression.

I can successfully parse the HTML into the DOM Tree, however when I try to extract Nodes via XPath I am getting nothing out.

Please note this is only a code snippet for relevance.

import org.cyberneko.html.parsers.DOMParser;
import org.dom4j.Document;
import org.dom4j.Node;
import org.dom4j.io.DOMReader;
import org.xml.sax.InputSource;

DOMParser parser = new DOMParser();

parser.parse(new InputSource("file:///Z:/homepage.htm"));
org.w3c.dom.Document doc = parser.getDocument();

DOMReader reader = new DOMReader();
Document document = reader.read(doc);

@SuppressWarnings("unchecked")
List<Node> nodes = document.selectNodes("//HEAD/LINK");

nodes = 0.

For completeness, here is a snippet of the HTML:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
                      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<HTML xmlns="http://www.w3.org/1999/xhtml">
    <HEAD>
        <META content="text/html;charset=UTF-8" http-equiv="Content-Type"/>
        <TITLE/>
        <LINK
            href="wcm/groups/visual/documents/webasset/####_ie_5_css.css"
            media="all" rel="stylesheet" type="text/css"/>
        <LINK
            href="wcm/groups/visual/documents/webasset/####_ie_5_5000_css.css"
            media="all" rel="stylesheet" type="text/css"/>
        <LINK
            href="wcm/groups/visual/documents/webasset/####_ie_6_css.css"
            media="all" rel="stylesheet" type="text/css"/>

Many thanks as always,

Joe

Upvotes: 2

Views: 10174

Answers (2)

reevesy
reevesy

Reputation: 3472

@BrianAgnew is right, your problem is namespace related.

The problem lies here

<HTML xmlns="http://www.w3.org/1999/xhtml">

Since the document has a default namespace xmlns="http://www.w3.org/1999/xhtml" your XPath expression //HEAD/LINK will not work as both the HEAD and LINK elements belong to the default namespace (xmlns="http://www.w3.org/1999/xhtml")

@BrianAgnew suggested using: 

document.selectNodes("//*[local-name()='HEAD']/*[local-name()='LINK']");

For more info on why local-name() works see

XPATHS and Default Namespaces and the answer on the same thread

There is another way of selecting these nodes without having to use the local-name() and that is to create an alias for the default namespace and then use that in your XPath expression:

e.g.

    Map<String, String> namespaceUris = new HashMap<String, String>();  
    namespaceUris.put("foobar", "http://www.w3.org/1999/xhtml");  

    XPath xPath = DocumentHelper.createXPath("//foobar:HEAD/foobar:LINK");  
    xPath.setNamespaceURIs(namespaceUris);  

    @SuppressWarnings("unchecked")
    List<Nodes> selectNodes = xPath.selectNodes(document);

Above we set the alias foobar to be the same URI (http://www.w3.org/1999/xhtml) as the default namespace. This the allows an xpath expression such as

//foobar:HEAD/foobar:LINK to work, of course you can use what ever alias you like.

Here's a sample app that uses both aproaches, its a bit rough but should give you the right idea

package org.foo.bar.foobar;

import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

import nu.xom.Nodes;

import org.cyberneko.html.parsers.DOMParser;
import org.dom4j.Document;
import org.dom4j.DocumentHelper;
import org.dom4j.Node;
import org.dom4j.XPath;
import org.dom4j.io.DOMReader;
import org.dom4j.io.XMLWriter;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;

public class App 
{
    public static void main( String[] args ) throws SAXException, IOException
    {

        DOMParser parser = new DOMParser();

        parser.parse(new InputSource("file:///Z:/homepage.htm"));
        org.w3c.dom.Document doc = parser.getDocument();

        DOMReader reader = new DOMReader();
        Document document = reader.read(doc);

        XMLWriter xmlWriter = new XMLWriter(System.out);
        xmlWriter.write(document);

        @SuppressWarnings("unchecked")

        List<Node> nodes = document.selectNodes("//*[local-name()='HEAD']/*[local-name()='LINK']");
        System.out.println("Number of Nodes: " +nodes.size());

        Map<String, String> namespaceUris = new HashMap<String, String>();  
        namespaceUris.put("foobar", "http://www.w3.org/1999/xhtml");  

        XPath xPath = DocumentHelper.createXPath("//foobar:HEAD/foobar:LINK");  
        xPath.setNamespaceURIs(namespaceUris);  

        @SuppressWarnings("unchecked")
        List<Nodes> selectNodes = xPath.selectNodes(document);
        System.out.println("Number of nodes: " +selectNodes.size());

    }
}

Here's the pom I used for good measure

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.foo.bar</groupId>
    <artifactId>foobar</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>foobar</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>dom4j</groupId>
            <artifactId>dom4j</artifactId>
            <version>1.6.1</version>
        </dependency>
        <dependency>
            <groupId>jaxen</groupId>
            <artifactId>jaxen</artifactId>
            <version>1.1.1</version>
        </dependency>
        <dependency>
            <groupId>nekohtml</groupId>
            <artifactId>nekohtml</artifactId>
            <version>1.9.6.2</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>
</project>

Also see One Fork, How To use Dom4J XPath with XML Namespaces which covers a very similar situation to the one you encountered

Upvotes: 3

Brian Agnew
Brian Agnew

Reputation: 272217

I suspect this is namespace-related.

document.selectNodes("//HEAD/LINK");

should be namespace-aware. e.g.

document.selectNodes("//*[local-name()='HEAD']/*[local-name()='LINK']");

XPath 2.0 will permit

document.selectNodes("//:HEAD/:LINK");

Upvotes: 3

Related Questions