CODEWITHSUNDEEP
pythonlxmllibxml2
ykaganovich
ykaganovich

Reputation: 14964

lxml: clean_html replaces html tag with div?

I'm using lxml 3.1.0 (installed with easy_install), and seeing strange result:

> from lxml.html.clean import clean_html
> clean_html("<html><body><h1>hi</h1></body></html>")
'<div><body><h1>hi</h1></body></div>'

the html tag is being replaced with div.

The same happens with the sample html as per http://lxml.de/lxmlhtml.html#cleaning-up-html

What gives? Am I running into a bug with lxml, or version incompatibility with libxml2, or is this somehow expected?

Upvotes: 5

Views: 3706

Answers (2)

crayzeewulf
crayzeewulf

Reputation: 6010

I think you need a Cleaner that leaves page_structure alone:

>>> from lxml.html.clean import Cleaner                                                           
>>> cleaner = Cleaner(page_structure=False)                                          
>>> cleaner.clean_html("<html><body><h1>hi</h1></body></html>")
'<html><body><h1>hi</h1></body></html>'

As described here, page_structure is True by default. I suspect that the documentation at the site you provided is incorrect or outdated.

Edit#1: Another confirmation that this is the expected behavior can be found in this test in the source code. A pull request has been submitted to correct the documentation.

Edit#2: The pull request has been merged into the master as of 2013-04-28.

Upvotes: 5

unutbu
unutbu

Reputation: 879749

Structural parts of the page such as <head>, <html> and <title> are removed if page_structure=True, which is the default. To change this:

import lxml.html.clean as clean
content = '<html><body><h1>hi</h1></body></html>'
cleaner = clean.Cleaner(page_structure=False)
cleaned = cleaner.clean_html(content)
print(cleaned)
# <html><body><h1>hi</h1></body></html>

See the doc string for class clean.Cleaner:

In [105]: clean.Cleaner?
Type:       type
String Form:<class 'lxml.html.clean.Cleaner'>
File:       /usr/lib/python2.7/dist-packages/lxml/html/clean.py
Definition: clean.Cleaner(self, doc)
Docstring:
Instances cleans the document of each of the possible offending
elements.  The cleaning is controlled by attributes; you can
override attributes in a subclass, or set them in the constructor.

``scripts``:
    Removes any ``<script>`` tags.

``javascript``:
    Removes any Javascript, like an ``onclick`` attribute.

``comments``:
    Removes any comments.

``style``:
    Removes any style tags or attributes.

``links``:
    Removes any ``<link>`` tags

``meta``:
    Removes any ``<meta>`` tags

``page_structure``:
    Structural parts of a page: ``<head>``, ``<html>``, ``<title>``.

``processing_instructions``:
    Removes any processing instructions.

``embedded``:
    Removes any embedded objects (flash, iframes)

``frames``:
    Removes any frame-related tags

``forms``:
    Removes any form tags

``annoying_tags``:
    Tags that aren't *wrong*, but are annoying.  ``<blink>`` and ``<marquee>``

``remove_tags``:
    A list of tags to remove.

``allow_tags``:
    A list of tags to include (default include all).

``remove_unknown_tags``:
    Remove any tags that aren't standard parts of HTML.

``safe_attrs_only``:
    If true, only include 'safe' attributes (specifically the list
    from `feedparser
    <http://feedparser.org/docs/html-sanitization.html>`_).

``add_nofollow``:
    If true, then any <a> tags will have ``rel="nofollow"`` added to them.

``host_whitelist``:
    A list or set of hosts that you can use for embedded content
    (for content like ``<object>``, ``<link rel="stylesheet">``, etc).
    You can also implement/override the method
    ``allow_embedded_url(el, url)`` or ``allow_element(el)`` to
    implement more complex rules for what can be embedded.
    Anything that passes this test will be shown, regardless of
    the value of (for instance) ``embedded``.

    Note that this parameter might not work as intended if you do not
    make the links absolute before doing the cleaning.

``whitelist_tags``:
    A set of tags that can be included with ``host_whitelist``.
    The default is ``iframe`` and ``embed``; you may wish to
    include other tags like ``script``, or you may want to
    implement ``allow_embedded_url`` for more control.  Set to None to
    include all tags.

This modifies the document *in place*.
Constructor information:
 Definition:clean.Cleaner(self, **kw)

Upvotes: 3

Related Questions