CODEWITHSUNDEEP
pythonxmllxml
Richard
Richard

Reputation: 65510

Get all text from an XML document?

How can I get all the text content of an XML document, as a single string - like this Ruby/hpricot example but using Python.

I'd like to replace XML tags with a single whitespace.

Upvotes: 7

Views: 19540

Answers (5)

kirbyfan64sos
kirbyfan64sos

Reputation: 10727

EDIT: This is an answer posted when I thought one-space indentation is normal, and as the comments mention it's not a good answer. Check out the others for some better solutions. This is left here solely for archival reasons, do not follow it!

You asked for lxml:

reslist = list(root.iter())
result = ' '.join([element.text for element in reslist])

Or:

result = ''
for element in root.iter():
    result += element.text + ' '
result = result[:-1] # Remove trailing space

Upvotes: -2

schettino72
schettino72

Reputation: 3170

Using stdlib xml.etree

import xml.etree.ElementTree as ET

tree = ET.parse('sample.xml') 
print(ET.tostring(tree.getroot(), encoding='utf-8', method='text'))

Upvotes: 11

Mark Amery
Mark Amery

Reputation: 154545

This very problem is actually an example in the lxml tutorial, which suggests using one of the following XPath expressions to get all the bits of text content from the document as a list of strings:

  • root.xpath("string()")
  • root.xpath("//text()")

You'll then want to join these bits of text together into a single big string, with str.join probably using str.strip to get rid of leading and trailing whitespace on each bit and ignoring bits that are made entirely of whitespace:

>>> from lxml import etree
>>> root = etree.fromstring("""
... <node>
...   some text
...   <inner_node someattr="someval">   </inner_node>
...   <inner_node>
...     foo bar
...   </inner_node>
...   yet more text
...   <inner_node />
...   even more text
... </node>
... """)
>>> bits_of_text = root.xpath('//text()')
>>> print(bits_of_text)  # Note that some bits are whitespace-only
['\n  some text\n  ', '   ', '\n  ', '\n    foo bar\n  ', '\n  yet more text\n  ', '\n  even more text\n']
>>> joined_text = ' '.join(
...     bit.strip() for bit in bits_of_text
...     if bit.strip() != ''
... )
>>> print(joined_text)
some text foo bar yet more text even more text

Note, by the way, that if you don't want to insert spaces between the bits of text you can just do

etree.tostring(root, method='text', encoding='unicode')

And if you're dealing with HTML instead of XML, and are using lxml.html to parse your HTML, you can just call the .text_content() method of your root node to get all the text it contains (although, again, no spaces will be inserted):

>>> import lxml.html
>>> root = lxml.html.document_fromstring('<p>stuff<p>more <br><b>stuff</b>bla')
>>> root.text_content()
'stuffmore stuffbla'

Upvotes: 0

l4mpi
l4mpi

Reputation: 5149

A solution that doesn't require an external library like BeautifulSoup, using the built-in sax parsing framework:

from xml import sax

class MyHandler(sax.handler.ContentHandler):
    def parse(self, filename):
        self.text = []
        sax.parse(filename, self)
        return ''.join(self.text)

    def characters(self, data):
        self.text.append(data)

result = MyHandler().parse("yourfile.xml")

If you need all whitespace intact in the text, also define the ignorableWhitespace method in the handler class in the same way characters is defined.

Upvotes: 2

Prashant Kumar
Prashant Kumar

Reputation: 22509

I really like BeautifulSoup, and would rather not use regex on HTML if we can avoid it.

Adapted from: [this StackOverflow Answer], [BeautifulSoup documentation]

from bs4 import BeautifulSoup
soup = BeautifulSoup(txt)    # txt is simply the a string with your XML file
pageText = soup.findAll(text=True)
print ' '.join(pageText)

Though of course, you can (and should) use BeautifulSoup to navigate the page for what you are looking for.

Upvotes: 6

Related Questions