CODEWITHSUNDEEP
pythonxmlxpathbeautifulsouplxml
Dim
Dim

Reputation: 93

How do I get all content between two html tags in Python?

I try to extract all content (tags and text) from one main tag on html page. For example:

`my_html_page = '''
<html>
    <body>
       <div class="post_body">
          <span class="polor">
             <a class="p-color">Some text</a>
             <a class="p-color">another text</a>
          </span>
          <a class="p-color">hello world</a>
          <p id="bold">
              some text inside p
             <ul>
                <li class="list">one li</li>
                <li>second li</li>
             </ul>
         </p>
         some text 2
         <div>
             text inside div
         </div>
         some text 3
      </div>
      <div class="post_body">
          <a>text inside second main div</a>
      </div>
      <div class="post_body">
          <span>third div</span>
      </div>
      <div class="post_body">
          <p>four div</p>
      </div>
      <div class="post">
          other text
      </div>
  </body>
<html>'''`

And I need to get using xpath("(//div[@class="post_body"])[1]"):

`
       <div class="post_body">
          <span class="polor">
             <a class="p-color">Some text</a>
             <a class="p-color">another text</a>
          </span>
          <a class="p-color">hello world</a>
          <p id="bold">
              some text inside p
             <ul>
                <li class="list">one li</li>
                <li>second li</li>
             </ul>
         </p>
         some text 2
         <div>
             text inside div
         </div>
         some text 3
      </div>
`

All inside tag <div class="post_body">

I read this topic, but it did not help.

I need to create DOM by beautifulsoup parser in lxml.

 import lxml.html.soupparser
 import lxml.html
 text_inside_tag = lxml.html.soupparser.fromstring(my_html_page)
 text = text_inside_tag.xpath('(//div[@class="post_body"])[1]/text()')

And i can extract only text inside tag, but I need extract text with tags.

If i tried use this:

for elem in text.xpath("(//div[@class="post_body"])[1]/text()"):
   print lxml.html.tostring(elem, pretty_print=True)

I have error: TypeError: Type '_ElementStringResult' cannot be serialized.

Help, please.

Upvotes: 2

Views: 1794

Answers (1)

har07
har07

Reputation: 89285

You can try this way :

import lxml.html.soupparser
import lxml.html

my_html_page = '''...some html markup here...'''
root = lxml.html.soupparser.fromstring(my_html_page)

for elem in root.xpath("//div[@class='post_body']"):
    result = elem.text + ''.join(lxml.html.tostring(e, pretty_print=True) for e in elem)
    print result

result variable constructed by combining text nodes within parent <div> with markup of all of the child nodes.

Upvotes: 2

Related Questions