CODEWITHSUNDEEP
pythonhtmlregex
Felipe Andrade
Felipe Andrade

Reputation: 61

Python regular expression for multiple tags

I would like to know how to retrieve all results from each <p> tag.

import re
htmlText = '<p data="5" size="4">item1</p><p size="4">item2</p><p size="4">item3</p>'
print re.match('<p[^>]*size="[0-9]">(.*?)</p>', htmlText).groups()

result:

('item1', )

what I need:

('item1', 'item2', 'item3')

Upvotes: 3

Views: 2860

Answers (5)

RichieHindle
RichieHindle

Reputation: 281835

You can use re.findall like this:

import re
html = '<p data="5" size="4">item1</p><p size="4">item2</p><p size="4">item3</p>'
print re.findall('<p[^>]*size="[0-9]">(.*?)</p>', html)
# This prints: ['item1', 'item2', 'item3']

Edit: ...but as the many commenters have pointed out, using regular expressions to parse HTML is usually a bad idea.

Upvotes: 2

Kenan Banks
Kenan Banks

Reputation: 212178

The regex answer is extremely fragile. Here's proof (and a working BeautifulSoup example).

from BeautifulSoup import BeautifulSoup

# Here's your HTML
html = '<p data="5" size="4">item1</p><p size="4">item2</p><p size="4">item3</p>'

# Here's some simple HTML that breaks your accepted 
# answer, but doesn't break BeautifulSoup.
# For each example, the regex will ignore the first <p> tag.
html2 = '<p size="4" data="5">item1</p><p size="4">item2</p><p size="4">item3</p>'
html3 = '<p data="5" size="4" >item1</p><p size="4">item2</p><p size="4">item3</p>'
html4 = '<p data="5" size="12">item1</p><p size="4">item2</p><p size="4">item3</p>'

# This BeautifulSoup code works for all the examples.
paragraphs = BeautifulSoup(html).findAll('p')
items = [''.join(p.findAll(text=True)) for p in paragraphs]

Use BeautifulSoup.

Upvotes: 5

Brett Bim
Brett Bim

Reputation: 3318

Beautiful soup is definitely the way to go with a problem like this. The code is cleaner and easier to read. Once you have it installed, getting all the tags looks something like this.

from BeautifulSoup import BeautifulSoup
import urllib2

def getTags(tag):
  f = urllib2.urlopen("http://cnn.com")
  soup = BeautifulSoup(f.read())
  return soup.findAll(tag)


if __name__ == '__main__':
  tags = getTags('p')
  for tag in tags: print(tag.contents)

This will print out all the values of the p tags.

Upvotes: 5

Stephan202
Stephan202

Reputation: 61609

Alternatively, xml.dom.minidom will parse your HTML if,

  • ...it is wellformed
  • ...you embed it in a single root element.

E.g.,

>>> import xml.dom.minidom
>>> htmlText = '<p data="5" size="4">item1</p><p size="4">item2</p><p size="4">item3</p>'
>>> d = xml.dom.minidom.parseString('<not_p>%s</not_p>' % htmlText)
>>> tuple(map(lambda e: e.firstChild.wholeText, d.firstChild.childNodes))
('item1', 'item2', 'item3')

Upvotes: 2

Peter Boughton
Peter Boughton

Reputation: 112230

For this type of problem, it is recommended to use a DOM parser, not regex.

I've seen Beautiful Soup frequently recommended for Python

Upvotes: 11

Related Questions