CODEWITHSUNDEEP
pythontext-processingnltk
rushter
rushter

Reputation: 21

Python html processing

I have a html file with russian text. How i can get all words in text without html tags, special symbols, etc ?

Example: 

<html>...<body>...<div id='text'>Foo bar! Foo, bar.</div></body></html>

I need:

['foo','bar','Foo','bar']

I tried nltk, but it does not support russian words.

Upvotes: 2

Views: 690

Answers (4)

Matt Alcock
Matt Alcock

Reputation: 12891

Use regex to remove the tags. Nltk is all about language analysis (nouns vs verbs) and word meaning (semantics) not string removal and pattern matching although I can see how someoneaybe confused.

Here is a removal function using regex

import re
def remove_html_tags(data):
    p = re.compile(r'<.*?>')
    return p.sub('', data)

Upvotes: 0

Misha Akovantsev
Misha Akovantsev

Reputation: 1825

Use lxml. It can strip tags, elements, and more:

import urllib2

from lxml import etree


URL = 'http://stackoverflow.com/questions/9230675/python-html-processing'

html = urllib2.urlopen(URL).read()
tree = etree.fromstring(html, parser=etree.HTMLParser())

tree.xpath('//script')
# [<Element script at 102f831b0>,
#  ...
#  <Element script at 102f83ba8>]

tree.xpath('//style')
# [<Element style at 102f83c58>]

tags_to_strip = ['script', 'style']
etree.strip_elements(tree, *tags_to_strip)

tree.xpath('//style')
# []

tree.xpath('//script')
# []

body = tree.xpath('//body')
body = body[0]

text = ' '.join(body.itertext())
tokens = text.split()
# [u'Stack',
#  u'Exchange',
#  u'log',
#  u'in',
#  ...
#  u'Stack',
#  u'Overflow',
#  u'works',
#  u'best',
#  u'with',
#  u'JavaScript',
#  u'enabled']

In case of text in russian you get tokens looking likes this:

# [u'\xd1\x8d\xd1\x84\xd1\x84\xd0\xb5\xd0\xba\xd1\x82\xd1\x8b\xe2\x80\xa6',
#  u'\xd0\x9c\xd0\xb0\xd1\x80\xd0\xba',
#  ...
#  u'\xd0\x9c\xd0\xb0\xd0\xb9\xd0\xb5\xd1\x80']

Errors handling is your home assignment.

Upvotes: 0

Ivan Kolodyazhny
Ivan Kolodyazhny

Reputation: 574

I'm using lxml library to parse xml/html. lxml works good with any unicode data.

Upvotes: 4

UncleZeiv
UncleZeiv

Reputation: 18488

Definitely try BeautifulSoup, it supports Unicode.

Upvotes: 4

Related Questions