CODEWITHSUNDEEP
python-3.xurllib2
Tribhuvanesh
Tribhuvanesh

Reputation: 13

Extracting source code from html file using python3.1 urllib.request

I'm trying to obtain data using regular expressions from a html file, by implementing the following code:

import urllib.request
def extract_words(wdict, urlname):
  uf = urllib.request.urlopen(urlname)
  text = uf.read()
  print (text)
  match = re.findall("<tr>\s*<td>([\w\s.;'(),-/]+)</td>\s+<td>([\w\s.,;'()-/]+)</td>\s*</tr>", text)

which returns an error:

File "extract.py", line 33, in extract_words
match = re.findall("<tr>\s*<td>([\w\s.;'(),-/]+)</td>\s+<td>([\w\s.,;'()-/]+)</td>\s*</tr>", text)
File "/usr/lib/python3.1/re.py", line 192, in findall
return _compile(pattern, flags).findall(string)
TypeError: can't use a string pattern on a bytes-like object

Upon experimenting further in the IDLE, I noticed that the uf.read() indeed returns the html source code the first time I invoke it. But then onwards, it returns a - b''. Is there any way to get around this?

Upvotes: 1

Views: 3535

Answers (1)

Lennart Regebro
Lennart Regebro

Reputation: 172309

uf.read() will only read the contents once. Then you have to close it and reopen it to read it again. This is true for any kind of stream. This is however not the problem.

The problem is that reading from any kind of binary source, such as a file or a webpage, will return the data as a bytes type, unless you specify an encoding. But your regexp is not specified as a bytes type, it's specified as a unicode str.

The re module will quite reasonably refuse to use unicode patterns on byte data, and the other way around.

The solution is to make the regexp pattern a bytes string, which you do by putting a b in front of it. Hence:

match = re.findall(b"<tr>\s*<td>([\w\s.;'(),-/]+)</td>\s+<td>([\w\s.,;'()-/]+)</td>\s*</tr>", text)

Should work. Another option is to decode the text so it also is a unicode str:

encoding = uf.headers.getparam('charset')
text = text.decode(encoding)
match = re.findall("<tr>\s*<td>([\w\s.;'(),-/]+)</td>\s+<td>([\w\s.,;'()-/]+)</td>\s*</tr>", text)

(Also, to extract data from HTML, I would say that lxml is a better option).

Upvotes: 2

Related Questions