CODEWITHSUNDEEP
pythonmacos
andy
andy

Reputation: 2161

url with non english characters

I want to crawl one list from a Chinese web, my program is:

import pandas as pd 
states = pd.read_html('http://baike.baidu.com/item/天津/132308',encoding='utf-8')
print(states[0])

because there are non English words"天津", there are some errors:

Traceback (most recent call last):
  File "/Users/biyuntian/Documents/nihao.py", line 2, in <module>
    fiddy_states = pd.read_html('http://baike.baidu.com/item/天津/132308',encoding='utf-8')
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/html.py", line 906, in read_html
    keep_default_na=keep_default_na)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/io/html.py", line 743, in _parse
    raise_with_traceback(retained)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pandas/compat/__init__.py", line 344, in raise_with_traceback
    raise exc.with_traceback(traceback)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 10-11: ordinal not in range(128)

how to solve this problem? by the way, I'm using python 3 on macbook air

Upvotes: 1

Views: 480

Answers (1)

Azat Ibrakov
Azat Ibrakov

Reputation: 10963

try to pass raw HTML obtained with other tool like requests library

import pandas as pd
import requests

response = requests.get('http://baike.baidu.com/item/天津/132308')
content = response.content
states = pd.read_html(content,
                      encoding='utf-8')
print(states[0])

gives us

       0         1       2               3
0   区划名称        面积    邮政编码            政府驻地
1    和平区    10平方千米  300041     小白楼街道曲阜道81号
2    河东区    39平方千米  300171    上杭路街道泰兴南路32号
3    河西区    37平方千米  300202      大营门街道绍兴道4号
4    南开区    39平方千米  300100     长虹街道黄河道390号
5    河北区    27平方千米  300143  望海楼街道狮子林大街284号
6    红桥区    21平方千米  300131    西于庄街道勤俭道202号
7   滨海新区  2270平方千米  300457    塘沽街道新港二号路35号
8    东丽区   460平方千米  300300     张贵庄街道跃进路38号
9    西青区   545平方千米  300380       杨柳青镇府前街2号
10   津南区   401平方千米  300350     咸水沽镇津沽路186号
11   北辰区   478平方千米  300400   果园新村街道北辰道389号
12   武清区  1570平方千米  301700   运河西街道雍阳西道118号
13   宝坻区  1523平方千米  301800     宝平街道建设路116号
14   宁河区  1414平方千米  301500       芦台镇光明路76号
15   静海区  1476平方千米  301600      静海镇迎宾大道99号
16   蓟州区  1593平方千米  301900       文昌街道府前街2号

EDIT

if you don't wanna use third-party requests library we can use function quote from standard library's urllib.parse package to escape non-ascii characters like

from urllib.parse import quote

import pandas as pd

states = pd.read_html(quote('http://baike.baidu.com/item/天津/132308', safe=':/'),
                      encoding='utf-8')
print(states[0])

Upvotes: 4

Related Questions