CODEWITHSUNDEEP
pythonbeautifulsoupweb-crawler
Mudassir
Mudassir

Reputation: 806

Python: BeautifulSoup returning garbage

I am building a basic data crawler in python using BeautifulSoup, for Batoto, the manga host. For some the reason, the URL works sometimes and other times it doesn't. For example:

from bs4 import BeautifulSoup
from urllib2 import urlopen

x= urlopen(*manga url here*)
y = BeautifulSoup(x)

print y

The result should be a tag soup of the page but instead I get a big wall of this

´ºŸ{›æP™oRhtüs2å÷%ëmßñ6Y›þ�GDŸ0Ë­͇켮Yé)–ÀØÅð&amp;ô]½f³ÓÞ€Þþ)ú$÷á�üv…úzW¿¾úà†lªÀí¥ï«·_    OTL_ˆêsÁÿƒÁÖ&lt;Ø?°Þ›Â+WLç¥àEh&gt;rýÜ&gt;x    ˆ‡eÇžù»èå»–Ùý e:›§`L_.‹¦úoÓ‘®e=‰ìÓ4Wëo’]~Ãõ¬À8&gt;x:²âœ2¸ Á|&amp;0ÍVpMLÎñ»v¥Ín÷-ÅÉ–T§`Ì.SÔsóë„œ¡×[˜·P6»�ùè�&gt;Ô¾È]Œ—·ú£âÊgí%ضkwýÃ=Üϸ2cïÑfÙ_�×]Õê“ž?„UÖ* m³/­`ñ§ÿL0³dµ·jªÅ}õ/õOXß×;«]®’ϯw‹·þ¡ÿ|Gýª`I{µœ}œí�ë–¼yÖÇ'�Wç�ëµÅþþ*ýœd{ÿDv:РíHzqÿÆ­÷æélG-èÈâpÇßQé´^ÐO´®Xÿ�ýö(‹šëñþ"4!SÃõ2{òÿÜ´»ûE</kî?x´&ý˜`Ù)uÂï¹ã[ÏŠ²y°kÆpù}¢></uŒ¸kpž¼cì∬ƒcubÆ¡¢=en2‚påÓb9®`áï|z…p"i6pvif¨þõ“⟒></t`$ò-e></cé”r)$�ˆ)ìªÜrd&mÉÊ*ßdÒuÄ.Æ-hx@9[s=m�Ýfd2o1ˆ]‡[Ôádœtë¤qâxæ°‹qËÁ×,½ŠmʇꇢùÅýl></sí°çù¡h?‡ÌÜœbá‰æÆý¡sd~¬></zz¡ózwÎ[à!n‰Àš5¤…¸‘ݹŽ></sÃ:›3Ìæ></lÑggu�».Б@4õë\ÃñÆ:¸5ÔwÛ·…)~ÛacÑ,d­³båÖ6></tg9y+wΉí%r8ƒ·}n`¼ÁÆ8˜”é²êÞ½°¶Ï></sÖ-di¨a±j9³4></ss„*w(ßibðïj*¶„)pâýÌ”a§%va{‰ò¦m mi></o³o˜Ÿ?¿Ñu-}{cÜ›a~:k²Ì></r+=ÅÌk˜c></wÓ¹âߊž‡ëf7vÑ�akÆ4ƒ‚></szŽµiÞêzâšÒ¬ú¢“âÀ#�-></qebndΑg*cxgsÆ€Ùüe¡³-ŠngÁ:�3ænæ5ï0`coäÏÖ9œ1Ða¯,æ—ªìàãÉÂð></j›h¶`à;)òiÖ š+></o”64ˆÎº9°��u—Úd¿ý¥pÎÖ‰0¢s:c�yƧ³t=ÕŸ“Ý‹41%}*,e³Ô¥ó></hiræe—';></v�fÞ«Ë¥n§Ð·¡kaììë\�`ùsõ©¸pv¦‘></bñ¼ut«w)Ø'¹ú@{)n0¡Žan¶Ë5èsª�–u–></y_x.mÅd:g}ëÕðhçð«õõ8ŠcËÕÌvž­v™-šêÙ`b¹˜ùÃΓçˤÔÙtx¹�ßïǶÎgþ°r‹$ò†aÆ–š?ì<y«Ëñõo{%ׇo{ú¥Á»æ]‡></u´¬Ø¸eÖïÝtßÚ'è3®nh±ûk4È@l«s]–Åec¹ÑtmÓl|ë£Þ¼~zôéõûwêÓÑñÉÆw\soøÊiyjvØÖ$¯ÈoºÙoyã]æ5]-t^[“¡aÑ{²Å¸6¦ðtŒçm¼ÂÎz´></wà™´»äõ@©õ></mÏu:=¼þ·'�qwúËö«m„l^ˆær¥30q±ÒšŸëù></l(„7¼=xi’?¤;ö$ØË4ßoóiòyoµxÉøþ¨—«g³Ãíß{|></body></html>

wrapped in html and body tags.

Sometimes I will keep trying and it works, but it is so inconsistent, I can't figure out the reason for it.

Any help would be appreciated.

Upvotes: 2

Views: 689

Answers (2)

unutbu
unutbu

Reputation: 880189

The server is returning gzipped bytes. So to download the content using urllib2:

import sys
import urllib2
import gzip
import io
url = "http://bato.to/comic/_/comics/rakudai-kishi-no-eiyuutan-r11615"
response = urllib2.urlopen(url)

# print(response.headers)
content = response.read()
if response.headers['Content-Encoding'] == 'gzip':
    g = gzip.GzipFile(fileobj=io.BytesIO(content))
    content = g.read()

encoding = response.info().getparam('charset')
content = content.decode(encoding)

This checks the content is the same as the page.text returned by requests:

import requests
page = requests.get(url)
# print(page.headers)

assert content == page.text

Since requests handles the gunzipping and decoding for you -- and more robustly too -- using requests is highly recommended.

Upvotes: 1

Padraic Cunningham
Padraic Cunningham

Reputation: 180461

It seems to be urlopen having issues with encoding, requests works fine:

x = requests.get("http://bato.to/comic/_/comics/rakudai-kishi-no-eiyuutan-r11615")
y = BeautifulSoup(x.content)    
print y


<!DOCTYPE html>
<html lang="en" xmlns:fb="http://www.facebook.com/2008/fbml">
<head>
<meta charset="utf-8"/>
<title>Rakudai Kishi no Eiyuutan - Scanlations - Comic - Comic Directory - Batoto -    Batoto</title>
.................

Using urlopen we get the following:

x = urlopen("http://bato.to/comic/_/comics/rakudai-kishi-no-eiyuutan-r11615")    
print x.read()


���������s+I���2���l��9C<�� ^�����쾯�dw�xzNT%��,T��A^�ݫ���9��a��E�C���W!�����ڡϳ��f7���s2�Px$���}I�*�'��;'3O>���'g?�u®{����e.�ڇ�e{�u���jf:aث
�����DS��%��X�Zͮ���������9�:�Dx�����\-�
�*tBW������t�I���GQ�=�c��\:����u���S�V(�><y�C��ã�*:�ۜ?D��a�g�o�sPD�m�"�,�Ɲ<;v[��s���=��V2�fX��ì�Cj̇�В~�
-~����+;V���m�|kv���:V!�hP��D�K�/`oԣ|�k�5���B�{�0�wa�-���iS
�>�œ��gǿ�o�OE3jçCV<`���Q!��5�B��N��Ynd����?~��q���� _G����;T�S'�@΀��t��Ha�.;J�61'`Й�@���>>`��Z�ˠ�x�@� J*u��'���-����]p�9{>����������#�<-~�K"[AQh0HjP
0^��R�]�{N@��
 ...................

So as you can see it is a problem with urlopen not BeautifulSoup.

Upvotes: 5

Related Questions