crawling web data using python html error

Question

i want to crawling data using python i tried tried again but it didn't work i can not found code's error i wrote code like this:

import re
import requests
from bs4 import BeautifulSoup

url='http://news.naver.com/main/ranking/read.nhn?mid=etc&sid1=111&rankingType=popular_week&oid=277&aid=0003773756&date=20160622&type=1&rankingSectionId=102&rankingSeq=1'
html=requests.get(url)
#print(html.text)
a=html.text
bs=BeautifulSoup(a,'html.parser')
print(bs)
print(bs.find('span',attrs={"class" : "u_cbox_contents"}))

i want to crawl reply data in news

as you can see, i tried to searing this:

span, class="u_cbox_contents" in bs

but python only say "None"

None

so i check bs using function print(bs)

and i check up bs variable's contents

but there is no span, class="u_cbox_contents"

why this happing?

i really don't know why

please help me

thanks for reading.

Answer 1

Requests will fetch the URL's contents, but will not execute any JavaScript.

I performed the same fetch with cURL, and I can't find any occurrence of u_cbox_contents in the HTML code. Most likely, it's injected using JavaScript, which explains why BeautifulSoup can't find it.

If you need the page's code as it would be rendered in a "normal" browser, you could try Selenium. Also have a look at this SO question.