CODEWITHSUNDEEP
pythonweb-scrapingbeautifulsoup
Ahmad
Ahmad

Reputation: 254

beautifulsoup4 doesn't return content

Hi, I followed and understood this article on how to read content from sites and it worked perfectly: geeksforgeeks.org:Reading selected webpage content using Python Web Scraping

But when I changed my code to work with another site it doesn't return any value. I am trying to get those Value1 and Value2 etc.. as shown below.

Please Note: It's Legal to read content from that web page.

import requests 
from bs4 import BeautifulSoup 

# the target we want to open     
url='https://hackerone.com/directory?offers_bounties=true&asset_type=URL&order_direction=DESC&order_field=started_accepting_at'

#open with GET method 
resp=requests.get(url) 

#http_respone 200 means OK status 
if resp.status_code==200: 
    print("Successfully opened the web page") 
    print("The news are as follow :-\n") 

    # we need a parser,Python built-in HTML parser is enough . 
    soup=BeautifulSoup(resp.text,'html.parser')     

    # l is the list which contains all the text i.e news  
    l=soup.find("tr","spec-directory-entry daisy-table__row fade fade--show") 

    #now we want to print only the text part of the anchor. 
    #find all the elements of a, i.e anchor 
    for i in l: 
        print(i.text) 
else: 
    print("Error")

Here's the website source code:

<tr class="spec-directory-entry daisy-table__row fade fade--show">
    <a href="/livestream" class="daisy-link spec-profile-name">Value1</a>
<tr class="spec-directory-entry daisy-table__row fade fade--show">
    <a href="/livestream" class="daisy-link spec-profile-name">Value2</a>
<tr class="spec-directory-entry daisy-table__row fade fade--show">
.
.
.

Upvotes: 3

Views: 603

Answers (3)

Khalid Ali
Khalid Ali

Reputation: 1224

JavaScript needed to render the webpage contents. Using prerenderio service is an easy/light way to get the data you're looking for from the page.

import requests 
from bs4 import BeautifulSoup 

# the target we want to open
# changed to use prerenderio service 
url='http://service.prerender.io/https://hackerone.com/directory?offers_bounties=true&asset_type=URL&order_direction=DESC&order_field=started_accepting_at'

#open with GET method 
resp=requests.get(url) 

#http_respone 200 means OK status 
if resp.status_code==200: 
    print("Successfully opened the web page") 
    print("The news are as follow :-\n") 

    # we need a parser,Python built-in HTML parser is enough . 
    soup=BeautifulSoup(resp.text,'html.parser')     

    # l is the list which contains all the text i.e news  
    l=soup.find("tr","spec-directory-entry daisy-table__row fade fade--show") 

    #now we want to print only the text part of the anchor. 
    #find all the elements of a, i.e anchor 
    for i in l: 
        print(i.text) 
else: 
    print("Error")

The returned data from the above code:

Successfully opened the web page
The news are as follow :-

LivestreamManaged
04 / 2019
73
$100
$150-$250

Edited: Responding to Ahmad's comment

Here's the code to only get the values for "Livestream" table row.

import requests 
from bs4 import BeautifulSoup 

# the target we want to open
# changed to use prerenderio service 
url='http://service.prerender.io/https://hackerone.com/directory?offers_bounties=true&asset_type=URL&order_direction=DESC&order_field=started_accepting_at'

#open with GET method 
resp=requests.get(url) 

#http_respone 200 means OK status 
if resp.status_code==200: 
    print("Successfully opened the web page") 
    print("The news are as follow :-\n") 

    # we need a parser,Python built-in HTML parser is enough . 
    soup=BeautifulSoup(resp.text,'html.parser')     

    # l is the list which contains all "tr" tags  
    l=soup.findAll("tr","spec-directory-entry daisy-table__row fade fade--show")

    # looping through the list of table rows
    for i in l:
        # checking if the current row is for 'Livestream'
        if i.find('a').text == 'Livestream':
          # printing the row's values except the first "td" tag
          for e in i.findAll('td')[1:]:
            print(e.text)
else: 
    print("Error")

Result:

Successfully opened the web page
The news are as follow :-

04 / 2019
73
$100
$150-$250

Upvotes: 5

KunduK
KunduK

Reputation: 33384

Looks like JS render to page.You can use both selenium and Beautiful soup to get the value.

from selenium import webdriver
import time
from bs4 import BeautifulSoup

driver=webdriver.Chrome()
driver.get("https://hackerone.com/directory?offers_bounties=true&asset_type=URL&order_direction=DESC&order_field=started_accepting_at")
time.sleep(5)
html=driver.page_source
soup=BeautifulSoup(html,'html.parser')
for a in soup.select("a.spec-profile-name[href='\/livestream']"):
    print(a.text)

Upvotes: 2

jxddk
jxddk

Reputation: 636

Looking at what the request actually fetches, it seems this page relies on dynamic content. The following text is returned in your request:

It looks like your JavaScript is disabled. To use HackerOne, enable JavaScript in your browser and refresh this page.

You get "TypeError: 'NoneType' object is not iterable" because without Javascript, there are no "tr" elements for BeautifulSoup to find and iterate over. You will have to use something like selenium to simulate a browser running Javascript in order to get the HTML you expect.

Upvotes: 0

Related Questions