CODEWITHSUNDEEP
pythonbeautifulsoupfindall
Andrej Licanin
Andrej Licanin

Reputation: 43

BS4 findall not returning all divs

I was trying to get to the bottom table in the site,but findall() kept returning empty objects so i got all the divs on the same level one by one and noticed that when i try to get the last two it gives me the []

the_page=urllib.request.urlopen("https://theunderminejournal.com/#eu/sylvanas/item/124105")
bsObj=BeautifulSoup(the_page,'html.parser')
test=bsObj.findAll('div',{'class':'page','id':"item-page"})
print(test)

I have gone through the bs4 object that i got and the 2 divs im looking for arent in it.Whats happening?

the div im looking for is in the https://theunderminejournal.com/#eu/sylvanas/item/124105

this is the div im trying to extract

Upvotes: 1

Views: 1226

Answers (1)

Ali
Ali

Reputation: 1357

You will need to use selenium instead of the normal requests libraries.

Note that I couldn't post all of the output as the HTML parsed was huge.

Code:

from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://theunderminejournal.com/#eu/sylvanas/item/124105")
bsObj = BeautifulSoup(driver.page_source,'html.parser')
test = bsObj.find('div', id='item-page')
print(test.prettify())

Output:

<div class="page" id="item-page" style="display: block;">
 <div class="item-stats">
  <table>
   <tr class="available">
    <th>
     Available Quantity
    </th>
    <td>
     <span>
      30,545
     </span>
    </td>
   </tr>
   <tr class="spacer">
    <td colspan="3">
    </td>
   </tr>
   <tr class="current-price">
    <th>
     Current Price
    </th>
    <td>
     <span class="money-gold">
      27.34
     </span>
    </td>
   </tr>
   <tr class="median-price">
    <th>
     Median Price
    </th>
    <td>
     <span class="money-gold">
      30.11
     </span>
    </td>
   </tr>
   <tr class="mean-price">
    <th>
     Mean Price
    </th>
    <td>
     <span class="money-gold">
      30.52
     </span>
    </td>
   </tr>
   <tr class="standard-deviation">
    <th>
     Standard Deviation
    </th>
    <td>
     <span class="money-gold">
      .
      .
      .
       </span>
      </abbr>
     </td>
    </tr>
   </table>
  </div>
 </div>
</div>

Upvotes: 4

Related Questions