CODEWITHSUNDEEP
python
Caden
Caden

Reputation: 3

Grabbing Nested Divs using BeautifulSoup4 (Python 2)

First off, it should be noted that I am using PyCharm 2016.2.3, and it's python 2.7

What my goal is: I am trying to grab the "Buy Price", located under the "Info" section. It is the first row down. Here is the link to the page I am trying to grab the Buy Price from. https://rsbuddy.com/exchange?id=1079&

If you view the source code from inspect element or view page source, you will see that buy price is a string located in this html line of code: 

    <div id="buy-price" class="col-md-7">41,060 gp</div>

The '41,060 gp' is what I am trying to grab, just to make that clear.

But if you look at where this div is nested, it is in between TONS of different divs before it. Divs with classes, id's, and both. When I say tons, I mean probably about 15+.

Now that you have a clear understanding of what I need to grab, I will show you my code right now.

import requests
from bs4 import BeautifulSoup

def items(max_pages):
    page = 1
    while page <= max_pages:
        url = "https://rsbuddy.com/exchange?id=1079&"
        source_code = requests.get(url)
        plain_text = source_code.text
        soup = BeautifulSoup(plain_text, "html.parser")
        for buy_price in soup.findAll('div', {'id': 'buy-price'}):
            item = buy_price.string
            print(item)    
        page += 1

items(1)

When I run this code, nothing happens. I don't get any errors, nothing. Just a blank screen with "Process Finished with exit code 0"

I'm not sure what exactly is causing the issue. Maybe I can grabbing the div wrong (It has worked for me on another situation), maybe I am making a dumb small mistake that I don't realize. I just don't get it at this point. I've spent hours trying to figure it out, researching online... Hopefully someone can help.

Note that I didn't specify the html code because it's simply way to much to put in here. Please just check out the website and look at the source code of it. Just hover your mouse over "Buy Price" under the info section, and right click and hit source code.

Upvotes: 0

Views: 36

Answers (1)

宏杰李
宏杰李

Reputation: 12158

This page is rendered by JavaScript, you can disable your browser's JavaScript and visit it again, and there is no data there. Requests can not handle the JavaScript part.

I recommend using Selenium to handle JavaScript.

your page without javascript

Upvotes: 2

Related Questions