CODEWITHSUNDEEP
pythonseleniumpyqt4beautifulsoupscreen-scraping
eamon1234
eamon1234

Reputation: 1585

Scraping Ajax using Python

I am trying to get the data in the table at this website which is updated via jquery after the page loads (I have permission) :

http://whichchart.com/

I currently use selenium and beautifulsoup to get data, however because this data is not visible in the html source, I can't access it. I have tried PyQt4 but it likewise does not get the updated html source.

The values are visible in firebug and chrome developer, so are there any python packages out there which can exploit this and feed it to beautifulsoup?

I'm not a massive techie so ideally I would like a solution which would work in Python or the next easiest software type.

I'm aware I can get it via proprietary "screen-scraper" software, but that is expensive.

Upvotes: 3

Views: 2875

Answers (1)

Maksym Kozlenko
Maksym Kozlenko

Reputation: 10363

Page is making AJAX call to get a data to http://whichchart.com/service.php?action=NewcastleCoal which returns values in JSON. So you can do the following:

  • Use urllib to get data using HTTP
  • Parse that data with json library reads method
  • Now you have a python object to process

If you need to process HTML page content I would suggest to use library like BeautifulSoup or scrapy

Upvotes: 5

Related Questions