CODEWITHSUNDEEP
javascriptpython
Hafiz Muhammad Shafiq
Hafiz Muhammad Shafiq

Reputation: 8670

How to scrape javascript webpage using python standard libs only

I have to scrape a website that uses javascript to display content. I have to use standard libs only as I will run this script on a server where there is not any browser. I have found selenium but it requires a browser that in my case is not possible to install.

Any idea or solution?

Upvotes: 1

Views: 49

Answers (2)

narko
narko

Reputation: 3865

You didn't mention anything about how the website is using javascript, but if it uses AJAX requests that are triggered after any kind of user interaction, you will need to use something like Selenium to automatize that behaviour. Here, you can find a short tutorial of how to scrape with Scrapy + Selenium. This of course requires a browser previously installed in your machine.

Upvotes: 1

ram hemasri
ram hemasri

Reputation: 1634

Have a look at Ghost.py http://jeanphix.me/Ghost.py/. It doesn't require a browser.

pip install Ghost.py

from ghost import Ghost
ghost = Ghost()
page, resources = ghost.open('http://stackoverflow.com/')

Upvotes: 2

Related Questions