CODEWITHSUNDEEP
javascriptpythonhtmlweb-scrapingbeautifulsoup
Biswarup Dass
Biswarup Dass

Reputation: 213

How to fetch data from a website using Python that is being populated by Javascript?

I want to fetch few data/values from a website. I have used beautifulsoup for this and the fields are blank when I try to fetch them from my Python script, whereas when I am inspecting elements of the webpage I can clearly see the values are available in the table row data. When i saw the HTML Source I noticed its blank there too. I came up with a reason, the website is using Javascript to populate the values in their corresponding fields from its own database. If so then how can i fetch them using Python?

Upvotes: 1

Views: 374

Answers (2)

Thierry Chappuis
Thierry Chappuis

Reputation: 11

The Python binding for Selenium and phantomjs (if you want to use a headless browser as backend) are the appropriate tools for this job.

Upvotes: 1

ToonAlfrink
ToonAlfrink

Reputation: 2626

Yes, you can scrape JS data, it just takes a bit more hacking. Anything a browser can do, python can do.

If you're using firebug, look at the network tab to see from which particular request your data is coming from. In chrome element inspection, you can find this information in a tab named network, too. Just hit ctrl-F to search the response content of the requests.

If you found the right request, the data might be embedded in JS code, in which case you'll have some regex parsing to do. If you're lucky, the format is xml or json, in which case you can just use the associated builtin parser.

Upvotes: 0

Related Questions