CODEWITHSUNDEEP
pythonweb-scraping
Ankit Goyal
Ankit Goyal

Reputation: 33

Parsing a Dynamic Web Page using Python

I am trying to parse a WebPage whose html source code changes when I press a arrow-key to get a drop-down list. I want to parse the contents of that drop down list. How can I do that?

Example of the Problem: If you go to this site: http://in.bookmyshow.com/hyderabad and select the arrow button on comboBox "Select Movie" a drop-down list of movies appears. I want to get a list of these movies.

Thanks in advance.

Upvotes: 0

Views: 2116

Answers (3)

argaen
argaen

Reputation: 4245

You may want to have a look at selenium. It allows you to reproduce exacly the same steps as you do because it also uses the browser (Firefox, Chrome, etc).

Ofc, it's not as fast as using mechanize, urllib, beautifulsoup and all this stuff, but it is worth a try.

Upvotes: 0

Matt
Matt

Reputation: 17629

The actual URL with the data used to populate the drop-down box is here:

I'd be a bit careful though and double-check with the site terms of use or if there are any APIs that you could use instead.

Upvotes: 3

Stewart
Stewart

Reputation: 1899

You will need to dig into the JavaScript to see how that menu gets populated. If it is getting populated via AJAX, then it might be easy to get that content by re-doing a request to the same URL (e.g., do a GET to "http://www.example.com/get_dropdown_entries.php").

Upvotes: 0

Related Questions