CODEWITHSUNDEEP
pythondynamicweb-scrapingweb-crawlerhttp-post
SaikiHanee
SaikiHanee

Reputation: 881

Access and webscrape dynamic pages using python

I am trying to access a web page using it's form parameters. I found the form parameters using Network headers in chrome's developer tab. But it's not working, instead it's just opening the page before these parameters are used (i.e www.irishancestors.ie/search/townlands/ded_index.php)

import webbrowser

webbrowser.open('http://www.irishancestors.ie/search/townlands/ded_index.php?action=listp&parish=Aghagallon')

My intention is to retrieve all the tables of each District Electoral Division of all the counties.

enter image description here

Upvotes: 0

Views: 36

Answers (1)

Wayne Werner
Wayne Werner

Reputation: 51807

webbrowser doesn't do what you think it does.

If you want to get/post data to a webpage, you should use requests

>>> import requests
>>> r = requests.get('https://api.github.com/events')
>>> r = requests.post('http://httpbin.org/post', data = {'key':'value'})

webbrowser is for launching your web browser.

Note that this won't work very well if there's a bunch of javascript in use (well, it might, but it requires a lot more work on your part). If you have a lot of Javascript, it may be easier to use selenium

Upvotes: 1

Related Questions