CODEWITHSUNDEEP
pythonweb-scraping
rodling
rodling

Reputation: 998

Using button on a page with python

I am scraping a page for some data, however I need to insert text into a text box, submit the form and scrape the result page. I looked at the page source, but I'm not sure how to activate the button or pass down the argument for it.

Website is http://archive.org/web/web.php Trying to look at some historicals, and no idea what to use for this. Open to any solution

Upvotes: 1

Views: 1172

Answers (1)

iloahz
iloahz

Reputation: 4561

First you should know that click on that button usually does a POST to some urls, passes the data in that form, here is:

<form id="wwmform" name="wwmform" method="get" action="http://web.archive.org/form-submit.jsp" onsubmit="document.location.href='http://web.archive.org/web/*/'+document.getElementById('wwmurl').value;return false;" style="display:inline;">
      <input id="wwmurl" type="text" name="url" size="50" value="http://">
      <button type="submit" name="type" value="urlquery" class="roundbox5">Take Me Back</button>
    </form>

you see the action attribute? That's where the data goes to.

So in python, you may need urllib and urllib2 to encode the data and post it to the target url and then fetch the outcome.

ps: watch out the onsubmit

Upvotes: 1

Related Questions