CODEWITHSUNDEEP
javascriptbrowserautomationbrowser-automation
Ryan
Ryan

Reputation: 15270

How can I automate a browser to visit 1000 pages and trigger a simple javascript function on each page?

I need to load 1000 urls on a tracking/reporting app (after authenticating via an HTML form) and trigger a "resubmit" javascript function. Unfortunately there is no bulk action to handle all at once so I'm left with automation. What are my options?

http://domain.com/0001.php
http://domain.com/0002.php
http://domain.com/0003.php
...
http://domain.com/1000.php

Each of the above pages has a resubmit() javascript function triggered by a href. How can I automate triggering these?

Example:

<form action="/resubmit" method="POST">
  <input type="hidden" name="security_token" value="SUPER-LONG-HASH">
  <input type="hidden" name="url" value="http://mysite.com/0001.html">
  <input type="hidden" name="redirect" value="long-string">
  <script type="text/javascript">
    window["resubmit"] = function () {
      document["resubmit"].submit();
      return false;
    }
  </script>
  <a href="javascript:resubmit()" class="resubmit-class">resubmit</a>
</form>

I'm on a Mac. Unix, Perl, Bash, PHP, Automator, FireFox iMarcos are all available.

Upvotes: 2

Views: 2555

Answers (4)

Ryan
Ryan

Reputation: 15270

I already voted up the other answers, but in the end I wound up using straight AppleScript. This was helpful because it used an existing session so I didn't have to deal with any authentication problems. Thanks for all of your help, everyone. I look forward to becoming familiar with the tools you shared.

set thePath to (path to desktop as Unicode text) & "list_of_urls.txt"
set theFile to (open for access file thePath)
set theContent to (read theFile)
close access theFile

set theURLs to every paragraph of theContent

tell application "Safari"
    repeat with theURL in theURLs
        make new document
        set URL of front document to theURL
        delay 5
        set theScript to "document.getElementsByClassName('resubmit-class')[0].click();"
        do JavaScript theScript in current tab of first window
        do JavaScript "window.resubmit()" in front document
        delay 5
        close front document
    end repeat
end tell

Upvotes: 3

Darryl Hein
Darryl Hein

Reputation: 145117

I don't know if this will help you, but you could try Fake. I think it would allow you to automate submitting the form and make a loop.

Upvotes: 1

Željko Filipin
Željko Filipin

Reputation: 57312

I would use Ruby+Watir for that. Example code (not tested):

require "watir-webdriver"
browser = Watir::Browser.new :firefox

urls = ["http://domain.com/0001.php", "http://domain.com/0002.php"] # add more URLs here
urls.each do |url|
  browser.goto url
  browser.a(:text => "resubmit").click
end

Upvotes: 2

nrabinowitz
nrabinowitz

Reputation: 55678

You should check out PhantomJS, "a headless WebKit with JavaScript API". It allows you to run a WebKit browser instance from the command line and execute Javascript.

You might be able to save some time using Pjscrape, a tool built on top of PhantomJS that can spider multiple pages or take a long list of URLs (disclaimer: this is my project). I haven't tried it with 1,000+ URLs, but I think you could do what you describe with the following 6 lines:

pjs.addSuite({
    urls: [...], // your very long list here
    scraper: function() {
        window.resubmit();
    }
});

Upvotes: 3

Related Questions