How do I scrape data from a javascript page for iOS?

Question

I'm developing a mobile app (iOS 5.0 and above compatible) for a website where users can view certain data directly from their landing page. The user can refine the results by selecting specific options (i.e. location and/or date). They don't offer any web service calls, and won't allow us to access their database. So my only resort of collecting the data is from 'scraping' the site directly.

My issue is that I don't know how make the changes that the users can make on the site from a http request from the phone. For example the site below:

can be pulled from the site and saved as a string with the following command

NSString *html = [NSString stringWithContentsOfURL:urlrequest encoding:NSUTF8StringEncoding error:&err];

I'm able to separate the relevant data:

// I decided to add the script function name in case anyone wanted to look for themselves
<script type="text/javascript" language="Javascript">
//<![CDATA[
function loadData(){

var winMsgTitle = "Date: 04/11/2012";

// this is the actual data I'm concerned with:
gLatLong = new GLatLng(31.59019444444444, -110.50655555555555);
tmpMapIcon = new MapIcon("0_0", 21, 15);
marker = createMarker(gLatLong, 0.0, createInfoWinMsg(winMsgTitle, ".00", ".84", "5711", "2012", "", "07:00"), tmpMapIcon);
map.addOverlay(marker);
point = null;marker = null;
gLatLong = new GLatLng(32.2938260182, -110.7896411419);
tmpMapIcon = new MapIcon("0_0", 21, 15);
marker = createMarker(gLatLong, 0.0, createInfoWinMsg(winMsgTitle, ".00", "1.00", "1254", "2012", "", "07:00"), tmpMapIcon);
map.addOverlay(marker);
point = null;marker = null;
gLatLong = new GLatLng(33.5966853633, -112.1744066477);
tmpMapIcon = new MapIcon("0_0", 21, 15);
marker = createMarker(gLatLong, 0.0, createInfoWinMsg(winMsgTitle, ".00", ".70", "256", "2012", "", "07:00"), tmpMapIcon);
map.addOverlay(marker);
point = null;marker = null;

Note: There are obviously more points, this is just snip it

In order to get another location, or date, I have to manually select via the site and select the options on the right hand pane. My question is, how do I make those changes programmatically via objective-c?

Answer 1

If i understand you correctly, you want to pull the data without displaying the website to your user?

If yes, the only solution i can currently think of would be an off-screen UIWebView, in wich you do some JavaScript-magic to simulate user-input, and then get the data out.

This is however a very hackish approach and i would not suggest you use this in a shipping application.
In fact, i would suggest not doing any webscraping at all, because if they change their sites structure, your app fails to work.

If you are really developing an app "for them" then go tell them that you need a webservice in order to write a good application.

If you are not working "for them" but are just trying to write an app that uses their service to get data, then what you are doing is not only complicated, but does most probably also violate their EULA and you should therefore not be doing that at all. Consider contacting them and see if you can get them to work with you in that case.

Answer 2

I suppose you use UIWebView to load that page.

One possible solution is to write JavaScript functions to simulate normal user operations on the right hand pane (for example, use jQuery's trigger function to select the drop down list, select 'Date Range', etc.).

These javascript functions could be coded as strings in your app code.

Then, call stringByEvaluatingJavaScriptFromString: on the webview to run these javascript functions:

[webView stringByEvaluatingJavaScriptFromString:javascript]

Here the javascript parameter is a string you construct that calls your javascript functions. For example:

NSString *javascript = [NSString stringWithFormat:@"selectRegion(%@)", targetRegion];

When doing this, I'd suggest you write these javascripts and test them using a browser's debug tool(Safari's Web Inspector, Firefox's FireBug plugin...) to make sure they work as expect first.