CODEWITHSUNDEEP
apisearchscreen-scrapingdata-mining
rinogo
rinogo

Reputation: 9163

What free/paid search API's allow for programmatic querying and caching/storage of the resulting data?

If you've done any serious research into search API's, you know that most of them have a huge slew of TOS/TOU restrictions that make them nearly impossible to use in anything but the most inane applications.

Bing's 2.0 API, Yahoo Search BOSS, Google Places, Google AJAX Search (dead), et al, are far too restrictive for us. I need to run a finite and relatively small number of queries (perhaps 500k) one time only, storing specific data from the results for use within our application.

For example, we need to match up business names with their target websites (we have written the algorithm to make a 'best guess' from a set of results if necessary; we just need a vanilla result set). Also, we need to match an address to this company in question.

Unfortunately, I can find ZERO search API's that will allow us to fire off queries in a programmatic, non-user-initiated manner.

We're even quite eager to give someone cold, hard cash for access to this kind of data; Google, Bing, Yahoo, and others simply seem to not want our money (as evidenced by their TOSes)...

Any thoughts?

Upvotes: 12

Views: 3783

Answers (2)

Donald
Donald

Reputation: 3911

If you know some visual basic I'd suggest playing around with Bing Ad Intelligence. It's a free Excel plugin and all you need to use it is a free Microsoft account.

The query limit is 20,000 words per query. You can get information on Clicks, Impressions, CTR, CPC, Average Bid and Total Cost. The query limit is a little lower if you use the more advanced keyword research features.

Upvotes: 0

seanieb
seanieb

Reputation: 1196

A freely accessible index of 5 billion web pages, their page rank, their link graphs and other metadata, hosted on Amazon EC2.

http://commoncrawl.org/

Their Terms of Service (or TOU) are pretty reasonable and unrestricted too:

http://commoncrawl.org/about/terms-of-use/

Upvotes: 3

Related Questions