Reputation: 2411
Using: Rails 3.1.1
I am using the googleajax gem to perform Google-searches in a script with severals of thousands searches.
After some 20 searches or so, I need to have a rescue that waits and retries since it seems like you cannot perform more than a certain amount of searches in a row. After approximately one minute the retry makes the search continue for 10 more searches. The result is that it takes about one minute to perform 10 searches, which makes the script incredibly slow.
It seems likely that Google has a block in the amount of searches one can perform (based on ip? based on googleajax referrer?) but is there a way around it?
What can I do to be able to perform Google searches through the googleajax gem without having to pause and wait all the time? What alternatives do I have?
The code (with unimportant parts cut out):
begin
puts "Searching with " + gsquery
results = GoogleAjax::Search.web(gsquery)[:results]
if results.count > 0
puts "#{results.count} results found for #{page.name}. Registering the connection!"
end
rescue
puts "Try again in 3 sec"
sleep 3
retry
rescue Timeout::Error
puts "Timeout Error, sleep 15 sec"
sleep 15
retry
end
Upvotes: 1
Views: 518
Reputation: 1
Ive found this neat little gem to be quite handy in my latest project. Ruby - Google Search API
Here is a simple use case for searching for an image. This basically states that if the item's name does not equal an empty string, return the search of the first 5 images using the item's name. If the item's name is equal to a empty string and thus being nil, do nothing.
- if item.name != ""
- Google::Search::Image.new(:query => item.name).first(5).each do |image|
= image_tag(image.uri)
Upvotes: 0
Reputation: 4136
Sorry, but I think you're out of luck. GoogleAjax uses the now deprecated web search API (it's been deprecated for over a year now), which may disappear at any point in the future, making the gem useless. Secondly, both the web search API and it's replacement are limited to a maximum number of queries a day, beyond which the service will just stop responding - it's 100 queries a day for the custom search API. To get more than that you'll have to pay (the rate is $5 / 1000 searches). The rate limit is based on the number of queries associated with a single API key.
I'd suggest that you:
Upvotes: 2