Ruby web spider & search engine library

Question

I'm looking for a Ruby library or gem (or set of gems) which will not only do spidering, but also collect the data into, say, a database, and allow basic searches on the data (i.e. a typical web search).

I've found several spidering libraries, so that part seems well covered (I was going to try Anemone first), but I can't find anything that will take the spidered data and allow querying on it. For lack of an existing one, I was going to write something myself with Anemone.

Any suggestions?

Answer 1

there is a ruby gem may help you: http://spidr.rubyforge.org/

Answer 2

There are lots of great stuff on github.com

Answer 3

That blog post might give you some pointers. Also, look into ferret for the search part.