CODEWITHSUNDEEP
databasenosqlweb-crawlerkey-valuedocument-storage
Batuhan Tüter
Batuhan Tüter

Reputation: 311

What is the best database for storing URIs on disk, for fast reads?

I am writing a Web crawler for Linked Data and I need to store crawled URIs on the disk (not necessarily distributed but could be). My crawler will check if a URI exists in the storage constantly. If a URI does exists, it will do nothing, if it does not exists, it will crawl the URI and write the URI to the storage. At first, since the storage will be rather empty, there will be more writes than reads but at some point, reads will be more than writes and I favor faster reads. I don't need any join operations etc. I am thinking about a document based NoSQL storage and I define a key="domain of a URI", value="an array of the whole URIs". I am not sure if I need a secondary index on the value.

Upvotes: -1

Views: 63

Answers (1)

golyadkin
golyadkin

Reputation: 84

Since you are only interested in searching and storing, SQLite is suitable for your purposes. It's a lightweight database engine.

Selamlar.

Upvotes: 1

Related Questions