CODEWITHSUNDEEP
phphtmltext
heilstalin
heilstalin

Reputation: 31

How would I be able to search a massive text file? (20+GB)

I'd like to be able to search a 20GB+ .txt file on Windows 7. I have ~20GB of free RAM for a program to run (less RAM than the size of the txt file,) and a software raid 0 between 2 laptop hard drives that yields me about 100MB/s R/W speeds. The end goal is to have a web interface where you could put in a search query and it would tell you what number into the text file the query was found. There is a webserver already running on this computer (WAMP). Any help would be greatly appreciated.

Upvotes: 0

Views: 257

Answers (1)

Brad
Brad

Reputation: 163612

There is no reason to re-invent searching. We have nice databases for this task these days. Even if you must start with a source text file of this size, I would load it into a database for searching. Once loaded and indexed, you have a very efficient rig.

Now, you didn't specify what you're searching for and how so it's hard to give specific advice. What I can tell you is that I've had great luck with text searches in Solr. It's built for tasks like these.

Upvotes: 5

Related Questions