CODEWITHSUNDEEP
linuxcommand-line
Nils
Nils

Reputation: 1996

Extract lines containing one of large number of strings from file

Let's say I have a file DATA with 10,000,000 lines. I have another file IDS with 100,000 strings. I want to extract all lines from DATA that contain one of the strings from IDS. An additional condition is that there is a 1:1 relationship between the files, so every ID has one line of DATA and every DATA has one ID.

What is the most efficient, least complicated way to do this using standard linux command-line utilities?

My ideas so far:

  1. Build a huge regex and use grep (easy, may exceeed some limit within grep)
  2. Go through IDS line by line and grep DATA for each string separately, merge results. (easy, probably very inefficient)
  3. Build a hashmap of IDS in python, loop through DATA, extract ID and check against hash map (a bit harder)

Upvotes: 0

Views: 384

Answers (2)

Maxim Razin
Maxim Razin

Reputation: 9466

grep -F -f IDS DATA

Don't miss -F: it prevents from interpreting IDS as regular expressions, and enables a much more efficient Aho-Korasick algorithm.

Upvotes: 3

Alex
Alex

Reputation: 11090

If IDS contains the exact strings you need to find in DATA, one string per line, try using

grep --file=IDS DATA > results

Upvotes: 2

Related Questions