file_A (~500MB, 1.6M lines) consists of all equal length search terms, 1 per line, not sorted.

file_B consists of all equal length text lines, 1 per line, not sorted

I've been able to run "grep -F -f file_A file_B >> output.txt" with any size file_B without problem on a box with 52GB ram. Problem is I'm now limited to 4GB ram and thus the size of file_A is now too large for this to run without exhausting available memory.

Short of manually chopping up file_A into smaller bites, is there any easy way to script this to grep for first 1000 lines of file_A, then when thats finished to automatically grep for lines 1001-2000, ect. until I've gone through all of file_A?

  • 1
    Use split to chop up the pattern file, that would be easiest. – Kusalananda Jul 4 '17 at 19:56
  • Does the answer below solve your problem? If so, please use the checkmark to indicate so. Otherwise, let us know what's missing. Thank you! – Jeff Schaller Jul 16 '17 at 12:36

Loop through chunks of file_A, sending them as stdin to the same grep statement; adjust 1000 to your available memory:

nlines=$(wc -l < file_A)
for((i=1; i < nlines; i += chunk)) 
  sed -n $i,+$((chunk - 1))p file_A | grep -F -f - file_B
done > output
| improve this answer | |
  • Correction, should be: sed -n $i,+$((chunk - 1))p file_A. Because, else we will have next ranges in the sed: sed 1, 1001; sed 1001, 2001; sed 2001, 3001; See? Last line of the previous range the same as first line of the next range. Repeating occur. Wanted to solve this the same way, but didn't know how to pass output to the grep -f. – MiniMax Jul 5 '17 at 14:07
  • excellent catch; thank you! I've updated the code accordingly – Jeff Schaller Jul 5 '17 at 14:20
  • Thanks for this code. This worked flawlessly for several months. I actually split the File_A into several segments and ran the above code on each segment using Parallel. I tweaked the chunk size to maximize how many instances I could run while keeping memory out of swap and it sped things up substantially. – Butch Nov 14 '17 at 5:19
  • However, something has changed (either the way the system interprets this, or more likely I've hosed something up that is negatively impacting things) #!/bin/bash nlines=$(wc -l < Xab) chunk=100000 for((i=1; i < nlines; i += chunk)) do echo "Xab started: $(date), line = ${i}" sed -n $i,+$((chunk - 1))p Xab | grep -F -f - . >> Match.lst done this will print the entire test file to Match.lst if there is a hit at grep. Not sure what the problem is – Butch Nov 14 '17 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.