What is the fastest way to grep 400gb binary file? I need one txt file from hdd dump and i know some strings from it and want to find this file in dump.

I tried to use grep -a -C 10 searchstring but grep crashes with out of memory when it tries to read large chunk of data without newline symbols. Also i would like to start searching from not from the beginning but from some point of file

  • Have you tried grep --mmap ... already? Real programmers would do that with vim -R -b 400gbfile and then /pattern. – ott-- Aug 13 '15 at 20:52
  • @ott your vim answer piqued my interest. Does vim really only load what it needs into memory? A quick internet search is not conclusive. Some people say it still has a hard time with large files. Others say "Ctrl-c" will stop it trying to load everything into RAM. So I'm not sure whether your answer is correct. – Michael Martinez Aug 13 '15 at 21:49
  • @ott-- Real programmers would do it in Fortran. – Parthian Shot Aug 13 '15 at 22:13

I would use strings that way :

strings 400Gfile.bin | grep -C 10 searchstring

To start at a given offset (eg: 20G),

dd if=400Gfile.bin bs=20G skip=1 | strings | grep -C 10 searchstring
  • When looking for plain text files add -w to strings (to make it print newlines and whitespace, i.e. full text file) and 10 should be the number of lines you expect your text file to have. With bs=20G you'll probably get an out of memory error on most systems; smaller bs and larger skip – frostschutz Aug 13 '15 at 21:06
  • @frostschutz strings has no -w option I'm aware of. Moreover, it does preserve spaces and newlines of both text and binary files without requiring any specific option. – jlliagre Aug 13 '15 at 21:11
  • The option was added in GNU strings (binutils 2.25). Without it yes | strings prints nothing so the output won't contain a whole plain text ASCII file if it contains short/empty lines. In that case have strings print the offset the string was found at and then grab the data from around that offset using dd. – frostschutz Aug 13 '15 at 21:23
  • @frostschutz Thanks. This Gnu extension is quite recent. I was confused by "newlines and whitespace" while you really meant "short and empty lines". – jlliagre Aug 13 '15 at 21:30
  • ... Hoping that any chunking doesn't split at a possible grep match. – Jeff Schaller Aug 13 '15 at 23:56

The problem with grep is that it needs to have an entire line in memory. If that line is so large that it doesn't fit in memory, than grep bombs. The only way around this dilemma is to feed small chunks to grep. (which is really what grep should be doing itself, anyway, but it doesn't).

Use dd so that you can specify an offset to start at, and use fold or grep --mmap to avoid running out of memory on a line that is larger than the available RAM. grep --mmap will prevent the system from choking, but may or may not prevent grep itself from choking. This would be a good thing for someone to test. fold will allow you to insert a newline at regular intervals, this satisfies the criteria to split the input into manageable chunks.

dd if=bigfile skip=xxx | fold | grep -b -a string

The -b gives you the byte offset which you will find useful to know where your text strings are located within the file.

I tested this on a 100GB logical volume on one of my KVM hypervisors, using the search string "Hard" and running vmstat in a separate window to monitor performance. The logical volume is basically formatted as a hard drive (partitions and filesystems) on which a guest Linux VM is installed. There wasn't any impact on system performance. It processed each gig in about 33 seconds (of course this will vary widely depending on your hardware).

You said you wanted quick performance. This should give you the quickest performance from using utilities in a shell script. The only way to get a quicker search would be to write a program in C that seeks to an offset, reads in a specified chunksize, feeds that chunk into a pattern matching algorithm, before moving on to the next chunk. Seems like this type of "improved grep" should already be in existence, but searching online I don't find one.

  • 1
    As noted in comment to other answer, this will fail to detect a string that happens to fall on a fold boundary. But if you run grep with a few different strings - `grep "string1|string2|string3" - chances are probably slim all three are always falling on a fold boundary, so you'll still find what you're looking for. – Michael Martinez Aug 14 '15 at 0:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.