I have a directory of thousands of zip files that are dated in the form
YYYYMMDD_hhmmss.zip and each about 300K. Within each zip file is about 400 xml files each about 3K.
I need to be able to search and find a given string within a date-range of the zip files.
The current (albeit mediocre) solution
I have the following one-liner
find /home/mydir/ -type f | sort | \ awk "/xml_20140207_000016.zip/,/xml_20140207_235938.zip/" | \ xargs -n 1 -P 10 zipgrep "my search string"
The point of it is to
- list all the files in my thousand-file directory
- sort this list of files
- retrieve a range of files based on given dates (this
awkcommand only prints lines after that first matched string and up to that second matched string)
- pass each line of the result which corresponds to a single file to
This one-liner runs horribly slow, even with 10 processes on a 24-core machine. I believe it's slow because of the
zipgrep command but I'm not wise enough to know how to improve it. I don't know if I should be, but I'm a little embarrassed that a colleague wrote a java tool that runs faster than this script. I'd like to reverse that if possible. Then, does anyone know how to make this command faster in this context? Or to improve any part of it at all?