I have a directory of thousands of zip files that are dated in the form YYYYMMDD_hhmmss.zip and each about 300K. Within each zip file is about 400 xml files each about 3K.

The problem

I need to be able to search and find a given string within a date-range of the zip files.

The current (albeit mediocre) solution

I have the following one-liner

find /home/mydir/ -type f | sort | \
awk "/xml_20140207_000016.zip/,/xml_20140207_235938.zip/" | \
xargs -n 1 -P 10 zipgrep "my search string"

The point of it is to

  1. list all the files in my thousand-file directory
  2. sort this list of files
  3. retrieve a range of files based on given dates (this awk command only prints lines after that first matched string and up to that second matched string)
  4. pass each line of the result which corresponds to a single file to zipgrep

The question

This one-liner runs horribly slow, even with 10 processes on a 24-core machine. I believe it's slow because of the zipgrep command but I'm not wise enough to know how to improve it. I don't know if I should be, but I'm a little embarrassed that a colleague wrote a java tool that runs faster than this script. I'd like to reverse that if possible. Then, does anyone know how to make this command faster in this context? Or to improve any part of it at all?

  • 1
    Your sure the year range is only up to the year 999? not that it changes the question much. – Anthon Jul 7 '14 at 13:25
  • 4
    zipgrep separately unzips every file in the zipfile to grep it, that seems ineffective if you're looking at all of them anyway. Maybe unzipping int a temp directory and grepping there or fiddling with the output of unzip -p or unzip -c will give a little improvement. – Ulrich Schwarz Jul 7 '14 at 14:24
  • @UlrichSchwarz I didn't know about that, I'll try it. Thanks! – fifosine Jul 7 '14 at 14:25
  • How high is the likelihood of the string occuring? Expanding on my previous suggestion, you could first check with unzip -c if the zip file is relevant to your results at all and only then do a closer inspection of the individual files therein. – Ulrich Schwarz Jul 7 '14 at 14:26
  • The likelihood of the string occurring is not high, but also the archived file names are not indicative of what's inside of them. – fifosine Jul 7 '14 at 14:27

There's a part you can easily improve, but it isn't the slowest part.

find /home/mydir/ -type f | sort | \
awk "/xml_20140207_000016.zip/,/xml_20140207_235938.zip/"

This is somewhat wasteful because it first lists all files, then sorts the file names and extracts the interesting ones. The find command has to run to completion before the sorting can begin.

It would be faster to list only the interesting files in the first place, or at least as small a superset as possible. If you need a finer-grained filter on names than find is capable of, pipe into awk, but don't sort: awk and other line-by-line filters can process lines one by one but sort needs the complete input.

find /home/mydir/ -name 'xml_20140207_??????.zip' -type f | \
awk 'match($0, /_[0-9]*.zip$/) &&
     (time = substr($0, RSTART+1, RLENGTH-5)) &&
     time >= 16 && time <= 235938' |
xargs -n 1 -P 10 zipgrep "my search string"

The part which is most obviously suboptimal is zipgrep. Here there is no easy way to improve performance because of the limitations of shell programming. The zipgrep script operates by listing the file names in the archive, and calling grep on each file's content, one by one. This means that the zip archive is parsed again and again for each file. A Java program (or Perl, or Python, or Ruby, etc.) can avoid this by processing the file only once.

If you want to stick to shell programming, you can try mounting each zip instead of using zipgrep.

… | xargs -n1 -P2 sh -c '
    mkdir "mnt$$-$1";
    fuse-zip "$1" "mnt$$-$1";
    grep -R "$0" "mnt$$-$1"
    fusermount -u "mnt$$-$1"
' "my search string"

Note that parallelism isn't going to help you much: the limiting factor on most setups will be disk I/O bandwidth, not CPU time.

I haven't benchmarked anything, but I think the biggest place for improvement would be to use a zipgrep implementation in a more powerful language.


Some quick ideas;

  • If all the files are in a single directory, you can get rid of the find
  • Your file name convention sorts itself by date, so you don't need the sort bit either
  • With this two pieces out of the way, and if the date range is known, you can use a simple filename glob instead of awk. For example (assuming your shell is bash):

    • All files of a single day

      echo xml_20140207_*.zip | xargs -n 1 -P 10 zipgrep "my search string"

    • Files created between 15:00 and 18:00, either in Feb 07 or Feb 10 2014:

      echo xml_201402{07,10}_1{5..7}*.zip | xargs -n 1 -P 10 zipgrep "my search string"

  • Thanks for your improvements, but the area that most needs it (xargs and zipgrep) remains. These are the commands that provide the bottleneck. As Peter Norvig says, "don't waste effort trying to speed up parts of your program that don't take much of the total time". – fifosine Jul 7 '14 at 14:07

It is not clear where your bottleneck is. Let us assume it is in reading the files. Depending on your storage system, it is faster to read the whole file before processing it. This is especially true for zipgrep which does a few seeks into the file: If the file is not completely in memory you will be waiting for the disk to seek.

find ... | parallel -j1 'cat {} >/dev/null; echo {}' | parallel zipgrep "my search string"

The above will cat one file at a time and thereby put it into memory cache, then run one zipgrep per CPU, which will then read from memory cache.

I have used RAID systems where you got a 6x speed up by reading 10 files in parallel than reading 1 file at a time or reading 30 files in parallel. If I had to run the above on that RAID system, I would adjust -j1 to -j10.

By using GNU Parallel instead of xargs you guard yourself from the mixing of output (see http://www.gnu.org/software/parallel/man.html#DIFFERENCES-BETWEEN-xargs-AND-GNU-Parallel).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.