How can I optimize this Unix command?

Question

The following command takes about 10 minutes to output the result

find . -name "muc*_*_20160920_*.unl*" | xargs zcat |
    awk -F "|" '{if($14=="20160920100643" && $22=="567094398953") print $0}'| head

How can I improve its performance?

Answer 1

That's already quite optimised. It's hard to know what the bottle neck is without knowing more details like:

• type of storage (HD, SSD, network, RAIDed)
• number and average size of matching files
• number of directories and other non-matching files
• number of fields in each line
• average length of a line

Things you can do in any case:

• replace -print | xargs with -exec cmd {} + or -print0 | xargs -r0 if your find/xargs support it. -print | xargs is not only wrong but also more expensive as xargs needs to decode characters to find out which ones are blanks and do some expensive quote processing.
• fix the locale to C (export LC_ALL=C). Since all the characters involved here (| and decimal digits for the file contents and latin letters, period and underscore for the file names) are part of the portable charset, if your charset is otherwise UTF-8 or some other multi-byte charset, switching to C with its single-byte charset will safe a lot of work for find and awk.
• simplify the awk part to: awk -F "|" '$14 == "20160920100643" && $22 == "567094398953"'.
• since you're piping the output to head, you may want to disable output buffering for awk so that it outputs those 10 lines as early as possible. With gawk or mawk, you can use fflush() for that. Or you could add a if (++n == 10) exit in awk.

To sum-up:

(export LC_ALL=C
find . -name "muc*_*_20160920_*.unl*" -exec zcat {} + |
  awk -F "|" '$14 == "20160920100643" && $22 == "567094398953" {
    print; if (++n == 10) exit}')

If CPU is the bottle-neck, on a multi-core GNU system, you could try:

(export LC_ALL=C
find . -name "muc*_*_20160920_*.unl*" -print0 |
  xargs -r0P 4 -n 100 sh -c '
    zcat "$@" | 
      awk -F "|" "\$14 == "20160920100643" && \$22 == "567094398953" {
        print; fflush()}"' sh | head)

To run 4 zcat | awk jobs in parallel on 100 files batches.

If that 20160920100643 is a time-stamp, you may want to exclude files that have been last-modified before that. With GNU or BSD find, add a -newermt '2016-09-20 10:06:42'.

If lines have a great number of fields, you get a penalty for awk splitting it and allocating so many $n fields. Using an approach that only considers the first 22 fields could speed things up:

grep -E '^([^|]*\|){13}20160920100643(\|[^|]*){7}\|567094398953(\||$)'

instead of the awk command. With GNU grep, add the --line-buffered option to output the lines as early as possible in the parallel approach or -m 10 to stop after 10 matches in the non-parallel one.

To sum up, if CPU is the bottle neck and you have at least 4 CPU cores on your system and there are at least 400 muc* files and you're on a GNU system (where grep is usually significantly faster than GNU awk):

(export LC_ALL=C
find . -name "muc*_*_20160920_*.unl*" -newermt '2016-09-20 10:06:42' -print0 |
  xargs -r0P 4 -n 100 sh -c '
    zcat "$@" | 
      grep --line-buffered -E \
        "^([^|]*\|){13}20160920100643(\|[^|]*){7}\|567094398953(\||$)"
  ' sh | head)

Note that in the parallel approach, you may get the output of the grep commands inter-mingled (though with line-buffering and provided lines are less than a few kilobytes large, line boundaries should be preserved).

Answer 2

@Stéphane Chazelas's answer provides a great many details on how you can optimize the command pipeline

find . -name "muc*_*_20160920_*.unl*" | xargs zcat |
    awk -F "|" '{if($14=="20160920100643" && $22=="567094398953") print $0}'| head

I'm going to provide another way to approach the problem where you actually measure where you're spending the most time. Once you find where the time is spent, you can determine what to do about it. If you want to improve your 10-minute run time, optimizing a step that takes 2 seconds is almost useless.

When I look at the command pipeline, three things draw my attention:

1. find . - What is the directory structure like? How many files per directory? Is the directory local to the system the command is being run on? A remote filesystem is going to be a lot slower.
2. -name "muc*_*_20160920_*.unl*" - How close are all the file names in the directory structure? Are they all "close" to the name and difficult/CPU intensive to match? Because every file in the directory tree has to have its name read from disk and compared to the pattern.
3. xargs zcat - The xargs doesn't seem to me that it will be too much of a performance problem, especially compared to the find issues above and the zcat itself. Even if it's 10,000 or even 10,000,000 file names, the time used in passing and parsing just the names is almost certainly negligible compared to the time spent finding the names and then opening and decompressing all the files themselves. How large are the files? Because you're decompressing the entirety of every file that matches your find's filename pattern.

How can you determine what's the major performance problem? Measure the performance of each command in the pipeline. (See https://stackoverflow.com/questions/13294554/how-to-use-gnu-time-with-pipeline for details on timing an entire pipeline.) You can run the following commands and see how much time each step contributes to the processing time for the entire pipeline:

/usr/bin/time find . - This tells you how long it takes to run through your directory tree. If this is slow, you need a better storage system. Flush your filesystem cache[s] before timing this to get a worst-case measurement, then run the timed find again and see how much caching impacts performance. And if the directory isn't local, try running the command on the actual system that the files are on.

/usr/bin/time find . -name "muc*_*_20160920_*.unl*" - This will tell you how long it takes to pattern-match the file names. Again, flush filesystem cache[s] and run it twice.

/usr/bin/time bash -c "find . -name 'muc*_*_20160920_*.unl*' | xargs zcat > /dev/null" - This is the one I suspect is the major component of your pipeline's long running time. If this is the problem, parallelizing the zcat commands per Stéphane Chazelas answer may be the best answer.

Continue adding steps from the original command pipeline to the one being tested until you find where you're spending most of your time. Again, I suspect it's the zcat step. If so, perhaps the zcat parallelization that @Stéphane Chazelas posted will help.

Parallelizing zcat may not help - it may even hurt performance and slow processing down. With only one zcat running at a time, IO may be in a nice streaming pattern that minimizes disk seeks. With multiple zcat processes running at once, the IO operations may contend and actually slow down processing as disk heads need to seek and any read-ahead done becomes less effective.

If the zcat step is your major performance bottleneck, and running multiple zcat processes at one time doesn't help or actually slows you down, your pipeline is IO-bound and you need to address the problem by using faster storage.

And again - if the directory isn't local the machine your running the command pipeline on, try running it on the machine that the file system actually is on.

Answer 3

As noted in comments zgrep is better choice for such kind of tasks with globstar option which allows to use ** as all path inside the directory except hidden

shopt -s globstar
zgrep -m 10 '^\([^|]*|\)\{13\}20160920100643|\([^|]*|\)\{7\}567094398953' ./**muc*_*_20160920_*.unl*
shopt -u globstar

Answer 4

As pointed out, it is not possible to give the correct answer without some extra details.

locate -0 -b -r '^muc.*_.*_20160920_.*.unl.*gz' | 
   xargs -0  zcat |
   awk -F "|" '$14=="20160920100643" && $22=="567094398953"'| head

• 1: locate (if available) is much faster than ** or find; The regular expression used must be tuned...

• 2 and 3: the PO's filter

As @rudimeier, wisely pointed out, there are issues regarding the availability and update state of locate. (eg in most linux machines locate is update daily; this way it will fail to find files created today)

Nevertheless if locate is available this will produce a very impressive speedup.

It would be interesting if the PO could provide the time ... of the various solutions