Performance of loop vs expansion

Question

Need expert suggestions on below comparison:

Code Segment using loop:

for file in `cat large_file_list`
do
    gzip -d $file
done

Code segment using simple expansion:

gzip -d `cat large_file_list`

Which one will be faster? Have to manipulate large data set.

Answer 1

Complications

The following will work only sometimes:

gzip -d `cat large_file_list`

Three problems are (in bash and most other Bourne-like shells):

1. It will fail if any file name has space tab or newline characters in it (assuming $IFS has not been modified). This is because of the shell's word splitting.

2. It is also liable to fail if any file name has glob-active characters in it. This is because the shell will apply pathname expansion to the file list.

3. It will also fail if filenames starts with - (if POSIXLY_CORRECT=1 that only applies to the first file) or if any filename is -.

4. It will also fail if there are too many file names in it to fit on one command line.

The code below is subject to the same problems as the code above (except for the fourth)

for file in `cat large_file_list`
do
    gzip -d $file
done

Reliable solution

If your large_file_list has exactly one file name per line, and a file called - is not among them, and you're on a GNU system, then use:

xargs -rd'\n' gzip -d -- <large_file_list

-d'\n' tells xargs to treat each line of input as a separate file name.

-r tells xargs not to run the command if the input file is empty.

-- tells gzip that the following arguments are not to be treated as options even if they start with -. - alone would still be treated as - instead of the file called - though.

xargs will put many file names on each command line but not so many that it exceeds the command line limit. This reduces the number of times that a gzip process must be started and therefore makes this fast. It is also safe: the file names will also be protected from word splitting and pathname expansion.

Answer 2

I doubt it would matter much.

I would use a loop, just because I don't know how many files are listed in the list file, and I don't (generally) know if any of the filenames have spaces in their names. Doing a command substitution that would generate a very long list of argument may result in an "Argument list too long" error when the length of the list generated is too long.

My loop would look like

while IFS= read -r name; do
    gunzip "$name"
done <file.list

This would additionally allow me to insert commands for processing the data after the gunzip command. In fact, depending on what the data actually is and what needs to be done with it, it may even be possible to process it without saving it to file at all:

while IFS= read -r name; do
    zcat "$name" | process_data
done <file.list

(where process_data is some pipeline that reads the uncompressed data from standard input)

If the processing of the data takes longer than the uncompressing of it, the question of whether a loop is more efficient or not becomes irrelevant.

Ideally, I would prefer to not work off a list of filenames though, and instead use a filename globbing pattern, as in

for name in ./*.gz; do
    # processing of "$name" here
done

where ./*.gz is some pattern that matches the relevant files. This way we are not depending on the number of files nor on the characters used in the filenames (they may contain newlines or other whitespace characters, or start with dashes, etc.)

Related:

• Understanding "IFS= read -r line"

Answer 3

Out of those two, the one with all files passed to a single invocation of gzip is likely to be faster, exactly because you only need to launch gzip once. (That is, if the command works at all, see the other answers for the caveats.)

But, I'd like to remind of the golden rule of optimization: Don't do it prematurely.

1. Don't optimize that sort of thing before you know it's a problem.

   Does this part of the program take a long time? Well, decompressing large files might, and you're going to have to do it anyway, so it might not be that easy to answer.

2. Measure. Really, it's the best way to be sure.

   You'll see the results with your own eyes (or with your own stopwatch), and they'll apply to your situation which random answers on the Internet might not. Put both variants in scripts and run time script1.sh, and time script2.sh. (Do that with a list of empty compressed files to measure the absolute amount of the overhead.)

Answer 4

How fast is your disk?

This should use all your CPUs:

parallel -X gzip -d :::: large_file_list

So your limit is likely going to be the speed of your disk.

You can try adjusting with -j:

parallel -j50% -X gzip -d :::: large_file_list

This will run half of the jobs in parallel as the previous command, and will stress your disk less, so depending on your disk this can be faster.