9

Need expert suggestions on below comparison:

Code Segment using loop:

for file in `cat large_file_list`
do
    gzip -d $file
done

Code segment using simple expansion:

gzip -d `cat large_file_list`

Which one will be faster? Have to manipulate large data set.

5
  • 1
    The correct answer will depend on how long it takes to start gzip on your system, the number of files in the file list and the size of those files.
    – Kusalananda
    Jul 1 '19 at 6:25
  • File list will have around 1000 - 10000 files. Size varies from some kilobytes to 500 MB. I have no idea how long it takes to start gzip in my system. any way check? Jul 1 '19 at 6:27
  • 1
    Ok, then it may also depend on the length of the filenames. If the filenames are long, some systems might generate a "argument list too long" error if you tried to do it without a loop since the command substitution would result in a too long command line for the shell to execute. If you don't want to depend on the number of files in the list, just use a loop. Are you spending a significant amount of time decompressing these files compared to the other processing that you will perform on them?
    – Kusalananda
    Jul 1 '19 at 6:54
  • Leon take a look at my test results: "huge-arglist" is 20x faster than "loop" in my setting.
    – user359065
    Jul 2 '19 at 16:08
  • for a happy medium between process starts and command line length, use something like xargs gzip -d < large_file_list but watch out for spaces in filenames, maybe with tr \\n \\0 large_file_list | xargs -0 gzip -d
    – w00t
    Jul 5 '19 at 8:13
19

Complications

The following will work only sometimes:

gzip -d `cat large_file_list`

Three problems are (in bash and most other Bourne-like shells):

  1. It will fail if any file name has space tab or newline characters in it (assuming $IFS has not been modified). This is because of the shell's word splitting.

  2. It is also liable to fail if any file name has glob-active characters in it. This is because the shell will apply pathname expansion to the file list.

  3. It will also fail if filenames starts with - (if POSIXLY_CORRECT=1 that only applies to the first file) or if any filename is -.

  4. It will also fail if there are too many file names in it to fit on one command line.

The code below is subject to the same problems as the code above (except for the fourth)

for file in `cat large_file_list`
do
    gzip -d $file
done

Reliable solution

If your large_file_list has exactly one file name per line, and a file called - is not among them, and you're on a GNU system, then use:

xargs -rd'\n' gzip -d -- <large_file_list

-d'\n' tells xargs to treat each line of input as a separate file name.

-r tells xargs not to run the command if the input file is empty.

-- tells gzip that the following arguments are not to be treated as options even if they start with -. - alone would still be treated as - instead of the file called - though.

xargs will put many file names on each command line but not so many that it exceeds the command line limit. This reduces the number of times that a gzip process must be started and therefore makes this fast. It is also safe: the file names will also be protected from word splitting and pathname expansion.

6
  • Thanks for detailed reply. I understand your mentioned 3 issues. File name is simple and won't face those challenges as the list will hold upto 20000. And my question is basically on performance of those two segments. Thanks. Jul 1 '19 at 6:20
  • 1
    @Leon The for loop will be —by far— the slowest. The other two methods will be very close in speed to each other.
    – John1024
    Jul 1 '19 at 6:36
  • 7
    Also, don't dismiss the potential problems: many many questions here on StackExchange are because word splitting or pathname expansion happened to people who weren't expecting it.
    – John1024
    Jul 1 '19 at 6:38
  • 5
    Note also that there's variation on reading a file with xargs: at least GNU version has --arg-file option (short form -a). So one could do xargs -a large_file_list -rd'\n' gzip -d instead. Effectively, there's no difference, aside from the fact that < is shell operator and would make xargs read from stdin (which shell "links" to file), while -a would make xargs explicitly open the file in question Jul 1 '19 at 7:34
  • 2
    terdon noted in another comment about using parallel to run multiple copies of gzip, but xargs (at least the GNU one), has the -P switch for that, too. On multicore machines that might make a difference. But it's also possible that the decompression is completely I/O-bound anyway.
    – ilkkachu
    Jul 2 '19 at 14:19
12

I doubt it would matter much.

I would use a loop, just because I don't know how many files are listed in the list file, and I don't (generally) know if any of the filenames have spaces in their names. Doing a command substitution that would generate a very long list of argument may result in an "Argument list too long" error when the length of the list generated is too long.

My loop would look like

while IFS= read -r name; do
    gunzip "$name"
done <file.list

This would additionally allow me to insert commands for processing the data after the gunzip command. In fact, depending on what the data actually is and what needs to be done with it, it may even be possible to process it without saving it to file at all:

while IFS= read -r name; do
    zcat "$name" | process_data
done <file.list

(where process_data is some pipeline that reads the uncompressed data from standard input)

If the processing of the data takes longer than the uncompressing of it, the question of whether a loop is more efficient or not becomes irrelevant.

Ideally, I would prefer to not work off a list of filenames though, and instead use a filename globbing pattern, as in

for name in ./*.gz; do
    # processing of "$name" here
done

where ./*.gz is some pattern that matches the relevant files. This way we are not depending on the number of files nor on the characters used in the filenames (they may contain newlines or other whitespace characters, or start with dashes, etc.)

Related:

5

Out of those two, the one with all files passed to a single invocation of gzip is likely to be faster, exactly because you only need to launch gzip once. (That is, if the command works at all, see the other answers for the caveats.)

But, I'd like to remind of the golden rule of optimization: Don't do it prematurely.

  1. Don't optimize that sort of thing before you know it's a problem.

    Does this part of the program take a long time? Well, decompressing large files might, and you're going to have to do it anyway, so it might not be that easy to answer.

  2. Measure. Really, it's the best way to be sure.

    You'll see the results with your own eyes (or with your own stopwatch), and they'll apply to your situation which random answers on the Internet might not. Put both variants in scripts and run time script1.sh, and time script2.sh. (Do that with a list of empty compressed files to measure the absolute amount of the overhead.)

0

How fast is your disk?

This should use all your CPUs:

parallel -X gzip -d :::: large_file_list

So your limit is likely going to be the speed of your disk.

You can try adjusting with -j:

parallel -j50% -X gzip -d :::: large_file_list

This will run half of the jobs in parallel as the previous command, and will stress your disk less, so depending on your disk this can be faster.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.