2

How to compress all files in the current directory using lz4 using GNU parallel?

I try ls | parallel lz4 but it outputs to stdout. Why? How to fix?

2 Answers 2

1

Use the -m option for multiple input files (implies automatic output filenames).

ls | parallel lz4 -m
3
  • 1) Is there any difference between ls | parallel lz4 -m and parallel lz4 -m ::: *? 2) I don't understand: why there is a need of -m if lz4 is called N times in parallel, each time with a single file name? What do I miss?
    – pmor
    Nov 11, 2022 at 13:15
  • 1) Yes, they're equivalent. 2) I'm not entirely certain but I don't think parallel is necessarily running one file at a time. Nov 11, 2022 at 13:49
  • 2) Both gzip (for example) and lz4 can take a single argument "input file". Then why ls | parallel gzip works (as expected), while ls | parallel lz4 doesn't? Can you guess / explain?
    – pmor
    Nov 14, 2022 at 8:35
0

but it outputs to stdout. Why? How to fix?

The answers are in the man page...
Why it outputs to stdout:

  •   When  no  destination is specified, result is sent on implicit output, which depends on stdout status. When stdout is Not the
       console, it becomes the implicit output. Otherwise, if stdout is the console, the implicit output is filename.lz4.

In this particular case (parallel runs lz4) the stdout is not the console (see steeldriver's answer HERE); also, if you check your output it contains warning(s):

Warning : using stdout as default output. Do not rely on this behavior: use explicit `-c` instead !

How to fix (next paragraph in the man page):

   •   It is considered bad practice to rely on implicit output in scripts. because the script's environment may change. Always  use
       explicit  output in scripts. -c ensures that output will be stdout. Conversely, providing a destination name, or using -m en‐
       sures that the output will be either the specified name, or filename.lz4 respectively.

So the proper way would be to either provide a destination name

parallel lz4 {} {}.lz4 ::: ./*

or use the -m switch (same result but no verbosity):

parallel lz4 -m ::: ./*

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.