How to concatenate results of multiple commands and pipe into another without intermediate file?

Question

Suppose I have four very large text files, all compressed with xz.

file1.log.xz
file2.log.xz
file3.log.xz
file4.log.xz

What I'd like to do is concatenate the uncompressed contents of these four files into a new file file.xz. The thing is, I would ideally like to not have to go through intermediate files.

The files are very large log files that are gigabytes in size. Compressed, they're under 100MB, but if I were to expand all four files then re-concatenate, I'd need at least 30GB of storage to store the uncompressed files. I could, of course, then cat all the uncompressed files into xz to recompress them:

cat file1.log file2.log file3.log file4.log | xz -ve9 - > newfile.log.xz

I know how I could concatenate two files at the command line without an intermediate, assuming one was uncompressed and one was compressed:

xz -d -c file2.log.xz | cat file1.log - | xz -ve9 - > files1and2.log.xz

But this will only work for one file, and one of them has to already be uncompressed.

I'm not sure if I can just cat the various .xz files together - let's assume they may have been compressed with different parameters.

On a higher level, the question itself could be asked: can you take the output of multiple (more than two) commands, concatenate those outputs, and pipe them into another process without intermediate files? (Hypothetical scenario: imagine I'm doing some kind of processing on all four very huge files using a script that outputs to stdout, and wanting to put the output into another compressed file.)

Is it possible to do this using only shell commands?

Answer 1

The xz documentation says

It is possible to concatenate .xz files as is. xz will decompress such files as if they were a single .xz file.

From my tests, this works even if the different files are compressed with different options; so in your case

cat -- *.log.xz > newfile.log.xz

will work fine.

To answer your more general question, you can pipe the output of a compound command, e.g.

for file in -- *.log.xz; do xzcat -- "$file"; done | xz -ve9 > newfile.log.xz

or any subshell. This would allow you to perform any processing you want to on your log files before recompressing them. However in the basic case this isn’t necessary either; you can decompress and recompress all your files by running

xzcat -- *.log.xz | xz -ve9 > newfile.log.xz

If you add -f this even works with uncompressed files, so

xzcat -f -- uncompressed.log *.log.xz | xz -ve9 > newfile.log.xz

would allow you to combine uncompressed and compressed logs.

Answer 2

try

for x in *.log.xz
do
  xz -d -c "$x"
done | xz -ve9 - > newfile.log.xz

(this can be onlined of course).

to add a new uncompresed file, use a sub shell ( () )

( cat newfile.log 
for x in *.log.xz
do
  xz -d -c "$x"
done ) | xz -ve9 - > newfile.log.xz

Answer 3

xzcat -f is the answer to the first part of your question. But you are right: You cannot simply cat *xz | xzcat if some of your files are compressed with -F lzma.

On a higher level, the question itself could be asked: can you take the output of multiple (more than two) commands, concatenate those outputs, and pipe them into another process without intermediate files?

The problem here is: If you do not store the intermediate output in files where do you store it?

If you store it in RAM, you are limited by the amount of free RAM. If you go above that, you machine is quickly going down the road to swaphell.

GNU Parallel stores in temporary files, but if you put these in a tmpfs file system, they are basically stored in RAM:

mkdir mytmp    
sudo mount tmpfs mytmp -t tmpfs -o rw,size=3P
parallel --tmpdir mytmp seq {}00000000 {}99999999 ::: 1 2 | grep 0000000

If, however, it is acceptable to mix the output on a line-by-line basis, then you only need to store a single line from each of the running programs in RAM.

This is what GNU Parallel (> version 20170822) does:

parallel --lb seq {}00000000 {}99999999 ::: 1 2 | grep 0000000

A third solution is to compress the temporary files using a fast compressor (e.g. pzstd, pigz, lz4, lzop):

parallel --compress seq {}00000000 {}99999999 ::: 1 2 | grep 0000000

(GNU Parallel autodetects which fast compressor you have installed).

Answer 4

Although @Archemar touches on it, it doesn't seem as though anyone has yet actually answered the question in the title straight-up:

How to concatenate results of multiple commands and pipe into another without intermediate file?

And re-stated in your post:

On a higher level, the question itself could be asked: can you take the output of multiple (more than two) commands, concatenate those outputs, and pipe them into another process without intermediate files?

The general way to do what you ask, as Archemar hints, is to use a sub-shell.

Bash syntax:

(
  command_one
  command_two
  command_three
...
  command_N
) | next_command