Suppose I have four very large text files, all compressed with xz.
file1.log.xz
file2.log.xz
file3.log.xz
file4.log.xz
What I'd like to do is concatenate the uncompressed contents of these four files into a new file file.xz
. The thing is, I would ideally like to not have to go through intermediate files.
The files are very large log files that are gigabytes in size. Compressed, they're under 100MB, but if I were to expand all four files then re-concatenate, I'd need at least 30GB of storage to store the uncompressed files. I could, of course, then cat
all the uncompressed files into xz
to recompress them:
cat file1.log file2.log file3.log file4.log | xz -ve9 - > newfile.log.xz
I know how I could concatenate two files at the command line without an intermediate, assuming one was uncompressed and one was compressed:
xz -d -c file2.log.xz | cat file1.log - | xz -ve9 - > files1and2.log.xz
But this will only work for one file, and one of them has to already be uncompressed.
I'm not sure if I can just cat
the various .xz files together - let's assume they may have been compressed with different parameters.
On a higher level, the question itself could be asked: can you take the output of multiple (more than two) commands, concatenate those outputs, and pipe them into another process without intermediate files? (Hypothetical scenario: imagine I'm doing some kind of processing on all four very huge files using a script that outputs to stdout, and wanting to put the output into another compressed file.)
Is it possible to do this using only shell commands?