2

I have a file named path.txt that contains the directory paths to some files as rows:

../../data/first.gz
../../data/second.gz

I want to read path.txt, read each line, store the content of those files (.gz files) into a new file.

I found a similar question here awk command for reading files that are the contents of another file and this code (file names changed to match my data).

awk '{ while ((getline a < $0) > 0) print a }' path.txt >> newfile

I am new to awk and bash. I do not know how to modify the above code to use zcat or similar to open zip files and print content to newfile. Could someone please help me to modify the code or propose a new one? Thanks in advance.

3 Answers 3

7

Use xargs with zcat (here assuming the GNU implementation for its -r and -d options):

<path.txt xargs -rd'\n' zcat -- >>output

To zcat output of each .gz file into individual output files, you don't really need to use a shell-loops at all here, just call an inline-script as following:

<infile xargs -rd'\n' -I{} sh -c 'zcat -- "$1" >output."${1##*/}"' xargs-sh {}
3
  • Thank you. I may have to use this inside a loop. So I choose this one line code as the answer.
    – MAZ
    Jan 7, 2022 at 22:27
  • 2
    all the other answers work just as well in a loop, AMS. Just replace the > with the >> operator. Jan 7, 2022 at 22:28
  • not really, you don't need a shell-loop here! see the updated answer for that. cc @AMS Jan 8, 2022 at 15:36
5

You don't need awk here, a simple shell loop is enough:

while IFS= read -r gz; do
    zcat < "$gz"
done < path.txt >> newfile

Or, to get each into its own file:

while IFS= read -r gz; do
    zcat < "$gz" > "${gz%.gz}.uncompressed" 
done < path.txt

here stripping a .gz suffix if any, and adding a .uncompressed suffix to avoid clobbering files whose name doesn't end in .gz. You could also set the noclobber option (set -o noclobber) to avoid clobbering existing files.

Using redirection instead of passing the file names as arguments to zcat avoids problems for file names starting with - and also avoids the output file being created if the input file can't be opened.

5

A simple method, since the "cat" in zcat is for "concatenate", and hence zcat takes a list of files to concatenate

zcat -- $(cat path.txt) > newfile

However, that's a bit dangerous, since path.txt might contain paths with spaces – and your bash will think that a space divides arguments. And suddenly, ../path/to/My document.txt.gz becomes ../path/to/My and document.txt.gz, and neither of these two exist!

IFS=$'\n'; zcat -- $(cat path.txt) > newfile

and because you only want that setting to apply to this one line, and not the rest of your shell session:

( IFS=$'\n'; zcat -- $(cat path.txt) > newfile )

A file called * would also be expanded to all the non-hidden file names in the current directory.

You hence want to tell your shell that it should only care about newlines as argument separators, and disable globbing:

( IFS=$'\n'; set -o noglob; zcat -- $(cat path.txt) > newfile )

(By the way, file names can contain newlines as well, but I don't think that's a problem solvable based on your unprocessed file list)

As Stéphane points out, for zsh instead of bash, there's shorter ways to do the same thing:

zcat -f -- ${(f)"$(<path.txt)"}

Because zsh is my "home" shell, I was happy to learn about ${(f)"somestring"}: it splits somestring at newlines; $(< path.txt) effectively does the same as $(cat path.txt), but does not need cat to output a file's content – the functionality is part of the shell.

3
  • Hey Stephane, thanks for the edit. Extracted it into an extra step to make it didactically less steep :) Jan 7, 2022 at 16:27
  • 2
    In zsh: zcat -f -- ${(f)"$(<path.txt)"}. See also readarray -t files < path.txt && zcat -f -- "${files[@]}" in bash. Jan 7, 2022 at 16:34
  • @StéphaneChazelas niiice! Yes, this is better :) Added it to my answer! Jan 7, 2022 at 17:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .