Concatenate multiple zipped files, skipping header lines in all but the first file

Question

I have a collection of gzipped files that I want to combine into a single file. They each have identical format. I want to keep the header information from only the first file and skip it in the subsequent files.

As a simple example, I have four identical files with the following content:

$ gzcat file1.gz
# header
1
2

I want to end up with

# header
1
2
1
2
1
2
1
2

In reality, I can have a varying number of files so I would like to be able to do this programatically. Here is the non-programatic solution I have so far...

cat <(gzcat file1.gz) <(tail -q -n +2 <(gzcat file2.gz) <(gzcat file3.gz) <(gzcat file4.gz))

This command works, but it is “hard coded” to handle four files, and I need to generalize it for any number of files.  I am using bash as the shell if that helps. My preference is for performance (in reality the files can be millions of lines long), so I am OK with a less-than-elegant solution if it is speedy.

Answer 1

If the command that you show in your question basically works (for a hard-coded number of files), then

first=1
for f in file*.gz
do
    if [ "$first" ]
    then
        gzcat "$f"
        first=
    else
        gzcat "$f"| tail -n +2
    fi
done > collection_single_file

should work for you.  I hope the logic is fairly clear.  Look at all the files (change the wildcard as appropriate for your file names).  If it’s the first one in the list, gzcat it, so you get the entire file (including the header).  Otherwise, use tail to strip the header.  After you’ve handled a file, then no other file will be the first.

This invokes tail N−1 times, instead of just once (like your answer).  Aside from that, my answer should perform the same as your answer.

Answer 2

A variation on G-Man's solution that does not use a separate variable to keep track of the first file:

set -- file*.gz

{
    gzcat "$1"; shift

    for file do
        gzcat "$file" | sed '1d'
    done
} >combined.txt

This uncompresses the first file and then loops over the remaining ones, passing each through a short sed script that deletes the first line. The output is redirected to combined.txt.

The set -- file*.gz command sets the positional parameters ($1, $2, etc., that collectively is the array $@) to the filenames matching the given pattern. The shift removes the $1 from the array after uncompressing it. The loop loops over the remaining filenames in the array and could also have been written

for file in "$@"; do
    gzcat "$file" | sed '1d'
done

The { ... } allows us to redirect the output of the commands within to a file in one go.

---

Even shorter, with the additional assumption that a "header line" is always starting with a # character (as in the example in the question), and that there are no other such lines in the data:

gzcat file*.gz | awk 'NR > 1 && /^#/ { next } 1' >combined.txt

or,

gzcat file*.gz | sed '2,${ /^#/d; }' >combined.txt

Both of these skips any line starting with # if it occurs on the second line or later in the combined contents of the uncompressed data.