Consider a large number of CSV files (*.csv) living in some folder. They all have the same exact header.

How can I efficiently concatenate them all into a single CSV file with the same single header?

I found a number of solutions that solve similar but more specific problems.

The current awk solution doesn't work.

$ cat concat_my_csv_files.sh
    #!/usr/bin/env zsh
    awk '
        FNR==1 && NR!=1 { while (/^<header>/) getline; }
        1 {print}
    ' $1/*.csv > $2

$ ./concat_my_csv_files /some/path/to/csv/files/ full_join.csv

when I do:

grep -F column_A full_join.csv

I see several rows having it.


2 Answers 2

awk '
    NR == 1 {print}
    FNR == 1 {next}
' *.csv

The NR variable is the record number of all the input.
The FNR variable is the record number of only the current file.

This prints the first line seen by awk (the header of the first file), then will skip the first line of each file, printing all the other lines.

  • 2
    Note, this can be logically shortened to: awk 'NR == 1 || FNR > 1' *.csv Commented Oct 5, 2015 at 21:25
  • How are we supposed to help you when you don't show us what your files look like? In the absence of actual information, I'm assuming your "header" is one line at the top of each file. Commented Oct 6, 2015 at 1:07

Basically you want "head -n 1 firstorany.csv; tail -n +2 *.csv".

set -- *.csv
head -n 1 "$1"
tail -n +2 "$@"

If you have *.csv as arguments in a sh script, omit the first line.

  • Thanks. How do I control where the output is written here? (e.g. see concat_my_csv_files.sh in the OP update). Commented Oct 5, 2015 at 22:33
  • 1
    ( head ...; tail ... ) > output.csv Commented Oct 7, 2015 at 14:44

Not the answer you're looking for? Browse other questions tagged .