1

My Linux directory contains a dump of files and they look like:

EDW_Infile_ABC_Daily_Activity_20190204.csv
EDW_Infile_ABC_Daily_Activity.zip
EDW_Infile_PQRInc_Daily_Activity_20190204.csv
EDW_Infile_PQRInc_Daily_Activity_zip
EDW_Infile_ABC_Daily_Payment_20190204.csv
EDW_Infile_PQRInc_Daily_Payment_20190204.csv
EDW_Infile_ABC_Daily_Status_20190204.csv
EDW_Infile_PQRInc_Daily_Status_20190204.csv

These files follow few common name patterns such as

EDW_Infile_*<3 to 8 bytes company name>*_Daily_Activity_*YYYYMMDD*.csv
EDW_Infile_*<3 to 8 bytes company name>*_Daily_Payment_*YYYYMMDD*.csv
EDW_Infile_*<3 to 8 bytes company name>*_Daily_Status_*YYYYMMDD*.csv

How can I -

1) Find all files for all customers, for all dates, which follow the pattern EDW_Infile_{3 to 8 bytes any name}_Daily_Activity_{Any Date}.csv

2) Each file contains a header. How can I combine all of them into one file and have only one header

1

I pushed my zsh knowledge a bit in order to answer more specifically, in case you weren't in control of the filenames and had files named like EDQ_Infile_some uninteresting stuff here_Daily_Activity_junk here.csv and so didn't want to use a * wildcard.

To gather the list of filenames ...

which follow the pattern EDW_Infile_{3 to 8 bytes any name}_Daily_Activity_{Any Date}.csv

I would set up this extended_glob pattern in zsh (don't type the $ -- that's the shell prompt):

$ set -o extended_glob
$ files=(EDW_Infile_?(#c3,8)_Daily_Activity_[[:digit:]](#c8).csv)

The pattern, apart from the plain text, is:

  • ? -- any (single) character
  • (#c3,8) -- require between three and eight characters, inclusive
  • [[:digit:]] -- require a digit
  • (#c8) -- require eight of them

See the list with:

$ print -l $files
EDW_Infile_ABC_Daily_Activity_20190204.csv
EDW_Infile_PQRInc_Daily_Activity_20190204.csv

To then ...

combine all of them into one file and have only one header

{ head -1 "${files[1]}"; for f in $files; do sed 1d "$f"; done; } > output.csv

This groups two commands and redirects their output to output.csv. The first command, head, takes the first line from the first file in the array; the second command then loops through all of the files and deletes the first line (default-printing the remainder to stdout).

0

You might want something like this

# collect all the "EDW_Infile_ABC" prefixes
declare -A prefix
for f in EDQ_Infile_*_Daily_Activity_*.csv; do
    p=${f%_*.csv}
    prefix[$p]=1
done

for p in "${!prefixes[@]}"; do
    awk 'NR==1 {print} FNR==1{next} {print}' "$p"_*.csv > "$p"_all.csv
    zip "$p".zip "$p"_all.csv
    rm  "$p"_all.csv
done

For bash, requires version 4 for associative arrays. Otherwise, we can work with positional parameters.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.