Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Say I have several file with the following naming:

000001
000002
000003
...

Each of these files is a csv file (may include escape characters). In total the folder has ~20GB of data. How can I stitch these files together into a single final file?

In case it matters I usually use Zsh.

share|improve this question
    
Is order of content in final file preserve, meaning content of file1 then content of file2... –  Gnouc Aug 13 at 20:04
    
@Gnouc - Yes. We want to preserve ordering. –  user815423426 Aug 13 at 20:24

3 Answers 3

up vote 5 down vote accepted
setopt extendedglob
cat <->.csv > all.csv

Where <-> matches any positive integer decimal number, will concatenate all those (in lexical order, which for 0 padded numbers is the same as numerical order) into all.csv.

That will double the space on disk though. If you don't intend to keep the original files, you could do:

for i in <->.csv; do
  cat $i && rm -f $i || break
done > all.csv
share|improve this answer
    
Thanks but I am getting zsh: no matches found: <->.csv with the first solution. –  user815423426 Aug 13 at 21:01
1  
@user815423426, <->.csv matches the files that are named like 0000001.csv (or any-positive-decimal-integer-number.csv as I said). If they're called 000001, replace it with <->. –  Stéphane Chazelas Aug 13 at 21:08
    
Oh man. I can't believe I missed that.. Sorry for the confusion. –  user815423426 Aug 13 at 21:14

If you use redirection, that will either append or overwrite contents to one file. If you want to append to one file, use:

cat file.csv file2.csv file3.csv >> all.csv

This next command will overwrite to all.csv:

cat file.csv file2.csv file3.csv > all.csv

But say you want to move all CSV files to one csv in a given directory (to append to):

cat *.csv >> all.csv

share|improve this answer

Why do you want to combine all those files into one file?

  1. If you want to feed them to a program that wants stdin, you can do this:

    cat *.csv | prog
    

    or any of the other combination methods mentioned earlier.

  2. If the program expects a named file as input, do something like this:

    mkfifo p
    cat *.csv > p &
    prog p
    
  3. If you want one file because one file is easier to backup, then tarring everything together can be useful. If you want to read the files w/o untarring them, then you can turn the tar file into a fuse-based fs.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.