1

How can I add output of each file incrementally in one singel output? I want to do this instead of running paste command on all files together. It is because I have 10k files and each file is 100 GB in size.

file1

a       1
b       2
c       3

file2

a       10
b       20
c       40

file3

a       0
b       0
c       0

Desired output

file1   file1   file2   file2   file3   file3
a       1       a       10      a       0
b       2       b       20      b       0
c       3       c       40      c       0

I know I can get some thing similar to desired output using paste -d "\t" file{1..3} but I want to perform operation one file after another but not all together and importantly I want to keep the file names.

1

paste command is a good choice if we just need to merge lines of files.

To prepend header line with filenames use combination awk + paste:

{ for f in file*; do awk '{ for(i=1;i<=NF;i++) printf("%s\t",FILENAME); exit }' "$f"; done; 
echo ""; paste -d"\t" file*; } | column -t

The output (for 3 input files):

file1  file1  file2  file2  file3  file3
a      1      a      10     a      0
b      2      b      20     b      0
c      3      c      40     c      0

Details:

  • { command; command; ...} - used to combine outputs of multiple commands

  • for f in file*; - for each file

  • printf("%s\t",FILENAME) - print filename for each column of respective file

  • exit - exits immediately after processing the 1st line

  • thanks but i get the following error because i have files with 200k rows each rows and I have 10 k files. Error - column: line 95075 is too long, output will be truncated . – user1703276 Jun 3 '17 at 14:51
  • @user1703276 "10K files" means that you'll get 20K columns (pretty long). Try without | column -t – RomanPerekhrest Jun 3 '17 at 16:07
  • Is it possible to remove the duplicate columns (odd columns) except the first column? thanks – user1703276 Jun 4 '17 at 17:03
  • @user1703276, you can create a new question for your new condition – RomanPerekhrest Jun 4 '17 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.