1

How can I add output of each file incrementally in one singel output? I want to do this instead of running paste command on all files together. It is because I have 10k files and each file is 100 GB in size.

file1

a       1
b       2
c       3

file2

a       10
b       20
c       40

file3

a       0
b       0
c       0

Desired output

file1   file1   file2   file2   file3   file3
a       1       a       10      a       0
b       2       b       20      b       0
c       3       c       40      c       0

I know I can get some thing similar to desired output using paste -d "\t" file{1..3} but I want to perform operation one file after another but not all together and importantly I want to keep the file names.

1 Answer 1

1

paste command is a good choice if we just need to merge lines of files.

To prepend header line with filenames use combination awk + paste:

{ for f in file*; do awk '{ for(i=1;i<=NF;i++) printf("%s\t",FILENAME); exit }' "$f"; done; 
echo ""; paste -d"\t" file*; } | column -t

The output (for 3 input files):

file1  file1  file2  file2  file3  file3
a      1      a      10     a      0
b      2      b      20     b      0
c      3      c      40     c      0

Details:

  • { command; command; ...} - used to combine outputs of multiple commands

  • for f in file*; - for each file

  • printf("%s\t",FILENAME) - print filename for each column of respective file

  • exit - exits immediately after processing the 1st line

4
  • thanks but i get the following error because i have files with 200k rows each rows and I have 10 k files. Error - column: line 95075 is too long, output will be truncated . Commented Jun 3, 2017 at 14:51
  • @user1703276 "10K files" means that you'll get 20K columns (pretty long). Try without | column -t Commented Jun 3, 2017 at 16:07
  • Is it possible to remove the duplicate columns (odd columns) except the first column? thanks Commented Jun 4, 2017 at 17:03
  • @user1703276, you can create a new question for your new condition Commented Jun 4, 2017 at 17:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .