how to add output as a new column with the file names

Question

How can I add output of each file incrementally in one singel output? I want to do this instead of running paste command on all files together. It is because I have 10k files and each file is 100 GB in size.

file1

a       1
b       2
c       3

file2

a       10
b       20
c       40

file3

a       0
b       0
c       0

Desired output

file1   file1   file2   file2   file3   file3
a       1       a       10      a       0
b       2       b       20      b       0
c       3       c       40      c       0

I know I can get some thing similar to desired output using paste -d "\t" file{1..3} but I want to perform operation one file after another but not all together and importantly I want to keep the file names.

Answer 1

paste command is a good choice if we just need to merge lines of files.

To prepend header line with filenames use combination awk + paste:

{ for f in file*; do awk '{ for(i=1;i<=NF;i++) printf("%s\t",FILENAME); exit }' "$f"; done; 
echo ""; paste -d"\t" file*; } | column -t

The output (for 3 input files):

file1  file1  file2  file2  file3  file3
a      1      a      10     a      0
b      2      b      20     b      0
c      3      c      40     c      0

---

Details:

• { command; command; ...} - used to combine outputs of multiple commands

• for f in file*; - for each file

• printf("%s\t",FILENAME) - print filename for each column of respective file

• exit - exits immediately after processing the 1st line