2

I have more than 10 files, the 1st column is the same in all files, I need to collect all the columns 2 in all files against the 1st column in one file. I tried with paste then awk but this print only the columns in the first 10 files

paste p{01..20}.dat | awk '{print $1" "$2" "$4" "$6" "$8" "$10" "$12" "$14" "$16"  "$18" "$20}' > output.dat

example for p01.dat file

0.000 1.3594
0.500 1.3600
1.000 1.3603
1.500 1.3601
2.000 1.3595
2.500 1.3584
3.000 1.3570
3.500 1.3552
4.000 1.3530
4.500 1.3506

all other files have the same format and the same 1st column

  • 1
    can you post the actual testable input example? – RomanPerekhrest Nov 13 '17 at 22:29
  • see the updates – Mohsen El-Tahawy Nov 13 '17 at 22:49
  • 1
    what does mean your collect all the columns 2 in all files against the 1st column - group all values from the 2nd field by the value of the 1st field? Post the expected output for your p01.dat – RomanPerekhrest Nov 13 '17 at 22:56
  • the 1st column does not change from file to file, so I want to print the first column from one file then all columns 2 from all files to generate only one output file. – Mohsen El-Tahawy Nov 13 '17 at 23:25
4

Just a variant of @John1024's solution:

paste -d '=' p*.dat | sed 's/=\S*//g'
  • intelligent solution – Mohsen El-Tahawy Nov 13 '17 at 23:56
  • Quite nicely done. – John1024 Nov 14 '17 at 0:06
  • 1
    MohsenEl-Tahawy and @John1024, thank you (it was a bit of luck). – JJoao Nov 14 '17 at 8:04
2

You need a for loop. Try:

paste p{01..20}.dat | awk '{printf "%s",$1; for (i=2;i<=NF;i=i+2) printf " %s",$i; print ""}'

Example

Let's take three input files:

$ paste p*.dat
1 a     1 aa    1 aaa
2 b     2 bb    2 bbb
3 c     3 cc    3 ccc

Our command generates:

$ paste p*.dat | awk '{printf "%s",$1; for (i=2;i<=NF;i=i+2) printf " %s",$i; print ""}'
1 a aa aaa a a a a a a a a a a a a a a a a a
2 b bb bbb b b b b b b b b b b b b b b b b b
3 c cc ccc c c c c c c c c c c c c c c c c c

Example using 20 files

We'll start with the three files above and create 17 more:

for i in {04..20}; do cp p01.dat p$i.dat; done

We can verify that paste works:

$ paste p{01..20}.dat
1 a     1 aa    1 aaa   1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a     1 a
2 b     2 bb    2 bbb   2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b     2 b
3 c     3 cc    3 ccc   3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c     3 c

We can also verify that the awk command works:

$ paste p{01..20}.dat | awk '{printf "%s",$1; for (i=2;i<=NF;i=i+2) printf " %s",$i; print ""}'
1 a aa aaa a a a a a a a a a a a a a a a a a
2 b bb bbb b b b b b b b b b b b b b b b b b
3 c cc ccc c c c c c c c c c c c c c c c c c
  • this still gives the same problem since paste p*.dat print only the columns in the first 10 files so the command after | will works only on the first 10 files – Mohsen El-Tahawy Nov 13 '17 at 22:47
  • @MohsenEl-Tahawy As the updated answer shows, I cannot reproduce that problem: paste works fine on 20 files. Can you try it again and, if it is still not working, provide more details? – John1024 Nov 13 '17 at 22:53
  • sorry, it was my bad, actually, I was trying to print 11 files but I discovered that one of the files is empty so I thought it prints only 10 files. now why the first column appears as s and not as 1 2 3 – Mohsen El-Tahawy Nov 13 '17 at 23:32
  • Very good. Also, the s was due to a typo: I had "s" where I needed "%s". Answer updated with corrected code. – John1024 Nov 13 '17 at 23:54
2

paste + cut + seq trick:

paste -d' ' p{01..20}.dat | cut -d' ' -f1,"$(seq -s',' 2 2 20)"

Test case (on 3 files):

$ head p0[1-3].dat
==> p01.dat <==
0.000 1.3594
0.500 1.3600
1.000 1.3603
1.500 1.3601
2.000 1.3595
2.500 1.3584
3.000 1.3570
3.500 1.3552
4.000 1.3530
4.500 1.3506

==> p02.dat <==
0.000 2.3594
0.500 2.3600
1.000 2.3603
1.500 2.3601
2.000 2.3595
2.500 2.3584
3.000 2.3570
3.500 2.3552
4.000 2.3530
4.500 2.3506

==> p03.dat <==
0.000 3.3594
0.500 3.3600
1.000 3.3603
1.500 3.3601
2.000 3.3595
2.500 3.3584
3.000 3.3570
3.500 3.3552
4.000 3.3530
4.500 3.3506

paste -d' ' p{01..03}.dat | cut -d' ' -f1,"$(seq -s',' 2 2 6)"
0.000 1.3594 2.3594 3.3594
0.500 1.3600 2.3600 3.3600
1.000 1.3603 2.3603 3.3603
1.500 1.3601 2.3601 3.3601
2.000 1.3595 2.3595 3.3595
2.500 1.3584 2.3584 3.3584
3.000 1.3570 2.3570 3.3570
3.500 1.3552 2.3552 3.3552
4.000 1.3530 2.3530 3.3530
4.500 1.3506 2.3506 3.3506

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.