0

My objective is to move into a subdirectory, call it SUB, which has many text files with 5 columns each and always the same number of rows. I want one large 'sum' file that adds all columns from 2-5, keeps the 1st column, and outputs the file. If I know the number of files is small, this can be done as follows:

cd SUB

numfiles=(*)
numfiles=${#numfiles[@]}

if [ $numfiles = 1 ]
then
paste * | awk '{print $1, $2, $3, $4, $5}' > sum
fi

if [ $numfiles = 2 ]
then
paste * | awk '{print $1, ($2+$7), ($3+$8), ($4+$9), ($5+$10)}' > sum
fi    

However, since the number of files in SUB could be anywhere between 1 and 100, this is a cumbersome method. Your help is appreciated!

P.S. The formatting of the new 'sum' file should be the same as the format of the input text files, that all look the same. For example, if SUB contains two files with entries

1 2 3 4 5
2 2 3 4 5

and

1 4 2 7 1
2 2 5 1 9

then the 'sum' file should be

1 6 5 11 6
2 4 8 5 14
  • You'd better to show example of text file and desired output (how you want to see information in sum.file) – Costas Feb 2 '15 at 23:21
  • That should help clarify things. – LionHead Feb 2 '15 at 23:27
  • Are source files columns space separated or tab separated? For the results file which would you prefer? Does it have to be entirely awk or is the use of some shell script acceptable? Since you're using paste, could I use perl? – roaima Feb 2 '15 at 23:28
  • Can you guarantee that the first column of numbers is exactly the same in all files? If not, what should happen in the output file when the values differ? – roaima Feb 2 '15 at 23:31
  • The first column is guaranteed to have all the same numbers by construction. Space separated. Space separated for results file. Shell script and awk are both acceptable. Perl as well. – LionHead Feb 2 '15 at 23:32
3
awk '
    NR==FNR {
        # assuming column 1 is the same for all files, remember it.
        # only need to do this for the first file
        col1[FNR] = $1
    }
    { for (i=2; i<=NF; i++) sum[FNR,i] += $i }
    END {
        for (i=1; i<=FNR; i++) {
            printf "%s", col1[i]
            for (j=2; j<=NF; j++) printf "%s%s", OFS, sum[i,j]
            print ""
        }
    }
' file1 file2
1 6 5 11 6
2 4 8 5 14

Should work with nawk, gawk and mawk.

  • What do think to printf OFS"%s", sum[i,j] ? – Costas Feb 3 '15 at 0:06
  • That, to me, is less clear, but it's functionally the same. – glenn jackman Feb 3 '15 at 0:13
1

Here you go. This actually uses gawk rather than vanilla awk. If you really need to use true awk I think you'll need to change the h[x][y] construction to something like h[x ";" y].

cd SUB
awk '
    BEGIN { PROCINFO["sorted_in"] = "@ind_num_asc" }
    { h[FNR][1] = $1; for (n=2; n<=5; n++) { h[FNR][n] += $n } }
    END { for (r in h) { print h[r][1], h[r][2], h[r][3], h[r][4], h[r][5] } }
' *

With your sample input it provides your expected output.

I've added the PROCINFO setting as recommended in the comments.

  • It almost works, except at least in my gawk it rearranges all the rows. Thanks though! – LionHead Feb 2 '15 at 23:50
  • The for (r in h) handles the array as a associative array, for which there is no defined order. Add this statement before the for command: PROCINFO["sorted_in"] = "@ind_num_asc" to impose a numerically ascending order to the array indices. See gnu.org/software/gawk/manual/gawk.html#Controlling-Scanning – glenn jackman Feb 2 '15 at 23:53
  • Also, in the first line, you can just loop from 1 to 5, no need to separate the first column since you don't do anything different with it. – glenn jackman Feb 2 '15 at 23:54
  • @glenn I like your solution, which was much better generalised without additional complexity. Thanks for the feedback. – roaima Feb 3 '15 at 0:04
  • @lionhead the PROCINFO as recommended by Glenn Jackman fixes the ordering. Although I appreciate you prefer his solution :-) – roaima Feb 3 '15 at 0:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.