1

I have basically 4 txt files with records such as

FILE 1

localhost_access_2018-07-19.tar.gz:13

localhost_access_2018-07-20.tar.gz:17

localhost_access_2018-07-21.tar.gz:12

localhost_access_2018-07-22.tar.gz:4

localhost_access_2018-07-23.tar.gz:2

localhost_access_2018-07-24.tar.gz:2905

localhost_access_2018-07-25.tar.gz:10440

localhost_access_2018-07-26.tar.gz:2644

localhost_access_2018-07-27.tar.gz:1896

localhost_access_2018-07-28.tar.gz:1238

localhost_access_2018-07-29.tar.gz:932


FILE 2

localhost_access_2018-06-19.tar.gz:0

localhost_access_2018-06-20.tar.gz:0

localhost_access_2018-06-21.tar.gz:1

localhost_access_2018-06-22.tar.gz:0

localhost_access_2018-06-23.tar.gz:0

localhost_access_2018-06-24.tar.gz:0

localhost_access_2018-06-25.tar.gz:0

localhost_access_2018-06-26.tar.gz:1

localhost_access_2018-06-27.tar.gz:0

localhost_access_2018-07-04.tar.gz:2

localhost_access_2018-07-05.tar.gz:3

localhost_access_2018-07-06.tar.gz:6

localhost_access_2018-07-07.tar.gz:0

localhost_access_2018-07-19.tar.gz:15

etc.

each of the 4 files has redundant dates such in this case localhost_access_2018-07-19.tar.gz:15 but different final numbers (after the :).

For the same dates, I have to: cut "local_host_access_" leaving only the date in a single new txt file (without changing it both in value and format), and I have to cut .tar.gz.

In addition I have to sum each of the values for the dates which are the same:

Hence, if there are 4 "localhost_access_2018-07-19" dates in the 4 different files, I have to sum each of the number XYZ after the value .gz:"XYZ".

Example:

localhost_access_2018-07-19.tar.gz:1

localhost_access_2018-07-19.tar.gz:2

localhost_access_2018-07-19.tar.gz:3

localhost_access_2018-07-19.tar.gz:4

should return in the single output of text, in a new file

2018-07-19:10

I've put my attempt as an answer, but I'd appreciate better solutions (and explanations). Thanks.

1

Try also this simple awk approach:

awk -F"[_.:]" '                                 # set field separator to "_", ".", or ":"
        {SUM[$3] += $NF                         # sum all trailing fields in array indexed by the date
        }
END     {for (s in SUM) print s, SUM[s]         # print the date and the respective sum
        }
' OFS=":" file[123]                             # set output field separator; have shell expand file names 1 - 3
0
awk -F 'localhost_access_' ' 
    {
         n=substr($2,1+index($2,":"));  
         gsub(".tar.gz.*","",$2);
         str[$2]+=n
    }
    END{
        for (i in str){
            print i":"str[i]
        }
    }' node1.txt node2.txt node3.txt node4.txt | sort -to > output.txt

    output_not_sorted=$(cat output.txt);

    # sort output by date

    exit

Let me know if this can be improved.

0

Given the two files that you show at the start of your question, and assuming that they actually don't contain blank lines:

$ awk -F ':' -v OFS=':' '
    { sum[$1]+=$2 }
    END { for (key in sum) {
        split(key,f,"[_.]")
        print f[3],sum[key] } }' file* | sort
2018-06-19:0
2018-06-20:0
2018-06-21:1
2018-06-22:0
2018-06-23:0
2018-06-24:0
2018-06-25:0
2018-06-26:1
2018-06-27:0
2018-07-04:2
2018-07-05:3
2018-07-06:6
2018-07-07:0
2018-07-19:28
2018-07-20:17
2018-07-21:12
2018-07-22:4
2018-07-23:2
2018-07-24:2905
2018-07-25:10440
2018-07-26:2644
2018-07-27:1896
2018-07-28:1238
2018-07-29:932

Use the filename as the key in an associative array called sum and collect the sum for each filename therein. At the end, iterate over the keys of sum and print out the date part of each key together with the sum. The date part of the key is the third field after splitting on dots and underscores.

The result is the piped though sort.


Shorter, but essentially the same as the above (but uses only the date as the key in the sum array):

awk -F '[_.:]' -v OFS=':' '
    { sum[$3]+=$6 }
    END { for (d in sum) print d, sum[d] }' file*
0
#!/bin/bash
# Sum duplicate row values with awk ; Columns 3 and 6
awk -F '[_.:]' '{seen[$3]+=$6}END{for (i in seen) print i, seen[i]}' OFS=":" node[1-4].txt | 

sort > log.txt

Source for "# Sum duplicate row values with awk" https://stackoverflow.com/questions/42024296/sum-duplicate-row-values-with-awk

  • Thanks for your answer. Is this a useful use of cat? – user175892 Oct 23 '18 at 14:13
  • Thanks for your comment. No! I've removed sed and the first sort too. – Gounou Oct 23 '18 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy