Text Processing using sed/awk/perl

Question

I have a text like this pattern on many lines. Can I group only the values and the name into a single line?

ABCDEFG_10_node10:2154  ABCDEFG_10_node10:54
ABCDEFG_10_node10:2254  ABCDEFG_10_node10:64
ABCDEFG_10_node10:410 ABCDEFG_10_node10:10
ABCDEFG_10_node10:210 ABCDEFG_10_node10:10 
ABCDEFG_10_node10:365
ABCDEFG_10_node10:890
ABCDEFG_10_node10:741
XXYZZ_71_node2:24: XXYZZ_71_node2:504:
X3y5z_53_node1:664: X3y5z_53_node1:990:
RCTY_11_node2:224: RCTY_11_node2:234:

Output expected:

ABCDEFG_10_node10: 2154,2254,410,210,365,890,741,54,64,10,10
XXYZZ_71_node2: 24,504
X3y5z_53_node1: 664,990
RCTY_11_node2: 224,234

I’m on AIX. How can I do this?

Answer 1

A Perl approach (assuming the order isn't important):

$ perl -lne 'while(/(\w+):(\d+)/g){
                push @{$k{$1}},$2
             } 
             END{
                print "$_: " . join ",", @{$k{$_}} for keys %k
             }' file 
ABCDEFG_10_node10: 2154,54,2254,64,410,10,210,10,365,890,741

That reads the input file line by line (-ln) and runs the script given by -e on it. The while(/(\w+):(\d+)/g) will collect all instances of non-whitespace, followed by a : and then more non-whitespace. Since they are being captured in parentheses, the $1 will be the name and the $2 the value. These are then pushed into a hash of arrays (hash %k, whose values are arrays). Finally, we print each key of the hash (the names) along with the array of its values, joined by ,.

If you value conciseness, you can do write the above as a one-liner:

perl -lne 'while(/(\S+):(\S+)/g){push @{$k{$1}},$2}}{$"=",";print"$_: @{$k{$_}}" for keys%k' file

pushed past legibility:

perl -nE'push@{$k{$1}},$2while/(\w+):(\d+)/g}{$"=",";say"$_: @{$k{$_}}"for keys%k' file

Answer 2

Awk solution:

awk -F':|[[:space:]]+' '{ 
         a[$1]=a[$1]? a[$1]","$2:$2; if(NF==4) b[$3]=b[$3]? b[$3]","$4:$4 
     }
     END{ for(i in a) printf "%s: %s%s\n",i,a[i],(i in b)? ","b[i]:"" }' file

---

• -F':|[[:space:]]+' - complex field separator

• a[$1]=a[$1]? a[$1]","$2:$2 - grouping values for each unique name ABCDEFG...

• if(NF==4) b[$3]=b[$3]? b[$3]","$4:$4 - if there is an additional right-side section - group values into additional array b

---

The output:

ABCDEFG_10_node10: 2154,2254,410,210,365,890,741,54,64,10

----------

If the order of values isn't important the above approach could be slightly simplified:

awk -F':|[[:space:]]+' '{ 
        a[$1]=a[$1]? a[$1]","$2:$2; if(NF==4) a[$3]=a[$3]? a[$3]","$4:$4 
     }
     END{ for(i in a) print i":",a[i] }' file

Answer 3

awk '{ for (i=1;i<=NF;i++) { split($NF,arr,":");if (dat[arr[1]]=="") { dat[arr[1]]=arr[2] } else { dat[arr[1]]=dat[arr[1]]","arr[2] } } } END { for ( i in dat ) { print i": "dat[i] } }' filename

An alternative awk solution to Roman's where we take each space delimited piece of data in turn and then further split the data using the split function in the array arr based on the character : We then build an array keyed on the ABC etc string with the string of numbers to be printed. We then loop through this array (dat) building a string starting with the key along with : and the string. This is then printed.