I have a text like this pattern on many lines. Can I group only the values and the name into a single line?

ABCDEFG_10_node10:2154  ABCDEFG_10_node10:54
ABCDEFG_10_node10:2254  ABCDEFG_10_node10:64
ABCDEFG_10_node10:410 ABCDEFG_10_node10:10
ABCDEFG_10_node10:210 ABCDEFG_10_node10:10 
XXYZZ_71_node2:24: XXYZZ_71_node2:504:
X3y5z_53_node1:664: X3y5z_53_node1:990:
RCTY_11_node2:224: RCTY_11_node2:234:

Output expected:

ABCDEFG_10_node10: 2154,2254,410,210,365,890,741,54,64,10,10
XXYZZ_71_node2: 24,504
X3y5z_53_node1: 664,990
RCTY_11_node2: 224,234

I’m on AIX. How can I do this?

  • 2
    Do you specifically insist on that order,  or would 2154,54,2254,64,410,10,210,10,365,890,741 be acceptable? – Scott Oct 6 '17 at 10:47
  • any order is acceptable scott. – satsensort Oct 6 '17 at 11:05
  • @satsensort, you have added some content to your input. Does it really contain trailing semicolons after some values XXYZZ_71_node2:24: , X3y5z_53_node1:990:? or it's just a typo? – RomanPerekhrest Oct 6 '17 at 11:16
  • Please make sure your examples accurately represent your file. Your new data have the extra : and also extra whitespace. Do we need to deal with that as well? – terdon Oct 6 '17 at 11:20
  • If you're happy with one or several of the answers, upvote them. If one is solving your issue, accepting it would be the best way of saying "Thank You!" :-) – Kusalananda Oct 12 '17 at 19:43

A Perl approach (assuming the order isn't important):

$ perl -lne 'while(/(\w+):(\d+)/g){
                push @{$k{$1}},$2
                print "$_: " . join ",", @{$k{$_}} for keys %k
             }' file 
ABCDEFG_10_node10: 2154,54,2254,64,410,10,210,10,365,890,741

That reads the input file line by line (-ln) and runs the script given by -e on it. The while(/(\w+):(\d+)/g) will collect all instances of non-whitespace, followed by a : and then more non-whitespace. Since they are being captured in parentheses, the $1 will be the name and the $2 the value. These are then pushed into a hash of arrays (hash %k, whose values are arrays). Finally, we print each key of the hash (the names) along with the array of its values, joined by ,.

If you value conciseness, you can do write the above as a one-liner:

perl -lne 'while(/(\S+):(\S+)/g){push @{$k{$1}},$2}}{$"=",";print"$_: @{$k{$_}}" for keys%k' file

pushed past legibility:

perl -nE'push@{$k{$1}},$2while/(\w+):(\d+)/g}{$"=",";say"$_: @{$k{$_}}"for keys%k' file

Awk solution:

awk -F':|[[:space:]]+' '{ 
         a[$1]=a[$1]? a[$1]","$2:$2; if(NF==4) b[$3]=b[$3]? b[$3]","$4:$4 
     END{ for(i in a) printf "%s: %s%s\n",i,a[i],(i in b)? ","b[i]:"" }' file

  • -F':|[[:space:]]+' - complex field separator

  • a[$1]=a[$1]? a[$1]","$2:$2 - grouping values for each unique name ABCDEFG...

  • if(NF==4) b[$3]=b[$3]? b[$3]","$4:$4 - if there is an additional right-side section - group values into additional array b

The output:

ABCDEFG_10_node10: 2154,2254,410,210,365,890,741,54,64,10


If the order of values isn't important the above approach could be slightly simplified:

awk -F':|[[:space:]]+' '{ 
        a[$1]=a[$1]? a[$1]","$2:$2; if(NF==4) a[$3]=a[$3]? a[$3]","$4:$4 
     END{ for(i in a) print i":",a[i] }' file
  • @satsensort, you're welcome – RomanPerekhrest Oct 6 '17 at 12:00
  • You may be able to simplify the awk by formatting the input to one column: fmt -w 1 input | sed 's/:$//' | awk ... (I also stripped off the trailing colon characters) – Jeff Schaller Oct 11 '17 at 1:46
awk '{ for (i=1;i<=NF;i++) { split($NF,arr,":");if (dat[arr[1]]=="") { dat[arr[1]]=arr[2] } else { dat[arr[1]]=dat[arr[1]]","arr[2] } } } END { for ( i in dat ) { print i": "dat[i] } }' filename

An alternative awk solution to Roman's where we take each space delimited piece of data in turn and then further split the data using the split function in the array arr based on the character : We then build an array keyed on the ABC etc string with the string of numbers to be printed. We then loop through this array (dat) building a string starting with the key along with : and the string. This is then printed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.