Text Processing using sed/awk/perl

Question

I have a text like this pattern on many lines. Can I group only the values and the name into a single line?

ABCDEFG_10_node10:2154  ABCDEFG_10_node10:54
ABCDEFG_10_node10:2254  ABCDEFG_10_node10:64
ABCDEFG_10_node10:410 ABCDEFG_10_node10:10
ABCDEFG_10_node10:210 ABCDEFG_10_node10:10 
ABCDEFG_10_node10:365
ABCDEFG_10_node10:890
ABCDEFG_10_node10:741
XXYZZ_71_node2:24: XXYZZ_71_node2:504:
X3y5z_53_node1:664: X3y5z_53_node1:990:
RCTY_11_node2:224: RCTY_11_node2:234:

Output expected:

ABCDEFG_10_node10: 2154,2254,410,210,365,890,741,54,64,10,10
XXYZZ_71_node2: 24,504
X3y5z_53_node1: 664,990
RCTY_11_node2: 224,234

I’m on AIX. How can I do this?

Answer 1

A Perl approach (assuming the order isn't important):

$ perl -lne 'while(/(\w+):(\d+)/g){
                push @{$k{$1}},$2
             } 
             END{
                print "$_: " . join ",", @{$k{$_}} for keys %k
             }' file 
ABCDEFG_10_node10: 2154,54,2254,64,410,10,210,10,365,890,741

That reads the input file line by line (-ln) and runs the script given by -e on it. The while(/(\w+):(\d+)/g) will collect all instances of non-whitespace, followed by a : and then more non-whitespace. Since they are being captured in parentheses, the $1 will be the name and the $2 the value. These are then pushed into a hash of arrays (hash %k, whose values are arrays). Finally, we print each key of the hash (the names) along with the array of its values, joined by ,.

If you value conciseness, you can do write the above as a one-liner:

perl -lne 'while(/(\S+):(\S+)/g){push @{$k{$1}},$2}}{$"=",";print"$_: @{$k{$_}}" for keys%k' file

pushed past legibility:

perl -nE'push@{$k{$1}},$2while/(\w+):(\d+)/g}{$"=",";say"$_: @{$k{$_}}"for keys%k' file

Answer 2

Awk solution:

awk -F':|[[:space:]]+' '{ 
         a[$1]=a[$1]? a[$1]","$2:$2; if(NF==4) b[$3]=b[$3]? b[$3]","$4:$4 
     }
     END{ for(i in a) printf "%s: %s%s\n",i,a[i],(i in b)? ","b[i]:"" }' file

---

• -F':|[[:space:]]+' - complex field separator

• a[$1]=a[$1]? a[$1]","$2:$2 - grouping values for each unique name ABCDEFG...

• if(NF==4) b[$3]=b[$3]? b[$3]","$4:$4 - if there is an additional right-side section - group values into additional array b

---

The output:

ABCDEFG_10_node10: 2154,2254,410,210,365,890,741,54,64,10

----------

If the order of values isn't important the above approach could be slightly simplified:

awk -F':|[[:space:]]+' '{ 
        a[$1]=a[$1]? a[$1]","$2:$2; if(NF==4) a[$3]=a[$3]? a[$3]","$4:$4 
     }
     END{ for(i in a) print i":",a[i] }' file

Answer 3

awk '{ for (i=1;i<=NF;i++) { split($NF,arr,":");if (dat[arr[1]]=="") { dat[arr[1]]=arr[2] } else { dat[arr[1]]=dat[arr[1]]","arr[2] } } } END { for ( i in dat ) { print i": "dat[i] } }' filename

An alternative awk solution to Roman's where we take each space delimited piece of data in turn and then further split the data using the split function in the array arr based on the character : We then build an array keyed on the ABC etc string with the string of numbers to be printed. We then loop through this array (dat) building a string starting with the key along with : and the string. This is then printed.

Answer 4

Using Raku (formerly known as Perl_6)

raku -e 'say .key, " => ", .value.words[1,3,5...*] for lines.split(/<[:\s]>/, :skip-empty).rotor(2).classify( *.[0]);'

Sample Input:

ABCDEFG_10_node10:2154  ABCDEFG_10_node10:54
ABCDEFG_10_node10:2254  ABCDEFG_10_node10:64
ABCDEFG_10_node10:410 ABCDEFG_10_node10:10
ABCDEFG_10_node10:210 ABCDEFG_10_node10:10 
ABCDEFG_10_node10:365
ABCDEFG_10_node10:890
ABCDEFG_10_node10:741
XXYZZ_71_node2:24: XXYZZ_71_node2:504:
X3y5z_53_node1:664: X3y5z_53_node1:990:
RCTY_11_node2:224: RCTY_11_node2:234:

Sample Output:

XXYZZ_71_node2 => (24 504)
RCTY_11_node2 => (224 234)
ABCDEFG_10_node10 => (2154 54 2254 64 410 10 210 10 365 890 741)
X3y5z_53_node1 => (664 990)

Briefly, lines are read in, destructively split on : and \s (omitting any empty elements via :skip-empty), rotor (join) every 2 elements, and classify-ing by the first of each pair. [I could make the call .classify( *.[0].unique) to clarify the intent, but Raku does the right thing either way].

If the OP really wants the format specified in his/her post, then substitute => with : and add a call to join at the end of the .value call, so the whole section prior to for reads: say .key, ": ", .value.words[1,3,5...*].join(",")

Sample Output (2):

X3y5z_53_node1: 664,990
RCTY_11_node2: 224,234
ABCDEFG_10_node10: 2154,54,2254,64,410,10,210,10,365,890,741
XXYZZ_71_node2: 24,504

https://docs.raku.org/routine/classify
https://raku.org