1

I have a text file with format word @@@ type @@@ sentence on every line, sorted by 'word' in ascending order. Some lines however are not unique and they begin with the same word as the previous line, ie see word1 below:

...
word0 @@@ type2 @@@ sentence0
word1 @@@ type1 @@@ sentence1
word1 @@@ type1 @@@ sentence2
word1 @@@ type1 @@@ sentence3
word1 @@@ type2 @@@ sentence4
word2 @@@ type1 @@@ sentence5
...

I want to join the lines that have the same word and type combination into one, by appending the sentences, so the file results in:

...
word0 @@@ type2 @@@ sentence0
word1 @@@ type1 @@@ sentence1 ;;; sentence2 ;;; sentence3
word1 @@@ type2 @@@ sentence4
word2 @@@ type1 @@@ sentence5
...

The word and type fields have no whitespace.

3

Assuming your input is sorted on both word and type fields as it appears from your posted sample input:

$ cat tst.awk
BEGIN { FS=" @@@ "; ORS="" }
{ curr = $1 FS $2 }
curr != prev {
    printf "%s%s", ORS, $0
    prev = curr
    ORS = RS
    next
}
{ printf " ;;; %s", $NF }
END { print "" }

$ awk -f tst.awk file
word0 @@@ type2 @@@ sentence0
word1 @@@ type1 @@@ sentence1 ;;; sentence2 ;;; sentence3
word1 @@@ type2 @@@ sentence4
word2 @@@ type1 @@@ sentence5

The above will work using any awk in any shell on every UNIX box, only stores 1 line at a time in memory, and will produce output in the same order as the input.

| improve this answer | |
  • 1
    I would never have been able to come up with this - Thank you – andrewz Mar 12 at 15:33
0

Here's one way, in awk:

$ awk -F'@@@' '{ $1 in a ? a[$1][$2]=a[$1][$2]" ;;; "$3 : a[$1][$2]=$3}END{for(word in a){for (type in a[word]){print word,FS,type,FS,a[word][type]} }}' file 
word0  @@@  type2  @@@  sentence0
word1  @@@  type1  @@@  sentence1 ;;;  sentence2 ;;;  sentence3
word1  @@@  type2  @@@  ;;;  sentence4
word2  @@@  type1  @@@  sentence5

Or, a bit more legibly:

awk -F'@@@' '{ 
                if($1 in a){ 
                    a[$1][$2]=a[$1][$2]" ;;; "$3
                }
                else{
                    a[$1][$2]=$3
                }
             }
             END{
                 for(word in a){
                     for (type in a[word]){
                         print word,FS,type,FS,a[word][type]
                     }
                 }
             }' file 

Note that this requires an awk implementation that understand multidimensional arrays, such as GNU awk (gawk), which is the default awk on Linux systems.

| improve this answer | |
  • That's not producing the expected output (see the 3rd line of output for an extra ;;; and there's extra blanks after all the ;;;s). Also you should mention that the use of for ... in will shuffle the order of the output and the script stores the whole file in memory. – Ed Morton Mar 11 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.