How to identify, sort descending and display the top 10 blocks of text, by category

Question

The history of transactions done on an entity within our systems look like below:

   1 BYM1 TSTAB 09NOV 0035 CAB
Sometext 01
   2 BYM1 TSTAB 09NOV 0035 CAB
Can be done - question   
   3 BYM1 TSTAB 09NOV 0035 CAB
Sometext 02
Sometext 03
   6 BYM3 TSTAA 09NOV 0400 CAA
Some 04 text 04
   7 BYM3 TSTAA 10NOV 0455 CAC
Sometext 06
Sometext 06 line 2
   8 BYM3 TSTAA 10NOV 0455 CAC
Sometext 07
   9 BYM2 TSTAC 10NOV 0619 CAD
Some 08 text 0008 ABCD
Some 08 text 0008 BB00
Some 08 text 0008 CC00
Some 08 text 0008 DD00
Some 08 text 0008 EE00
  10 BYM2 TSTAC 10NOV 0627 CAD
Something BBBBBSSDGFSDSF
  11 BYM2 TSTAC 10NOV 0627 CAD
Something else
  12 BYM2 TSTAC 10NOV 0627 CAD
What text here
  13 BYM4 TSTAC 10NOV 0711 CAD
Tired figuring out
  19 BYM3 TSTAA 11NOV 0438 CAE
Some 04 text 05 05 05
  20 BYM3 TSTAA 11NOV 0441 CAF
Not so confidential now
  21 BYM3 TSTAA 11NOV 0441 CAF
Some 00 text 0009 X1X2
  43 BYM3 TSTAA 11NOV 0441 CAD
Some 0A text 0009 ABCD
  44 BYM3 TSTAA 11NOV 0441 CAD
Some 1B text
  45 BYM3 TSTAA 12NOV 1455 CAC
Something 0AADDBB
8782 BYM3 TSTAA 12NOV 1610 CAD
Something 0AADDBB
8830 BYM3 TSTAA 12NOV 1612 CAA
Something 0AADDBB
9999 BYM3 TSTAA 12NOV 1722 CAA
Something 0AADDBB

The blocks of text start with the line which has a number in the first 4 characters. (The number is actually a running sequence number and every transaction is indexed with that). The category (of the transaction) of the block is defined by the last three characters in the line which has the number.

I am looking for a awk, sed (, vi, grep) script to search for blocks of text belonging to a "category", sort the resultant blocks in descending order of the index (numbers), display the number of blocks I have asked for.

For example, if I want to search of 4 blocks of category "CAD" the output I would like to see is:

8782 BYM3 TSTAA 12NOV 1622 CAD
Something 0AADDBB
  44 BYM3 TSTAA 11NOV 0441 CAD
Some 1B text
  43 BYM3 TSTAA 11NOV 0441 CAD
Some 0A text 0009 ABCD
  13 BYM4 TSTAC 10NOV 0711 CAD
Tired figuring out

How can I acheive this. Any help would be greatly appreciated :-)

Answer 1

Here is a solution for gawk (GNU awk; i.e., the version of awk found on most “Linux” systems).  Assume that $cat is set to the category you want to search for, and $num is set to the number of records you want to display.

awk -vRS='\n[ 0-9][ 0-9][ 0-9][0-9] ' -vcat="$cat" -vnum="$num" \
    '   BEGIN { first=1; rec_ind=0}
        {       if (first) {
                        rec = $0
                        first=0
                } else {
                        rec = save_seq $0
                }
                findnl = index(rec, "\n")
                if (findnl < 7) exit
                thiscat = substr(rec, findnl-3, 3)
                if (cat == thiscat) records[++rec_ind] = rec
                if (length(RT) == 0) {
                        # print "This should be the last record."
                        save_seq = "Does not matter"
                } else if (length(RT) == 6) {
                        save_seq = substr(RT, 2, 5)
                } else {
                        print "Invalid RT: len =", length(RT)
                        exit
                }
        }
        END   { num_recs = asort(records, sorted_records, "@val_num_desc")
                if (num < num_recs) num_recs = num
                for (i=1; i<=num_recs; i++) {
                        print sorted_records[i]
                }
              }
    '

Notes:

• -vRS='\n[ 0-9][ 0-9][ 0-9][0-9] ' sets awk’s RS (record separator) variable to a regular expression that consists of a newline, followed by an integer sequence number of up to four digits, followed by a space.  I included the newline because your data have four-digit numbers (followed by spaces) in the interior of lines, where they are not to be interpreted as record separators.  Note that this regex is a little sloppy, as it will accept  007 and 12 4.

  Setting this as awk’s record separator means that each of your “transactions” will be treated as a single awk record, even though it contains multiple lines.  There are a couple of drawbacks:

  • Since the RS pattern includes a newline at the beginning, the    1  at the beginning of your data will not be recognized as a record separator.
  • Since this is the record separator pattern, it is not considered to be part of the record, even though it contains vital information.

  We’ll deal with those problems.

• -vcat="$cat" and -vnum="$num" similarly set the awk variables cat and num to the values of the corresponding shell variables.
• BEGIN { first=1; rec_ind=0} initializes the first flag to true (1), so we can recognize the first record and handle it specially, and the record index (rec_ind) to 0, for the accumulation of records that match the desired category.

• if (first) is true (we are processing the first record), set rec equal to the awk record, $0.  Remember, this includes all the lines up to (but not including) the next line that begins with a four-digit number.  Also, it includes the four-digit number at the beginning of the first line.  Then we set the first flag to false (0).

  If this isn’t the first record, then it’s missing its four-digit number (because that’s the record separator), so we construct the record (rec) by concatenating the saved sequence number (save_seq) with $0.  (I’ll discuss save_seq momentarily.)

• findnl = index(rec, "\n") finds the first newline in the record (remember, records contain multiple lines).  If it’s less than 7 characters in from the beginning, then there isn’t room for a sequence number and a category (without overlapping), let alone the other fields, so this is an error.  Otherwise, extract the category of this record (thiscat) from the last three characters before the first newline — i.e., the last three characters of the first line of the transaction.  Then, if thiscat matches the category that we are looking for, save the record in the records array.
• RT is the record terminator — the characters that match the RS pattern at the end of the current record.  Unfortunately, the terminator of the current record is really the beginning of the next one.  If the current record is the last one, then RT will be an empty string (length 0); otherwise, it should always be 6 characters long (a newline, four characters that are either spaces or digits, and a space).  Extract the last five characters (i.e., discard the newline) and save that as save_seq, because it is the sequence number of the next transaction.
• When we get to the end of the data, sort the records (sorting the values, treating them as numbers, in descending order). Then print up to num of them.

Answer 2

To follow linux principe "one task -- one tool":

1. Prints just necessary block (as in example CAD)

   sed '/^\s*[0-9].*CAD/!d;:a;N;/\n\s*[0-9]/! s/\n/\x0/;ta;P;D'

2. Sort it in reverse order

   sort -rn

3. Take just first asked blocks (as in example 4)

   head -4

Please note, that most linux commands operate with lines (not blocks) so the ones was converted into lines by changing \new line to null-symbol(\x0) then converted back by tr.
So, all line:

sed '/^\s*[0-9].*CAD/!d;:a;N;/\n\s*[0-9]/! s/\n/\x0/;ta;P;D' test.txt |
sort -rn |
head -4 |
tr '\0' '\n'

---

I like the G-Man answer's idea to change RowSeparator but this is not much suitable in the case. More simple to do it in ordinary way

awk '
/^[ 0-9]{4} /{                 #for start block string
    if($NF==cat){              #if it is a needed block
        idx=$1
        BLOCK[idx]=$0          #put line onto array with asigned index
    }
    else
        idx=0                  #otherways asign index to 0
    next                       #end itteration, go to start with next line
}
idx{                           #pass inappropriate blocks (with 0-index)
BLOCK[idx]=BLOCK[idx] "\n" $0  #add line to array element with index
}
END{                           #when finish all lines
    for(i=0;i<num;i++){        #do num times
        max=0                  #asing `max` variable to min value
        for(idx in BLOCK){     #for each index in array
            idx=idx+0          #convert string index into decimal
            if(idx>max)        
                max=idx        #find maximum index (field No.1 in block)
        }
        if(!max)             
            exit               #exit script if array empty (no more blocks)
        print BLOCK[max]       #print block with maximum index
        delete BLOCK[max]      #remove array element for furure search
     }
}' cat="CAD" num=4 test.txt

Answer 3

Assuming your block start can be detected by a line of 6 fields starting with a number, and that your data contains no character code \001 (control-a), for example, you can join all the lines of a block into one line, replacing the newlines by this arbitrary code. Then sort the lines, take the first 4 and replace the code by a newline again.

#!/bin/bash
num=${1?number} cat=${2?category}
awk -vcat="$cat" '
 /^ *[0-9]+ / && NF==6 { ok = ($NF==cat) 
                         if(ok && sep!="")sep = "\n"
                       }
                    ok { printf "%s%s",sep,$0; sep = "\001" }
                   END { if(sep!="")printf "\n" }' |
sort -nr -k1,1 | head -"$num" |
tr '\001' '\n'

The awk joins the lines if field $NF (the last field) is the wanted category. The sep variable is initially empty "" then becomes \001 inside a block, and \n when a new block starts. At the end a final newline is added unless there were no matches.

Answer 4

try changing the values of v= and num=

$ awk '$NF==v{F=1;print;next}F&&NF!=6{print}F&&NF==6{F=0}' v="CAC" test.txt | awk '$NF~v{val=j++;F=1}F{Arr[val]=Arr[val]"\n"$0}END{n=asorti(Arr,S_Arr);for(i=n;i>=n-num;i--){print Arr[i]}}' v="CAC" num=4


  45 BYM3 TSTAA 12NOV 1455 CAC
Something 0AADDBB

   8 BYM3 TSTAA 10NOV 0455 CAC
Sometext 07

   7 BYM3 TSTAA 10NOV 0455 CAC
Sometext 06
Sometext 06 line 2

$ awk '$NF==v{F=1;print;next}F&&NF!=6{print}F&&NF==6{F=0}' v="CAD" test.txt | awk '$NF~v{val=j++;F=1}F{Arr[val]=Arr[val]"\n"$0}END{n=asorti(Arr,S_Arr);for(i=n;i>=n-num;i--){print Arr[i]}}' v="CAD" num=4


8782 BYM3 TSTAA 12NOV 1610 CAD
Something 0AADDBB

  44 BYM3 TSTAA 11NOV 0441 CAD
Some 1B text

  43 BYM3 TSTAA 11NOV 0441 CAD
Some 0A text 0009 ABCD

  13 BYM4 TSTAC 10NOV 0711 CAD
Tired figuring out