Text processing - How to get multiple patterns in order from a file

Question

I have this file.txt.Z which contains this:

AK2*856*1036~AK3*TD1*4**~AK4*2**1*~AK4*7**1*~AK3*TD5*5**~AK4*3**6*2~AK3*REF*6**~AK4*2**1*~AK3*REF*7**~AK4*2**1*~AK3*REF*8**~AK4*2**1*~AK3*DTM*9**~AK4*2**4*20~AK4*2**4*20~AK3*CTT*12**7~AK5*R
AK2*856*1037~AK3*HL*92**~AK4*3**7*O~AK5*R~AK9*R*2*2*0~SE*25*0001~GE*1*211582~IEA*1*000211582

Each record consists of several fields starting with a header (usually AK with a number), separated by ~. If you replace the ~ with an indented line break it would read:

AK2*856*1036
  AK3*TD1*4**
  AK4*2**1*
  AK4*7**1*
  AK3*TD5*5**
  AK4*3**6*2
  AK3*REF*6**
  AK4*2**1*
  AK3*REF*7**
  AK4*2**1*
  AK3*REF*8**
  AK4*2**1*
  AK3*DTM*9**
  AK4*2**4*20
  AK4*2**4*20
  AK3*CTT*12**7
  AK5*R
AK2*856*1037
  AK3*HL*92**
  AK4*3**7*O
  AK5*R
  AK9*R*2*2*0
  SE*25*0001
  GE*1*211582
  IEA*1*000211582

Each field has subfields separated by *. For example, subfield AK201 is the first field after an AK2 header, so it's 856 for the example lines.

As you can see there is 2 lines that has a starting string of AK2. That is like a line header or as we call it, Segment Header. There are two segment header in file.txt.Z. What I want is to get these data from each segment header in order:

Data Needed:

• AK202 (second field after AK2 header) - AK2*856*this_numeric_value before the asterisk or ~.
• AK301 (first field after AK3 header) - ~AK3*this_string_value before the * or ~.
• AK502 (second field after AK5 header) - ~AK5*some_string_value*this_numeric_value before the * or ~.
• AK401 (first field after AK4 header) - ~AK4*this_numeric_value before the * or ~.
• Every numerical value from AK4 or AK5 field are should always be at least 2 digits. e.g. AK502 = 2; AK502 = 02 or AK401 = 9; AK401 = 09.
• If there is no AK3 field, don't output anything. (I have already a script for this)
• If a line contains more than one AK3-AK5-AK4-sequences, they should get concatenated with a whitespace
• If the AK5 field misses after the AK3 field, look for AK4 field instead.
• If neither an AK4 nor an AK5 field are present after the AK3 field, Only output the AK301(first field after AK3 header).
• If there is more than one AK4 field after an AK3 field, concatenate the AK502-AK401-sequences by commas

Output:

GS: 1036 - TD102,07 TD503 REF02 DTM02,02 CTT
GS: 1037 - HL03

How to do this? Just ask me if you're confused with my question.

Edit: This is my code: This is inside a while loop

while read FILE
do
    AK2=`zgrep -oP 'AK2.[\w\s\d]*.\K[\w\s\d]*' < $FILE`
    AK3=`zgrep -oP 'AK3.\K[\w\s\d]*' < $FILE`
    AK5=`zgrep -oP 'AK5.[\w\s\d]*.\K[\w\s\d]' < $FILE`
    AK5_ERROR=`if [[ $AK5 =~ ^[0-9]+$ ]]; then  printf "%02d" $AK5 2> /dev/null; else 2> /dev/null; fi`
    AK4=`zgrep -oP 'AK4.\K[\w\s\d]*' < $FILE`
    AK4_ERROR=`if [[ $AK4 =~ ^[0-9]+$ ]]; then  printf "%02d" $AK4 2> /dev/null; else 2> /dev/null; fi`

    if [[ $AK3 ]]
    then
        if $AK5 2> /dev/null
        then
            echo "GS: $AK2 - $AK3$AK4_ERROR"
        else
            echo "GS: $AK2 - $AK3$AK5_ERROR"
        fi
    else
        echo "Errors are not specified in the file."
    fi
done < file.txt.Z

The problem with my original code is that It doesn't concatenate $AK3 and, $AK5 or $AK4.

Answer 1

The following perl script produces your sample output exactly when given your sample input.

It may not work exactly as you want on your real data file, but it is not being presented as a complete working solution. It is presented as a base to start working from - play with the script, mess around with it, break it, fix it, change it to do what you want.

It is undoubtedly far from optimal, but it would be difficult to improve it much without a far more detailed knowledge / better explanation of your input data and your required output.

It processes each input line (aka "record", or "segment" using your terminology) and builds up a string to be printed out after the record has been processed. Each output line is built up according to your specifications in the Data Needed section of your question.

#!/usr/bin/perl

use strict;

while(<>) {
  next unless /AK3/;  # skip lines that don't contain AK3

  # process each "segment" aka "record".
  my @fields = split /~/;

  # get segment "header" and 2nd sub-field of that header.
  my @segment = split(/\*/,$fields[0]);
  my $segment_header = $segment[2];
  shift @fields;

  my $output = "GS: $segment_header -";

  my $groupoutput = ''; # output for a given AK3 "group"
  my $last_go = ''; # used to avoid duplicates like "REF02 REF02 REF02"

  foreach my $f (@fields) {
    my @subfields = split /\*/,$f;

    if ($f =~ m/^AK3/) {

        if (($groupoutput) && ($groupoutput ne $last_go)) {
          $output .= " $groupoutput";
          $last_go = $groupoutput;  # remember the most recent $groupoutput
        };

        $groupoutput = $subfields[1];

    } elsif ($f =~ m/^AK4/) {
        my $ak401 = $subfields[1];
        $groupoutput .= sprintf("%02i,",$ak401) if ($ak401 > 0);
    } elsif ($f =~ m/^AK5/) {
        my $ak502 = $subfields[2];
        $groupoutput .= sprintf("%02i",$ak502) if ($ak502 > 0);
    };
  };

  # append the group output generated since the last seen AK3 (if any)
  # i.e. don't forget to print the final group on the line.
  $output .= " $groupoutput" if (($groupoutput) && ($groupoutput ne $last_go));

  # clean up output string before printing.
  $output =~ s/, / /g;
  $output =~ s/\s*$|,$//;

  print $output, "\n";
}

I saved this script as mysteryprocess.pl because I couldn't think of a more appropriate name. Then I ran it with your sample data (in a file called input):

example output:

$ ./mysteryprocess.pl input 
GS: 1036 - TD102,07 TD503 REF02 DTM02,02 CTT
GS: 1037 - HL03

---

That "REF02 REF03 REF02" thing bothered me, so here's another version. This one uses an array and a hash (@groups and %groups) to build up the output line, and another hash (%gseen) to prevent dupes within a record by remembering values that we've already seen and included in the output.

The groups data is stored in %groups, but hashes are un-ordered in perl, so the @groups array is used to remember the order that we first saw a particular group.

BTW, %groups should probably be a hash-of-arrays aka HoA (i.e. a hash which contains an array in every element), which would avoid the need for cleaning up $output just before printing it (by using perl's join() function rather than simply appending a comma and the new value to the strings). But I think this script is already complicated enough for a newcomer to perl to understand.

#!/usr/bin/perl

use strict;

while(<>) {
  next unless /AK3/;  # skip lines that don't contain AK3

  # process each "segment" aka "record".
  my @fields = split /~/;

  # get segment "header" from 1st field,  and then 2nd sub-field of that header.
  # NOTE: "shift" returns the first field of an array AND removes it from
  # the array.
  my @segment = split(/\*/, shift @fields);
  my $segment_header = $segment[2];

  my $output = "GS: $segment_header -";

  my @groups=(); # array to hold each group name (ak301) in the order that
                 # we see them
  my %groups=(); # hash to hold the ak401/ak502 values for each group
  my %gseen =(); # used to avoid dupes by holding specific values of ak301+ak401
                 # and ak301+ak502 that we've seen before.

  my $ak301='';

  foreach my $f (@fields) {
    my @subfields = split /\*/, $f;

    if ($f =~ m/^AK3/) {

        $ak301 = $subfields[1];
        if (!defined($groups{$ak301})) {
          push @groups, $ak301;
        };

    } elsif ($f =~ m/^AK4/) {

        my $ak401 = sprintf("%02i",$subfields[1]);
        $ak401 = '' if ($ak401 == 0);
        next if ($gseen{$ak301.'ak4'.$ak401});

        if (!defined($groups{$ak301})) {
          $groups{$ak301} = $ak401;
        } else {
          $groups{$ak301} .= ',' . $ak401;
        };
        $gseen{$ak301.'ak4'.$ak401}++;

    } elsif ($f =~ m/^AK5/) {

        my $ak502 = sprintf("%02i",$subfields[1]);
        $ak502 = '' if ($ak502 == 0);
        next if ($gseen{$ak301.'ak5'.$ak502});

        if (!defined($groups{$ak301})) {
          $groups{$ak301} = $ak502;
        } else {
          $groups{$ak301} .= ',' . $ak502;
        };
        $gseen{$ak301.'ak5'.$ak502}++;

    };
  };

  # construct the output string in the order we first saw each group
  foreach my $group (@groups) {
    $output .= " $group" . $groups{$group};
  };

  # clean up output string before printing.
  $output =~ s/, |  +/ /g;
  $output =~ s/\s*$|,$//;

  print $output, "\n";
}

With the following input

AK2*856*1036~AK3*TD1*4**~AK4*2**1*~AK4*7**1*~AK3*TD5*5**~AK4*3**6*2~AK3*REF*6**~AK4*2**1*~AK3*REF*7**~AK4*2**1*~AK3*REF*8**~AK4*2**1*~AK3*DTM*9**~AK4*2**4*20~AK4*2**4*20~AK3*CTT*12**7~AK5*R
AK2*856*1037~AK3*HL*92**~AK4*3**7*O~AK5*R~AK9*R*2*2*0~SE*25*0001~GE*1*211582~IEA*1*000211582
AK2*856*1099~AK3*TD1*4**~AK4*2**1*~AK4*7**1*~AK3*TD5*5**~AK4*3**6*2~AK3*REF*6**~AK4*2**1*~AK3*REF*7**~AK4*2**1*~AK3*REF*8**~AK4*3**1*~AK3*REF*8**~AK4*2**1*~AK3*DTM*9**~AK4*2**4*20~AK4*2**4*20~AK3*CTT*12**7~AK5*R

The output is now:

$ ./mysteryprocess.pl input 
GS: 1036 - TD102,07 TD503 REF02 DTM02 CTT
GS: 1037 - HL03
GS: 1099 - TD102,07 TD503 REF02,03 DTM02 CTT

Notes:

• DTM02,02 also got collapsed into just DTM02. The dupe elimination happens for everything now.
• The merging of groups (i.e. elements with the same AK301 "name") also occurs no matter where in a record an element appears. The previous version only merged adjacent fields/subfields if they were the same.

I'm not sure if either of these changes are what you want.

---

ps: if you don't have perl installed, this code would translate fairly easily to awk. It's a very simple (simplistic even), straight-forward algorithm.

Answer 2

another go, to show an awk version, as cas suggested. Can probably be done far more neatly, but was a learning experience anyway.

#!/usr/bin/awk -f

function get_slice(elem, fc,       tmpArr) {
        split(elem, tmpArr, "*")
        return tmpArr[fc]
    }

BEGIN { FS="~" }

/AK2/ { 
    res = get_slice($1, 3) " - "
    tmpStr = ""
    # only continue with this line if there are any AK3 fields.
    # otherwise may as well skip whole thing.
    if (match($0, /AK3/)) {
        loc=2
        for (loc=2; loc<=NF; loc++)
            if ($loc ~ /AK3/) break

        for ( ; loc<=NF; loc++) {
            if ($loc ~ /AK3/) {
                # check to see whether the previous loop generated a duplicate
                # tmpStr will be "" the first time
                if (index(res, tmpStr) == 0)
                    res = res " " tmpStr
                tmpStr = get_slice($loc, 2)
                # c is a count of how many fields have been added after AK3.
                # once positive, "," will be added.
                c = 0
                }
            # add the other fields
            else if ($loc ~ /AK4/) { 
                if ((s = get_slice($loc, 2)) != "")
                    tmpStr = tmpStr sprintf("%s%02d", c++ ? "," : "", s) 
            } else if ($loc ~ /AK5/) { 
                if ((s = get_slice($loc, 3)) != "")
                    tmpStr = tmpStr sprintf("%s%02d", c++ ? "," : "", s) 
            }
        }
        # this is repeated at the end, to make sure the final set is printed.
        if (index(res, tmpStr) == 0)
            res = res " " tmpStr
        print res
        }
    }

simply initially splitting fields on '~' and then looping through all the available fields per line. Only when a field is needed is it then split into subfields on '*' to get the asked for elements. 'get_slice' returns "" if nothing is found, so this has to be checked for.

I think I've understood the question..