1

I'm not sure how to title this, but here's what I've got:

A file with text like:

[10:03:43] 10:03:35 22JUN22  ConUP     CW1B1 T     CW1B3 T     CW1B5 T     CW1B7 T  
[10:03:44]                          CW2B1 T     CW2B4 T     CW2B5 T     CW2B7 T  
[10:03:44]                          CW3B1 T     CW3B4 T     CW3B5 T     CW4B4 T  
[10:03:44]                          CW4B8 T     CW5B4 T     CW5B8 T     CW6B4 T  
[10:03:44]                          CW6B8 T     CW7B8 T     CW8B4 T     CW8B8 T  
[10:03:44]                          CW9B4 T     CW9B8 T    CW10B4 T    CW10B8 T  
[10:03:44]                         CW11B4 T    CW11B8 T    CW12B4 T    CW12B8 T   
[10:04:16] 10:04:28 22JUN22  ConUP     CW1B1 T     CW1B3 T     CW1B5 T     CW1B7 T  
[10:04:36]                          CW2B1 F     CW2B3 F     CW2B5 F     CW2B7 F  
[10:04:36]                          CW3B1 T     CW3B4 T     CW3B5 T     CW4B4 T  
[10:04:36]                          CW4B8 T     CW5B4 T     CW5B8 T     CW6B4 T  
[10:04:37]                          CW6B8 T     CW7B8 T     CW8B4 T     CW8B8 T  
[10:04:37]                          CW9B4 T     CW9B8 T    CW10B4 T    CW10B8 T  
[10:04:37]                         CW11B4 T    CW11B8 T    CW12B4 T    CW12B8 T  

Then the actual file is full of stuff like that with the True and False values changing.

Whenever a have an entry of True, and it hasn't changed since the last entry, I wanted to remove that entry, only keeping the entry if the value has changed to False (or from False to True).

My thought is I need to find the first occurrence of a string (CW1B1 for example) then store the T or F value that follows it as a variable. Then check for the next occurrence, and compare the T/F value, if it's the same, the use cut or send to remove the entry, and then go look for the next. If it's value has changed, just ignore it and move on.

Expected output:

[10:03:43] 10:03:35 22JUN22  ConUP     CW1B1 T     CW1B3 T     CW1B5 T     CW1B7 T  
[10:03:44]                          CW2B1 T     CW2B4 T     CW2B5 T     CW2B7 T  
[10:03:44]                          CW3B1 T     CW3B4 T     CW3B5 T     CW4B4 T  
[10:03:44]                          CW4B8 T     CW5B4 T     CW5B8 T     CW6B4 T  
[10:03:44]                          CW6B8 T     CW7B8 T     CW8B4 T     CW8B8 T  
[10:03:44]                          CW9B4 T     CW9B8 T    CW10B4 T    CW10B8 T  
[10:03:44]                         CW11B4 T    CW11B8 T    CW12B4 T    CW12B8 T   
[10:04:16] 10:04:28 22JUN22  ConUP       
[10:04:36]                          CW2B1 F     CW2B3 F     CW2B5 F     CW2B7 F  
[10:04:36]                          
[10:04:36]                          
[10:04:37]                          
[10:04:37]                          
[10:04:37]                         

Maybe there is a better way to do it, but I still need help coming up with the framework, then I can go and create all the possible variables required in the script.

4
  • 1
    I don't see any next occurrence of any of the strings. So - both lines should go unmodified to the output?
    – RudiC
    Jun 21 at 19:23
  • Please edit your question to provide sample input and expected output that demonstrates your requirements and we can copy/paste to test a potential solution,
    – Ed Morton
    Jun 21 at 22:35
  • Are the "occurrences" always 7 lines? So we can split the data into 7-line chunks? Jun 23 at 3:42
  • No, this is just a snippet. It can vary.
    – ditch
    Jun 24 at 11:47

3 Answers 3

0

This may be what you want (untested since no testable sample input/output provided):

awk '
    {
        for ( i=2; i<=NF; i++ ) {
            if ( $i ~ /^[TF]$/ ) {
                tag = $(i-1)
                val = $i
                if ( map[tag] == val ) {
                    $(i-1) = $i = ""
                }
                map[tag] = val
                i++
            }
        }
        $0 = $0
        $1 = $1
        print
    }
' file
1
  • I'm sorry my sample data was insufficient. I have updated the original post with accurate samples. Is that more useful? Thank you.
    – ditch
    Jun 22 at 15:34
0

Using Raku (formerly known as Perl_6)

raku -e 'my @a = words;  @a.splice(1, *-128);  @a.splice(64,*-62);  \
         @a.=rotor(9); @a = @a>>.[1..*-1].flat.rotor(2); my @b; for 0..27 -> $i {  \
         @a[$i] eq @a[$i+28] ?? @b.push(@a[$i]~"+") !! @b.push(@a[$i+28]~"x") }; \
         .say for @b.rotor(4)>>.join("\t|\t");'  

OUTPUT:

CW1B1 T+    |   CW1B3 T+    |   CW1B5 T+    |   CW1B7 T+
CW2B1 Fx    |   CW2B3 Fx    |   CW2B5 Fx    |   CW2B7 Fx
CW3B1 T+    |   CW3B4 T+    |   CW3B5 T+    |   CW4B4 T+
CW4B8 T+    |   CW5B4 T+    |   CW5B8 T+    |   CW6B4 T+
CW6B8 T+    |   CW7B8 T+    |   CW8B4 T+    |   CW8B8 T+
CW9B4 T+    |   CW9B8 T+    |   CW10B4 T+   |   CW10B8 T+
CW11B4 T+   |   CW11B8 T+   |   CW12B4 T+   |   CW12B8 T+

Here's an idea using Raku: basically the code builds a dashboard you can run on the tail of your log file. Take input, break it into words, and assign it to @a array. Use splice to remove the three extra words from row 1 and row 8. Now, a couple of re-arrangements allow you to 1) recreate 9-element rows with rotor(9), and then 2) drop the first column and pair remaining 8 columns into 2-element rows, e.g. ("CW1B1", "T"), ("CW1B3", "T"), ("CW1B5", "T")....

From this point you know you have 28 elements, numbered 0..27. Use iterate throught these 28 elements searching for eq string-equality between @a[$i] and @a[$i+28]. Raku's ternary operator, i.e. "Test ?? True !! False" pushes the first string onto the @b array if True (add a + indicating the two values have remained constant), and pushes the second string onto the @b array if False (add an x indicating the value has changed). Then just re-arrange the rows (with rotor(4)) back to 4 pairs-per-row, and print.


Sample Input:

[10:03:43] 10:03:35 22JUN22  ConUP     CW1B1 T     CW1B3 T     CW1B5 T     CW1B7 T  
[10:03:44]                          CW2B1 T     CW2B4 T     CW2B5 T     CW2B7 T  
[10:03:44]                          CW3B1 T     CW3B4 T     CW3B5 T     CW4B4 T  
[10:03:44]                          CW4B8 T     CW5B4 T     CW5B8 T     CW6B4 T  
[10:03:44]                          CW6B8 T     CW7B8 T     CW8B4 T     CW8B8 T  
[10:03:44]                          CW9B4 T     CW9B8 T    CW10B4 T    CW10B8 T  
[10:03:44]                         CW11B4 T    CW11B8 T    CW12B4 T    CW12B8 T   
[10:04:16] 10:04:28 22JUN22  ConUP     CW1B1 T     CW1B3 T     CW1B5 T     CW1B7 T  
[10:04:36]                          CW2B1 F     CW2B3 F     CW2B5 F     CW2B7 F  
[10:04:36]                          CW3B1 T     CW3B4 T     CW3B5 T     CW4B4 T  
[10:04:36]                          CW4B8 T     CW5B4 T     CW5B8 T     CW6B4 T  
[10:04:37]                          CW6B8 T     CW7B8 T     CW8B4 T     CW8B8 T  
[10:04:37]                          CW9B4 T     CW9B8 T    CW10B4 T    CW10B8 T  
[10:04:37]                         CW11B4 T    CW11B8 T    CW12B4 T    CW12B8 T  

https://stackoverflow.com/questions/3416467/how-to-tail-f-the-latest-log-file-with-a-given-pattern
https://docs.raku.org/routine/splice
https://docs.raku.org/language/operators#index-entry-operator_ternary
https://raku.org

0

The following perl script will remove duplicate variable values from each input line. Only variables that we either haven't seen before or that have changed since the last time they were seen will be included in the output. Empty lines (i.e. those with no new or changed variables) aren't printed.

$ cat remove-dupes.pl 
#!/usr/bin/perl

use strict;

# %vars is a hash (associative array) to store last-seen T/F values
# for the "variables".  This is global in scope, i.e. defined here
# outside of the main loop because its keys & values need to be
# remembered across lines.
my %vars;

# read in and process each input line
while(<>) {
  chomp; # remove newline character at end of line

  # @out is an array to hold "VAR [TF]" values that we either
  # haven't seen before or have changed since last time we saw
  # them.  i.e. the values we want to output.  It needs to be
  # reset to empty every time we read a line.
  my @out = ();

  # extract the first word (i.e. "ConUp") from input line
  # by deleting all "VAR [TF]" strings from the line
  my $first_word = $_;
  $first_word =~ s/ ?\w+\s+[TF]//g;
  # add it to @out if it isn't empty
  push @out, $first_word if $first_word;

  # Iterate over each "VAR [TF]" string in the line. Split
  # it into the key (variable name) and value (T or F).
  # If it's new or changed then add it to @out and
  # remember its current value in %vars
  while (/ ?\b(\w+ [TF])\b/g) {
    my ($key,$val) = split / /, $1;
    next if ($vars{$key} eq $val);
    push @out, $1;
    $vars{$key} = $val;
  };

  # join (with spaces) and print @out if it isn't empty.
  if (@out) {print join(" ",@out), "\n"};
}

NOTE: if there are any other strings that aren't "VAR [TF]" on any line, they will all be moved to the beginning of that line in the output. This may or may not be considered to be a bug. Since the only such instance in your sample data is the "ConUp" at the beginning of the first line, it works as required. The variable probably shouldn't be called $first_word, but that's a lot shorter than $anything_that_doesnt_match_a_boolean_variable_pattern

Your input file doesn't contain any duplicate variable values that should be removed, so I've created one that does (and adds a few instances of variables that change, too):

$ cat file2.txt 
ConUp  CW1B1 T CW1B2 T CW1B3 F CW1B4 F
CW2B1 T CW2B2 F CW2B3 F CW2B4 T
CW2B1 F CW2B2 F CW2B3 F CW2B4 T
CW2B1 F CW2B2 F CW2B3 F CW2B4 T
CW2B1 T CW2B2 F CW2B3 F CW2B4 T
CW2B1 F CW2B2 F CW2B3 T CW2B4 T

Sample output:

$ ./remove-dupes.pl file2.txt 
ConUp  CW1B1 T CW1B2 T CW1B3 F CW1B4 F
CW2B1 T CW2B2 F CW2B3 F CW2B4 T
CW2B1 F
CW2B1 T
CW2B1 F CW2B3 T

Updated version:

This version extracts the first 34 characters of each line as the "first word", uses sprintf() to format each element of @out as a 10-character-wide string, and strips trailing spaces from the output. It also uses \s* and \s+ instead of a single space to match (or split) on any white-space characters. Comments have been removed for brevity, as the comments in the original script are still applicable (changes are trivial).

$ cat remove-dupes2.pl 
#!/usr/bin/perl

use strict;
my %vars;

while(<>) {
  chomp;
  my @out = ();

  my $first_word;
  ($first_word = $_) =~ s/\b\w+\s+[TF]//g;
  $first_word =~ s/^(.{34})\s+$/$1 /;
  push @out, $first_word if $first_word;

  while (/\s*\b(\w+\s+[TF])\b/g) {
    my ($key,$val) = split /\s+/, $1;
    next if ($vars{$key} eq $val);
    push @out, sprintf("%-10s",$1);
    $vars{$key} = $val;
  };

  my $out = join(" ",@out);
  $out =~ s/\s+$//; # remove trailing spaces
  print "$out\n";
}

Sample output using your new input data (input.txt). It's still not exactly the same as your sample output, there are still a few whitespace differences, but the output is a LOT closer than the output from the original script above.

$ ./remove-dupes2.pl input.txt 
[10:03:43] 10:03:35 22JUN22  ConUP  CW1B1 T    CW1B3 T    CW1B5 T    CW1B7 T
[10:03:44]                          CW2B1 T    CW2B4 T    CW2B5 T    CW2B7 T
[10:03:44]                          CW3B1 T    CW3B4 T    CW3B5 T    CW4B4 T
[10:03:44]                          CW4B8 T    CW5B4 T    CW5B8 T    CW6B4 T
[10:03:44]                          CW6B8 T    CW7B8 T    CW8B4 T    CW8B8 T
[10:03:44]                          CW9B4 T    CW9B8 T    CW10B4 T   CW10B8 T
[10:03:44]                          CW11B4 T   CW11B8 T   CW12B4 T   CW12B8 T
[10:04:16] 10:04:28 22JUN22  ConUP
[10:04:36]                          CW2B1 F    CW2B3 F    CW2B5 F    CW2B7 F
[10:04:36]
[10:04:36]
[10:04:37]
[10:04:37]
[10:04:37]

BTW, if you don't want the almost-empty lines (with just [HH:MM::SS]) to be printed, change the print "$out\n"; line to print "$out\n" unless ($out =~ /^\[[^]]\]\s*$/);.

Using diff to confirm that the only differences are spaces (output.txt is your sample output):

$ ./remove-dupes2.pl input.txt > out.txt
$ diff --ignore-space-change -u output.txt out.txt
$ 
2
  • I'm sorry my sample data was insufficient. I have updated the original post with accurate samples. I am gather data with the program Grabserial, so there are timestamps from that, as well as timestamps from the unit I'm recording from. I think this would potentially cause an issue with this solution then?
    – ditch
    Jun 22 at 15:33
  • 1
    @ditch I just ran my remove-dupes.pl script with your new sample data. Aside from some whitespace differences, the output was identical to your sample output. (confirmed with diff --ignore-space-change). The output from my script is indented more than your sample data and there's only a single space between each VAR [TF] pair....they're both fairly easy cosmetic fixes.
    – cas
    Jun 23 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.