9

I have this thing that I maintain at work, and it has a pretty arcane DSL that it uses. And the tooling for it is not great. To deal with the poor tooling, I've written some scripts to try to find some issues with the code before I send it to production.

The current problem I'm trying to solve has to do with variable names. Variables are named like @@Variable@@. If there is only 1 @, or more than 2 @s, then it is a fatal error.

Right now I've got it looping through the files in question, and grepping for @@@ and raising an error when it finds 3 or more consecutive @'s. So that part is cool.

But I'm sort of stuck on the single @. There can be more than one variable on a line.

@@Var1@@ words words words @@Var2@@  #This works
@Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
@@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.

There are loads of permutations of the above, and there is no limit to the number of variables on a line.

This awk script works if there is a single variable on any given line, but it doesn't work if there is more than 1 variable on a line.

awk '/@/ && ! /@@.*@@/' test.txt

What I really need to do is match anything where there is only a single instance of @. In the sample code above, it would match on all lines except line 1.

6
  • Could there be @ that are not part of variables? Commented Oct 3, 2021 at 16:49
  • If you can have punctuation before/after the variables (e.g. foo @@var@@.) or single-char variables (e.g. foo @@v@@) then include those plus any other non-trivial cases you can think of in your sample input/output as you're getting answers (mine included) that will work for the sample input you posted but will fail for var names in other contexts.
    – Ed Morton
    Commented Oct 3, 2021 at 17:17
  • There should not be any @ that are not part of variables. But there's nothing specific that would prohibit that, aside from best practice.
    – Kirk
    Commented Oct 4, 2021 at 12:46
  • 2
    @Kirk Please edit your question adding all the cases that will work, as in your comment seemingly @@Var3 is also a valid case, and I'm afraid most answers here could fail with this new information. Commented Oct 4, 2021 at 13:52
  • 1
    I typed that comment too quickly (which is why I asked the question in the end). @@Var3 is not valid. @@Var3@@ is.
    – Kirk
    Commented Oct 7, 2021 at 13:00

6 Answers 6

7
$ sed 'h;s/@@[^@ ]*@@//g;/@/!d;g' file
@Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
@@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.

The sed command removes valid variable placeholders and reports the lines that still contain @ characters. It would also find lines containing placeholders with more than two @ on either side.

We can report the original faulty lines by saving each line to the hold space with h. The substitution that removes potentially valid placeholder is then run, and we remove the line if it doesn't contain any @ characters afterward. We fetch the original line from the hold space with g and print it if it does.

The pattern for a valid placeholder, @@[^@ ]*@@, could be changed to @@[[:alpha:]_][[:alnum:]_]*@@ if your variables follow the same naming rules as most programming languages.

Suppose you need to be able to include @ characters in the text itself. In that case, you need to remove all valid constellations in which @ may occur that are not variable placeholders before the substitution in the command above.


A more systematic approach would be to extract lines containing placeholders with too many @ characters on one side or the other, delete the correct ones, and then pull the lines out that have placeholders with only one @ character on either side of the variable's name.

sed -e '/@\{3,\}[^@ ]*@\{1,\}/b' \
    -e '/@\{1,\}[^@ ]*@\{3,\}/b' \
    -e h \
    -e 's/@@[^@ ]*@@//g' \
    -e '/@[^@ ]*@/!d' \
    -e g file

The above would allow your text to contain @ characters elsewhere, given that they don't occur in a pattern that looks like a placeholder.

1
  • I never even thought of using sed. Seems like it might be time for me to reread the Sed & Awk book.
    – Kirk
    Commented Oct 4, 2021 at 12:43
6
$ grep -E '(^|[^@])@([^@]|$)|@@@' file
@Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
@@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.

or:

$ awk '/(^|[^@])@([^@]|$)|@@@/' file
@Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
@@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.

or analyzing one field at a time:

$ cat tst.awk
{
    for (i=1; i<=NF; i++) {
        if ( $i ~ /^@[^@]|[^@]@$|@@@/ ) {
            print "Failed line:", NR, $0
            print "\tbecause of field", i, $i
        }
    }
}

$ awk -f tst.awk file
Failed line: 2 @Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
        because of field 1 @Var1@@
Failed line: 3 @@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
        because of field 1 @@Var1@
Failed line: 4 @@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
        because of field 5 @Var2@@
Failed line: 5 @@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.
        because of field 5 @@Var2@

You don't need anything additional to find @@@ cases, the above includes finding that case too.

3

Another solution with grep -E. The regex works with awk as well

grep -E '[^@]@[^@]|^@[^@]|@[^@]$' tmpfile
@Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
@@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.

awk '/[^@]@[^@]|^@[^@]|@[^@]$/' tmpfile
@Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
@@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.
1
  • 1
    Would it be better to use this regex instead '[^@]@[^@]|^@[^@]|[^@]@$' as this wold match a line that is ending with a variable and a single @ at the end of the line
    – Angel115
    Commented Oct 3, 2021 at 19:26
3

The following answer and script is based around two dynamically created perl arrays (@- and @+) that hold the start and end positions of regex matches.

If there are exactly two or exactly four @ characters in a row (tested via a bitwise AND with 6 (2+4)), the match is considered good, and execution continues with the next (if any) match on the same line. This test could be done with if ($count == 2 || $count == 4) but why do two comparisons per match when you only need to do one?

Also, if there are 4 @s in a row, but the rest of the line is missing a matching @@, the line is also considered bad and handled as above.

Otherwise, the entire line is deemed bad and printed (with the line number and start & end positions of the match) and the rest of the line is skipped, with execution continuing with the next input line (this is to avoid printing the same bad line multiple times).

$ perl -e '
    LINE: while (<>) { # iterate over the input lines
      if (/@@@@[^@]+(?!@@)/) { # four @s but no ending @@
        printf "%i:(%02i,%02i):%s", $., $-[0], $+[0], $_;
        next;
      };
      while (/\@+/g) { # iterate over the @+ matches in the current line
        $count = $+[0] - $-[0];
        if ($count & 6 != 0) { # count is not 2 or 4
          printf "%i:(%02i,%02i):%s", $., $-[0], $+[0], $_;
          next LINE;
        };
      };
    }' input.txt 
2:(00,01):@Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
3:(06,07):@@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
4:(27,28):@@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
5:(33,34):@@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.
6:(06,13):@@Var1@@@@var2@@ words words words   #This will work because @@@@
7:(06,09):@@Var1@@@var2@@ words words words    #This will fail because @@@
8:(06,60):@@Var1@@@@ words words words words   #This will fail because @@@@ but no ending @@

Explanations of the "inscrutably weird" perl variables and syntax:

  • $. is the current input line number (similar to NR in awk)
  • $-[0] is the start pos of the current match in the current line
  • $+[0] is the end pos of the current match in the current line
  • $_ is the current input line contents
  • LINE: is a label, needed so that the second next statement is applied to the outer loop, not the inner loop.

If you want plainer output with fewer or no diagnostics, replace both of the printf statements with one of the following:

print "$.:$_";  # line numbers and line contents only
print;          # line contents only

BTW, if you don't want to print the match positions, the script can be simplified:

perl -e '
    LINE: while (<>) {
      while (/\@+/g) {
        $count = $+[0] - $-[0];
        if ($count & 6 != 0 || /@@@@[^@]+(?!@@)/) {
          printf "%i:%s", $., $_;
          #print;
          next LINE
        }
      }
    }' input.txt

BTW, the input.txt file I tested with was basically the same as your sample input with two lines added:

$ cat input.txt 
@@Var1@@ words words words @@Var2@@  #This works
@Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
@@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.
@@Var1@@@@var2@@ words words words   #This will work because @@@@
@@Var1@@@var2@@ words words words    #This will fail because @@@
@@Var1@@@@ words words words words   #This will fail because @@@@ but no ending @@

FYI, from man perlvar:

@LAST_MATCH_END, @+

This array holds the offsets of the ends of the last successful submatches in the currently active dynamic scope.

$+[0] is the offset into the string of the end of the entire match. This is the same value as what the pos function returns when called on the variable that was matched against.

The nth element of this array holds the offset of the nth submatch, so $+[1] is the offset past where $1 ends, $+[2] the offset past where $2 ends, and so on.

You can use $#+ to determine how many subgroups were in the last successful match. See the examples given for the @- variable.

and

@LAST_MATCH_START, @-

$-[0] is the offset of the start of the last successful match. $-[n] is the offset of the start of the substring matched by n-th subpattern, or undef if the subpattern did not match. [...more details and examples with substr() deleted...]

1
  • BUGS: there are still some cases with @@@@ on a line that are "bad" that won't be caught by this script. the solution is probably to count the number of @@ matches on a line and check if they're divisible by 2.
    – cas
    Commented Oct 4, 2021 at 1:37
3

With grep -P, where supported, you can match on single @s or sequences of 3 or more @s not surrounded by @s using negative look-around operators:

<test.txt grep --color -P '(?<!@)(@|@{3,})(?!@)'

That would however still flag the @@@@ in @@var1@@@@var2@@ and fail to flag unmatched @@s like in @@var1 or @@var1@@var2@@.

Another approach would be to do:

<test.txt grep --color -P '@@\w+@@(*SKIP)(*FAIL)|@+'

Which would flag the @'s that are not part of @@word@@ sequences.

$ <test.txt grep --color -P '@@\w+@@(*SKIP)(*FAIL)|@+'
@Var1@@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@ words words words @@Var2@@   #This will fail because Var1 is wrong.
@@Var1@@ words words words @Var2@@   #This will fail because Var2 is wrong.
@@Var1@@ words words words @@Var2@   #This will fail because Var2 is wrong.

0

Using Raku (formerly known as Perl_6)

raku -pe '.=trim; s:g/ (<-[_]>? .+ <-[_]>?) /_$0_/;'  |
raku -ne '.words.split( / <-[@]>+ / ).map(*.chars).grep( * % 2 ).say;'

Sample Input (from @cas, only lines #1 and #6 are valid):

@@Var1@@ words words words @@Var2@@  
@Var1@@ words words words @@Var2@@   
@@Var1@ words words words @@Var2@@   
@@Var1@@ words words words @Var2@@   
@@Var1@@ words words words @@Var2@   
@@Var1@@@@var2@@ words words words   
@@Var1@@@var2@@ words words words    
@@Var1@@@@ words words words words   

Sample Output (without grep call at end):

(0 2 2 2 2 0)
(0 1 2 2 2 0)
(0 2 1 2 2 0)
(0 2 2 1 2 0)
(0 2 2 2 1 0)
(0 2 4 2 0)
(0 2 3 2 0)
(0 2 4 0)

Sample Output (with grep call at end):

()
(1)
(1)
(1)
(1)
()
(3)
()

[Note above--the output of the first raku -pe '…' call is piped into the input of the second raku -ne '…' call].

A few things were tried out, the first of which is "protecting" start-of-string and end-of-string ends by adding _ underscores where none existed previously. Splitting on words leaves the _ underscores intact (noted often in this forum, thx @Stéphane Chazelas). Then using Raku's destructive split operator on everything non-@ and counting chars, some nice numeric patterns are revealed. Erroneous lines seem to be easily grepped-out. The only undetected line is line #8. Limited analysis (8 test lines), but a fair start.

https://docs.raku.org/routine/grep
https://raku.org

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .