The ERE regex to split() string between a delimiter and end-of-word

Question

I'm using a long gawk 3.1.6 script to do a complex conversion of Zim markdown text into GtkDialog code and am stuck on the following problem...

Sample ASCII input...

[[link|label label]] [[link]] @tag more text

Commandline test to find right regex...

re="[][][][]"; echo '[[link|label label]] [[link]] @tag more text' | awk -v RE=$re '{split($0,A,RE); printf "\n(" A[1] ")(" A[2] ")(" A[3] ")(" A[4] ")(" A[5] ")(" A[6] ")(" A[7] ")(" A[8] ")\n"}'

The regex "[][][][]" splits out the two hyperlink forms quite nicely so that's not a problem.

It would be more understandable if we could divided it in two -- "[][]" and "[][]". We are looking for either "[[" or "]]" to split on. The order of the characters in the class have to be reversed to comply with class meta-character restrictions.

The problem is in also splitting out the "@tag" into just "tag". "tag" could be any alphanumeric text either followed by a space or the end of the string.

Executing the commandline test above yields...

()(link|label label)( )(link)( @tag more text)()()

But I need it to yield...

()(link|label label)( )(link)( )(tag)(more text)

I've tried numerous regex strings like "[][][][]|@[[:alnum:]]*" which drops the entire word and yields...

()(link|label label)( )(link)( )( more text)()

and "[][][][]|@" which yields...

()(link|label label)( )(link)( )(tag more text)()

Any ideas?

Answer 1

I don't think you can do this in a single regex, but since you're using gawk, you can use some gawk extensions:

• save the separators using the split() function
• use the match() function

awk '{
    n = split($0, a, /\[\[|\]\]|@[[:alnum:]]+/, s)
    for (i=1; i<=n; i++) {
        printf "(%s)", a[i]
        if (match(s[i], /^@(.+)/, m))
            printf "(%s)", m[1]
    }
    print ""
}' <<END
[[link|label label]] [[link]] @tag more text
some text with @anothertag and [[another|link]]
END

()(link|label label)( )(link)( )(tag)( more text)
(some text with )(anothertag)( and )(another|link)()

Answer 2

This is ugly and horrible but provides the desired output:

$ echo '[[link|label label]] [[link]] @tag more text' | 
 awk -vFS="[\\\\[\\\\] @]" '{
  OFS=":"; 
  printf "\n(" $1 ")(" $3" "$4 ")(" $5 ")(" $9 ")(" $10 ")(" $13 ")("; 
  rest=$14;
  for(i=15;i<=NF;i++){rest=rest" "$(i)}
  printf "%s)\n", rest;

 }'
()(link|label label)()(link)()(tag)(more text)

The trick is setting the field separator to any of [,],@ or (space). If you can do so within your program, it should give the desired output.

Answer 3

Implemented Solution

This is, in effect, an implementation of glenn's answer with the addition of the user-defined function stringSplit() to use instead of the Gawk 3.1.6 builtin split() function which does not support the optional 4th argument ([seps], an array to hold seperators) that we need. Gawk 3.1.6 does support the similar purpose optional 3rd argument to match() needed, but [seps] is not available till Gawk 4.0.0.

# stringSplit(str,fld,rx,[sep])
#     Split string on regex delimeter preserving regex-seperators. Gawk 3.1.6
#     equivalent to builtin split() function of later versions which add
#     support for an optional 4th argument, ([seps]), an array to hold the
#     evaluated regular-expressions.
# Arguments:
#     str
#       string to split
#     fld
#       array of the resulting fields
#     rx
#       regular expression (regex) to split on
#     [sep]
#       optional array of seperator strings matching the regex
# Revised:
#     20140117 docsalvage
#
function stringSplit(str,fld,rx,sep,    searchstr,searchndx,match1,matchn,matches) {
  searchstr = str                     # copy of str to use in while() loop
  searchndx = match(searchstr, rx)    # index in searchstr where rx(regex) found
  match1    = searchndx               # preserve result of first match attempt
  matchn    = 1                       # match number (index in array of matches)
  matches   = 0                       # number of matches returned by split()
  #
  while (RLENGTH > 0) {               # more reliable than while(searchndx > 0)
    # save match
    sep[matchn] = substr(searchstr, searchndx, RLENGTH)
    #
    # match() only searches from beginning so give it just remainder of str
    searchstr   = substr(searchstr, searchndx + RLENGTH)
    #
    # printf("sep[%2d]: %s, searchndx: %2d, RLENGTH: %2d, searchstr: %s\n", matchn, sep[matchn], searchndx, RLENGTH, searchstr)
    #
    # search for next rx
    searchndx = match(searchstr, rx)
    matchn = matchn + 1
  }
  #
  if (match1)  matches = split(str,fld,rx)
  #
  return matches
}

BEGIN {
  print
  print "Test of:"
  print "  stringSplit()"
  print
  #
  str = "[[link|label label]][[link]] @tag more text some text with @anothertag and [[another|link]]"
  rx  = "[][][][]|@[[:alnum:]]+"
  #
  # fld - array of fields
  # sep - array of seperators
  #
  tags = 0
  matches = stringSplit(str,fld,rx,sep)
  #
  # arrayDebug("fld",fld)
  # arrayDebug("sep",sep)
  # print
  #
  print "Results:"
  printf "  "
  # per glenn jackman answer at
  #   http://unix.stackexchange.com/questions/109491/the-ere-regex-to-split-string-between-a-delimiter-and-end-of-word
  for (i=1; i<=matches; i++) {
    printf "(%s)", fld[i]
    if (match(sep[i], /^@(.+)/, m))  { printf "(%s)", m[1]; ++tags }
  }
  #
  print
  print
  print "Summary:"
  printf("  %d matches + %d tags = %d printed using regex(rx): %s\n  on string(str): %s\n", matches, tags, matches + tags, rx, str)
  print
}