I need to parse a big file replacing possibly occurring substrings, that are part of possibly occurring strings that match a regex, with a value taken from an array whose indexes are the substrings in question.

The file is a regular text file, i.e. has lines separated by newline character, and each line can have any arbitrary character between ASCII 32 and ASCII 126, basically any printable character except control ones in the C locale.

The extended regex that precisely matches the interesting strings is \<prefix-[[:alnum:]]{2,}\>, while the substring in question is whatever comes after the dash sign.

With a sample (synthesized) input like:

# arbitrary number of comment lines of any length
:prefix-foo ; arbitrary strings
# arbitrary number of comment lines of any length foo -prefix-foo-
-bar -foo-xx arbitrary string -yet-more strings prefix-foo-bar MORE strings
YET more --STRINGS prefix-bar -prefix-foo-STRingS--
even MORE strings ; prefix -foo -yy--more-and-prefix-bar-and-more

and having a sample dictionary like:

dictionary["foo"] = 2
dictionary["bar"] = 15

the desired output would be:

# arbitrary number of comment lines of any length
:prefix-2 ; arbitrary strings
# arbitrary number of comment lines of any length foo -prefix-2-
-bar -foo-xx arbitrary string -yet-more strings prefix-2-bar MORE strings
YET more --STRINGS prefix-15 -prefix-2-STRingS--
even MORE strings ; prefix -foo -yy--more-and-prefix-15-and-more

I thought awk would be the best tool particularly for its native ability to rewrite a whole $0 record by just replacing the single $1...$n fields. I therefore came up with the following script:

#!/usr/bin/gawk -f

    # first fill in dictionary
    while ("cmd-providing-dictionary" | getline) {
            dictionary[$1] = $2
    # pattern that matches interesting fields
    field_regex = "\\<prefix-[[:alnum:]]{2,}\\>"
    # I don't care default splitting of line
    FS = OFS = ""
    # split line in fields as per regex
    if (patsplit($0, fields, field_regex, seps)) {
        FS = OFS = "-"
        # for each field, split it on dash character,
        # modify its substring as per dictionary,
        # and finally rebuild it
        for (fn in fields) {
            $0 = fields[fn]
            if ($2 in dictionary) {
                    $2 = dictionary[$2]
                    fields[fn] = $0
        FS = OFS = ""
        # clear whole record and rebuild it with
        # fields computed above + original separators
        $0 = ""
        for (fn in fields)
            $fn = seps[fn - 1] fields[fn]
        $(fn+1) = seps[fn]

Albeit I'm not Strong with the Awk, the above seems to be doing the correct job quickly enough, but looks a bit clumsy and feels like I'm forcing awk doing things in an unnatural way. I'm wondering whether there's a better way to obtain the same result. Or also a better tool.

My first idea was a simple regex replacement using gsub() or gensub() but I found no (clean) way to use a regex's subexpression (which would be \<prefix-([[:alnum:]]{2,})\>) as the key to lookup the array and use that value in the replacement string. On the other hand, looping over all dictionary keys to apply always all gsubs is not really feasible because the dictionary is very large and would thus be very inefficient.

  • 1
    I don't see how to do better with awk. For the second part you could use split(fields[fn],array,"-") to not use FS and OFS, but then you would need to cope with a split into more than 2 parts. – meuh May 25 at 18:28

Just for comparison, here is an non-expert perl version which gains a lot by being able to call a function from within a substitution. It is as if you could say

gsub(regexp, call_function(matched_part), variable_to_change)

where the function returns the replacement string.

use strict;
my %d;
sub fix{
  my ($prefix,$str) = @_;
  $str = $d{$str} if defined $d{$str};
  return "$prefix$str";
open(D,"dictionary") or die;
  $d{$1} = $2 if $_ =~ m/^([^ ]+) ([^ \n]+)/;
  $_ =~ s/\b(prefix-)([[:alnum:]]{2,})\b/fix($1,$2)/ge;

Here the substitute command $_ =~ s/regex/fix($1,$2)/ge changes the current line $_ globally (g), and executes (e) the replacement string fix(), where $1 and $2 are the captured groups (within ()) of the regex.

  • Thank you. Yes, that’s the sort of thing I thought could be possible with awk too. Perl is really the swiss-army knife they say it is. Pity of me I always disliked its syntax and refused to learn it :-D – LL3 May 26 at 16:51
  • 1
    Yes, perl syntax is hard, and I only use it for small solutions for well-known problems it handles well, like manipulating strings. Note, python has a similar feature re.sub(regex, replaceby, instring) where replaceby can be a function, and it is a cleaner language. – meuh May 26 at 17:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.