2

I need to replace strings in many large textfiles but I have a list of exception strings (200+ items). For example:

# I want to replace every "dank". Except when it comes in the following form:
 
dankine
dankzwd
nudankip
dankphys
danko.mod
... (The list is 200+ items long)

My current regex looks like this:

sed -e "s/dank/monk/g" /path/to/file

The content of the file looks like this:

xdankine redankus
dankzwd 
danke dankbe
testdank

this is the content of the file after execution:

xmonkine remonkus
monkzwd 
monke monkbe
testmonk

But I want the content to look like this:

xdankine remonkus
dankzwd 
monke monkbe
testmonk

since dankine and dankzwd are in my exclusion list.

The file can contain more than one possible replacement per line.

How can I accomplish this?

4
  • If it's always surrounded by spaces, you can change the pattern to s/ dank / monk / - or use character classes (\s for whitespace especially)
    – Panki
    Aug 19 '21 at 12:41
  • @Panki Unfortunatly it is not alway surrounded by spaces.
    – Jan
    Aug 19 '21 at 12:44
  • Another question occurred about what you expect for dankdank if dankda is in the exclusion list: 1) The whole dankda is protected, so it needs to stay dankdank. 2) The dankda protects only the dank, but not the da, so it can become dankmonk. 3) After the second dank of 2 has been replaced the first one is not protected anymore and it shall be monkmonk. 4) You don't care at all.
    – Philippos
    Aug 20 '21 at 12:17
  • @Philippos Case 1) should be the exptected behaviour.
    – Jan
    Aug 20 '21 at 12:38
4

If you have only one occurences of dank per line like in your example, you can use inverted addresses:

sed -E '/dankine|dankzwd|nudankip|dankphys|danko\.mod/!s/dank/monk/'

If there can be multiple occurences per line, you could use a character which can't be part of the file, e. g. #, change all dank to #, change back for the word list and change # to monk for the rest:

sed 's/dank/#/g;s/#ine/dankine/g;s/#zwd/dankzwd/g;s/nu#ip/nudankip/g;s/#phys/dankphys/g;s/#o\.mod/danko.mod/g;s/#/monk/g'

(If any character can occur, use a newline instead)

Update: New requirement to read the exclusion list from a file

Write your blacklist to the file exclusion.list with a trailing newline (the script will use this to detect where the first file ends):

sed -e '1,/^$/{H;d;}' -e 'G;s/\n/&&/;:loop' -e 's/\(.*da\)\(nk.*\)\(.*\n\1\2\n\)/\1#\2\3/;tloop' -e 's/\n.*//;s/dank/monk/g;s/da#nk/dank/g' exclusion.list file

or, as multiline perhaps easier to read

sed '1,/^$/{H;d;}
  G
  s/\n/&&/
  :loop
  s/\(.*da\)\(nk.*\)\(.*\n\1\2\n\)/\1#\2\3/
  tloop
  s/\n.*//
  s/dank/monk/g
  s/da#nk/dank/g' exclusion.list file

Anyhow, this may still be easier to write than to read. The concept is

  • Read the exclusion list to hold space
  • For each line of the file append that list in the hold space
  • Replace each dank in the file which appears in the list by da#nk to prevent it from being replaced later
  • Then remove the list, replace all dank by monk and finally remove the # from the da#nks.

Adding l after the :loop illustrates the working principle.

Thanks to Stéphane for hinting the dankfoodank problem, which is solved here. The requirement for dankdank case however remains unclear Should it be dankmonk because only the dank of dankda is protected or should it remain dankdank because the da of the second dank is protected as being a part of dankda or is this out of scope?

3
  • @Jan that is the sort of thing you need to mention in the question so we don't waste our time and yours providing solutions that won't help you. Please edit your question and make sure all of your requirements are listed.
    – terdon
    Aug 19 '21 at 13:47
  • That doesn't work if for instance you have both foodank and dankfoo as exclusions and dankfoodank in the input. Also if there's a dankda exclusion and dankdank in the input like with @guest_7's answer. Aug 20 '21 at 11:03
  • Thanks, @StéphaneChazelas I addressed the dankfoodank case in my update, but I'm not sure about the right handling of the dankdank case.
    – Philippos
    Aug 20 '21 at 12:12
2

Using any awk in any shell on every Unix box and using literal string operations so we don't care about any regexp or backreference metachars in the input or exceptions list:

$ cat tst.awk
NR==FNR {
    mask[$0] = RS NR RS
    next
}
{
    delete changed
    for (exception in mask) {
        while ( s=index($0,exception) ) {
            $0 = substr($0,1,s-1) mask[exception] substr($0,s+length(exception))
            changed[exception]
        }
    }

    gsub(/dank/,"monk")

    for (exception in changed) {
        while ( s=index($0,mask[exception]) ) {
            $0 = substr($0,1,s-1) exception substr($0,s+length(mask[exception]))
        }
    }

    print
}

$ awk -f tst.awk exceptions file
xdankine remonkus
dankzwd
monke monkbe
testmonk

The above assumes you don't have exceptions that are substrings of other exceptions like dankfoo and dankdankfoo since you don't show cases like that in the example in your question. If you do then make sure the exceptions file is sorted such that the longer superstrings come before the shorter substrings and iterate on them in the order they were input so you don't replace xdankdankfooy with xdank<replacement>y instead of x<replacement>y when masking the exceptions in the first loop.

1

So as not to overload the regex since the exclusion list can be 200+ strong, we first generate the sed code using the exclusion list file and apply that generated code on the data input.

GNU sed

sed -e '
  1i\
s/dank/\\n/g
  h;s:[\&/]:\\&:g
  x;s/dank/\n/g
  s:[][^$\/.*]:\\&:g
  s/\n/\\n/g;G
  s:\n:/:;s:.*:s/&/g:
  $a\
s/\\n/MONK/g
' excludes.txt | sed -f - file

Output:-

xdankine reMONKus
dankzwd
MONKe MONKbe
testMONK

Proof of concept:-

  • First off, turn all danks to literal newlines, a char guaranteed not to be found.
  • Then turn a line in exclude list like nudankip to as given below and likewise for all lines in exclude list.
  • s/nu\nip/nudankip/g
  • The complexity is due to the fact that we need to escape the exclude list fir the rhs and lhs of a sed s/// expression.
7
  • 1
    That doesn't work if you have for instance dankda as an exclusion and dankdank in the input (in which case you get MONKMONK instead of dankMONK) Aug 20 '21 at 10:55
  • 1
    Please note that using \n for newline in the replacement is not POSIX and will fail with most sed implementations. It will work for GNU sed, however.
    – Philippos
    Aug 20 '21 at 11:04
  • Thanks for your answer. Is it possible to expand the sed with with a second regex? Like a second -e ?
    – Jan
    Aug 20 '21 at 13:24
  • @Jan yes you can add multiple -e
    – guest_7
    Aug 20 '21 at 14:39
  • @Stephane you are correct but since the OP hadn't given any indications of such inputs and what heuristics to choose in getting the output so I refrained from going there.
    – guest_7
    Aug 20 '21 at 14:43
1

With perl, you could do:

perl -pe '
  BEGIN{
    chomp (@excl = <STDIN>);
    $re = "(" . join( "|", map {qr{\Q$_\E}} @excl) . ")|dank"
  }
  s{$re}{$1//"monk"}ge' input < exclusion.list

That constructs a regexp such as:

(dankine|dankzwd|nudankip|dankphys|danko\.mod)|dank

And we replace any occurrence of it with either $1 (so what was matched, so essentially doing nothing) if $1 is set (one of the exclusions was matched) or monk otherwise (dank was matched instead).

Note that if the exclusions include both dankzwd and zwddank, that will still turn dankzwddank into dankzwdmonk as it replaces dankzwd with dankzwd ($1) first, and then only dank is left for it to replace.

An approach to address that would be to record all the places where any exclusion occurs in a mask string, and then when it comes to replace dank, only do the replacement where the mask says it's fine.

perl -spe '
  BEGIN {
    chomp (@excl = <STDIN>);
    $word_len = length $word;
  }
  my $len = length;
  my $mask = "-" x $len;
  my $i;
  for my $e (@excl) {
    my $e_len = length $e;
    my $hide = "#" x $e_len;
    for (my $o = 0;
         $o < $len && ($i = index($_, $e, $o)) >= 0;
         $o = $i + 1) {
      substr($mask, $i, $e_len) = $hide;
    }
  }
  s{dank}{substr($mask, pos, $word_len) =~ /-/ ? $repl : $&}ge
  ' -- -word=dank -repl=monk input < exclusion.list

So for instance, if an input line contains:

dodankdankdankoodankdodank

and the exclusions have: dankdank, dankdo the mask would be built progressively from

 --------------------------
 --########---------------- # first dankdank
 --############------------ # second dankdank
 --############--######---- # first and only dankdo
 dodankdankdankoodankdodank
                       ^^^^

Then that leaves only the dank at the end that is not affected by exclusions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.