1

I am working in CentOS 7 trying to do find / sed one liners to fix a ton of files. Specifically two in a row:

  1. First, to add "ignoreOlderThan = 14d" immediately after every [monitor://...] (working)
  2. Second, to find a [monitor://...] group that has two "ignoreOlderThan" and remove the last occurrence.

I have several hundred files that look similar to this (this is the current test file I'm using):

[default]
host = 10.2.2.15

[monitor://apath]
ignoreOlderThan = 14d
index=test
sourcetype=whatever
ignoreOlderThan = 30d

[monitor://truck]
ignoreOlderThan = 14d

[monitor://apath]
ignoreOlderThan = 14d
index=test
sourcetype=whatever
ignoreOlderThan = 30d

The first full command I use is:

find -name inputs.conf -exec sed -ie 's/\(\[monitor:.*\]\)/\1\nignoreOlderThan = 14d/g' {} +

This one works. It adds ignoreOlderThan = 14d immediately below a [monitor://...].

The second, more complex one, does not work:

find -name inputs.conf -exec sed -ie 's/\(\[monitor[^\]]+\][^\[]?\)\(ignoreOlderThan\s?=\s?[0-9]+\w\)\([^\[]+?ignoreOlderThan\s?=\s?[0-9]+\w\)\([^\[]+\)?/\1\3\4/g' {} +

I've tested several possible scenarios using regex101:

https://regex101.com/r/okCSfl/6

https://regex101.com/r/okCSfl/7

https://regex101.com/r/okCSfl/8

https://regex101.com/r/okCSfl/9

The regex works, so I think the issue is somewhere in the sed command, where I'm a lot weaker. I've escaped the parenthesis as needed for capture groups and the command runs... but it doesn't do anything. I thought it may be because sometimes the 4th capture group doesn't exist, but I've also tested a file where every group would have all 4.

I also read that some sed interprets everything as one line, which is why some of my test cases have no spaces at all between newlines.


Edit: @choroba pointed out that sed does one line at a time and suggested perl and gave an example. I played around a little and got it working with the following:

find -name inputs.conf -exec perl -0777 -pi -e 's/(\[monitor:[^[]+?)^(ignoreOlderThan\s?=\s?[0-9]+\w)([^[]+?^ignoreOlderThan\s?=\s?[0-9]+\w[^[]+)/$1$3/gms' {} +

Demonstrated here:

https://regex101.com/r/okCSfl/10

  • sed processes the input line by line. Its regex can't easily match multiple lines. – choroba May 22 at 14:12
0

sed processes the input line by line. Its regex can't easily match multiple lines.

Perl, on the other hand, can read the whole file when the option -0777 is specified:

perl -0777 -pe 's/^(\[monitor:[^[]+^ignoreOlderThan .*)^ignoreOlderThan = \w+/$1/gms' input > output
  • -0777 slurps the whole file
  • -p prints the input after processing it
  • /g repeats the substitution
  • /s makes . match a newline (which it normally doesn't)
  • /m makes ^ match at the beginning of each newline, not just the whole string (and similarly for $ but we don't need it here)
  • Interesting, that's good to know. How would you integrate this into the find portion? I know nothing about perl. The regex looks much different than what I was using. Does $1 equate to \1 in my sed? – user2752794 May 22 at 14:31
  • You can integrate it similarly to sed. Also, I tried a simple regex, but feel free to try anything else you like. $1 is the first matching group, \1 only works in the pattern itself, e.g. s/(.)\1/$1/. – choroba May 22 at 14:35
  • find -name inputs.conf -exec perl -0777 -pi -e 's/([monitor:[^[]+?)^(ignoreOlderThan\s?=\s?[0-9]+\w)([^[]+?^ignoreOlderThan\s?=\s?[0-9]+\w[^[]+)/$1$3/gms' {} + This is what I ended up with – user2752794 May 22 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.