How to improve this 'sed' search & replace command?

Question

How to search and replace recursively through multiple files in a directory, using essential tools installed in mostly any Debian/Ubuntu machine?

There are multiple answers in Stack* where to find answers to this question, such as here or here. But all are somewhat lacking in essential ways. They don't provide a correct solution except for some "easy" subset of possible inputs.

After some searching and careful study of manpages for grep, xargs and sed, this is the best "search and replace" command I've been able to build for Bash:

grep -ErlIZ -- '<OldPattern>' . | xargs -0rL1 sed -ri 's/<OldPattern>/<NewPattern>/g'

(note I want being able to use helpful and advanced shell features where possible so I'm not worried much yet about POSIX or portability -- and I don't care much either about mostly outdated versions of GNU tools in Mac)

This one-liner has multiple features:

• Explicitly ignores binary files, for safety (not sure if this is really needed, though)
• Uses grep | xargs to filter out candidate files and provide good performance in huge directories
• Accepts patterns that start with a dash (-)
• Accepts paths with spaces
• Accepts regex capture groups in the search patterns

But due to deficiencies in the sed feature set, the regex engine is always greedy and there is no option to disable this behavior (only ugly workarounds). This means only one substitution can be done per line, at least for some cases (I can show some examples if requested).

Resorting to a while loop makes it run as many times as needed to really cover all possible substitutions:

while FILES="$(grep -ErlI -- '<OldPattern>' .)"; do
    echo "$FILES" | xargs -rL1 sed -ri 's/<OldPattern>/<NewPattern>/g'
done

But now Bash cannot store null bytes, so options grep -Z and xargs -0 had to be dropped. I believe this drops compatibility with paths that contain spaces.

• Is it possible to combine the while loop solution with the -Z, -0 options to support paths with spaces?

• Or maybe... is there any other, different but better way to build a robust and reliable search-and-replace command? (succinct is a feature so, as close to a one-liner as possible)

---

EDIT: Adding an example where the greedy regexp in sed is a problem for the non-loop version.

With this input line:

set(requires "gstreamer-1.5 gstreamer-base-1.5 gstreamer-sdp-1.5 libjsonrpc")

The pattern (gst.*)1\.5 would match this:

set(requires "[gstreamer-1.5 gstreamer-base-1.5 gstreamer-sdp-1.5] libjsonrpc")

Because it is greedy, it gets the from the first gst to the last 1.5. Say the substitution is \1AAA: the \1 will keep the (capture group), and the AAA will just print these letters instead of the original 1.5. The result will be:

set(requires "gstreamer-1.5 gstreamer-base-1.5 gstreamer-sdp-AAA libjsonrpc")

So this command would need to be run 3 times in total to actually substitute all possible matches in this line. The while loop version just runs everything once and again until the search pattern cannot be found any more, which is when the replace work has actually been finished.

Answer 1

If you want to run a substitution again and again as long as it succeeds, that can be done in sed with a conditional loop using the t command:

grep -ErlIZ -- '<OldPattern>' . |
   xargs -r0 sed -Ei -e :1 -e 's/<OldPattern>/<NewPattern>/g' -e t1

Here also passing as many files as possible to sed instead of running one sed per file for efficiency, and using -E which is more portable than -r outside of GNU systems (and consistent with grep -E).

bash can't store NUL in its variables but you can use an array instead to store the list of files.

With bash 4.4+:

readarray -td '' files < <(grep -ErlIZ -- '<OldPattern>' .)

Which you can then output with:

((${#files[@])) && printf '%s\0' "${files[@]}" | xargs -r0 ...

Or use a temp file. On Linux, that could be done as:

exec 3<<EOF # creates a deleted empty temp file opened on fd 3
EOF

grep -ErlIZ -- '<OldPattern>' . > /dev/fd/3 || exit

# and later:
while xargs -r0a /dev/fd/3 ...; do...

exec 3<&- # file was already deleted, closing it means its data is now
          # reclaimed.

Your (gst.*)1\.5 should probably be: (\<gst[^[:space:]]*)-1\.5\> if you want the variable part not to including whitespace character and it not to match on tagst-1.11.51 for instance.

Using non-greedy operators would probably not help much in that example. A perl-like gst.*?1.5 would still match gstreamer-1.3 foobar-1.5 in set(requires "gstreamer-1.3 foobar-1.5 gstreamer-sdp-AAA libjsonrpc")