4

I have a large JSON file in which I want to replace one string with another. It should not, but it might happen that this string would be used in a context where I do not want to replace it.

I know how many times it is in the file in the right context so I would like to also print how many occurrences of the string were replaced with sed. How can I do that?

For finding and replacing the string I use:

sed -i "" "s/my_string/new_string/g" my_file.json

Note that I'm on mac but I would need it for Linux too, like:

sed -i "s/my_string/new_string/g" my_file.json

I know I could just run a grep to find the string in the file and return the count like this:

grep -o my_string my_file.json | wc -l

but that's not what I'm asking. I am asking if there is a way how to do the same as any text editor (word, notepad, geany, ...) does -- if I give a string, it tells me how many times it saw this string and replaced it with another.

More information -- it will run in a Bash script so if there would be another better way, I am open to that.

6
  • 2
    awk would be a much better fit. It has a gsub() function which returns the count of substitutions it made. You can add that to a global count, and send it to stderr in an END block. You could also print each line that had any substitutions made for context, and with a file name and line number. Nov 9, 2021 at 12:00
  • 1
    Why would you have the JSON parsed by a text editor like sed? and not a JSON parser like jq?
    – Inian
    Nov 9, 2021 at 12:02
  • @schrodigerscatcuriosity no, grep will print each match on a separate line. For example echo "aaa" | grep -o a | wc -l will return 3.
    – terdon
    Nov 9, 2021 at 12:09
  • @Paul_Pedant thanks for the comment, would you like to write it into an answer?
    – My Work
    Nov 9, 2021 at 13:19
  • @Inian in this specific case, I do not really care about the fact that it has a json structure. The strings are not saved as separate variables -- they're wrapped in text and I don't think it would be easier. But maybe I'm wrong, I have not used jq ever before :)
    – My Work
    Nov 9, 2021 at 13:22

6 Answers 6

7

Just use perl instead. It is almost as portable as sed (more portable in this case since the syntax will be the same on any machine with perl installed), has very similar syntax for both the -i and the substitution operator, and can be told to print the number of substitutions:

perl -i  -lpe '$k+= s/my_string/new_string/g; END{print "$k"}' my_file.json

That will do the substitution in place and print the number of substitutions to standard output.

1
  • 2
    +1. Even better, use perl with its JSON library module. @MyWork, extracting, deleting, or changing data in structured text like json should be done with a parser, not with regular expressions alone.
    – cas
    Nov 9, 2021 at 12:57
6

If you must use sed then one way could be

As taken from the suggestions from @Phillipos it is changed to:

sed -i "" -e '
  s/my_string/new_string\
/g;s/\n//w /dev/stdout
  s///g
' my_file.json | wc -l
  • place a newline after every occurrence of my_string + do the changes also.
  • then take away one newline since sed implicitly puts one while printing.
  • the writing to stdout happens conditionally, only when the substitution succeeded , meaning when that line had my_string mng
  • then we take away the newline markers.
3
  • 1
    Genius! I love it! I'm just not sure whether \n will work with BSD sed on Mac. Any reason for not performing the regular substitution (line 5) along with adding the newline (line 2)?
    – Philippos
    Nov 9, 2021 at 14:55
  • My attempt is regular sed. The optimizations you suggested are actually hi quality and I am going ahead and using them, thanks.!
    – guest_7
    Nov 9, 2021 at 16:04
  • Now with the literal newline it should work on any sed I can remember. I admire the solution. (-:
    – Philippos
    Nov 9, 2021 at 16:44
4

You can force the vim editor to report in streaming mode:

ex -nsc 'redir! >/dev/stderr' -c '%s/pattern/PATTERN/g' -c 'redir END' -c 'wq' my_file

3 substitutions on 2 lines

ex - mode of vim (or vim -e)
-n - no create swap file
-s - script
-c cmdline mode (or +'command')
'redir! >/dev/stderr' - redirecting to shell standard error
'redir END' - end of redirecting. Can be omitted
'wq' - save changes and exit the editor. If you replace it with 'q!', then you can get the output and compare with expectations without changing the file.

1
  • Thanks for the explanation of the code, I really appreciate it.
    – My Work
    Nov 16, 2021 at 14:11
3

sed doesn't count anything but line numbers. So if you want to use sed and avoid using other tools like guest_7 did, you need to teach it to count:

sed -n -i"" ':start
  s/my_string/new_string/
  Tcont
  H
  bstart
  :cont
  p
  $!d
  x
  y/\n_/_\n/
  s/_[^_]*/-/g
  s/^/0123456789+_0/
  :loop
  s/\(.\)\(.\)\(.*_.*\)\1-/\1\2\3\2/
  s/_+/_10/
  s/\(.\)\(.\)\(.*_.*\)\1+/\1\2\3\20/
  tloop
  s/.*_//
  w/dev/stdout
  d' my_file.json

It's probably too obvious to explain. We need to loop over the replacement and count by adding a line to Hold space. The handling for the last line is about couting the lines collected in hold space.

3
  • 2
    "It's probably too obvious to explain". I cannot overstate how wrong that statement is! :) I am no sed expert, but I do have a basic familiarity with it and am very comfortable with regexes, and this just looks like dark magic to me. I suspect you are doing something very clever here, but some explanation would be really helpful.
    – terdon
    Nov 16, 2021 at 15:08
  • @terdon You got my irony, I guess. The real reasons for not giving an explanation are (1) this task makes no sense besides coding fun and (2) I'm dissatified with the solution. It should be more elegant instead of this quick&dirty one.
    – Philippos
    Nov 16, 2021 at 17:31
  • 1
    Ah. I will blame this on the limits of text-based communication, then. I honestly thought that you were so used to sed syntax that it genuinely felt obvious to you!
    – terdon
    Nov 16, 2021 at 17:49
2

This is a bare-bones awk version to count changes and affected lines:

#! /bin/bash

Awk='
BEGIN { fmtEnd = "Made %d substitutions on %d lines.\n"; }
{
    n = gsub (/exit/, "return");
    if (n) { Lines++; Count += n; }
    print;
}
END { printf (fmtEnd, Count, Lines) > "/dev/stderr"; }
'
    awk "${Awk}" doFifo > doFifo.fix

Output (stderr) is merely this, which could be rearranged to make it easier to recover the count:

Made 8 substitutions on 6 lines.

GNU/awk does have the -i inplace extension, but I am ultra-conservative about update-in-situ. My clients complain a lot, and assert their data is always 100% correct, so I keep audit trails and every data version.

Below is an awk variant which notes every changed line. This is still not production level: I would want it to accept the pattern and replacement as args, deal with multiple files in one run, name the output files based on the inputs, and summarise by file and overall total. Maybe allow an array of pattern -> replacement too.

#! /bin/bash

AwkFull='
BEGIN {
    reFix = "exit"; txFix = "return";
    fmtEnd = "Made %d substitutions on %d lines.\n";
    fmtSub = "\n.... %d Changes on file %s line %d:\n";
    fmtSub = fmtSub "Was: %s\nNow: %s\n";
}
{
    New = $0;
    n = gsub (reFix, txFix, New);
    if (n == 0) { print $0; next; }

    Lines++; Count += n;
    printf (fmtSub, n, FILENAME, FNR, $0, New) > "/dev/stderr";
    print New;
}
END { printf (fmtEnd, Count, Lines) > "/dev/stderr"; }
'
    awk "${AwkFull}" doFifo > doFifo.fix

This shows every changed line like:

.... 2 Changes on file doFifo line 64:
Was:    (exit)  printf 1>&7 '%(%T)T  Received exit command\n' -1
Now:    (return)    printf 1>&7 '%(%T)T  Received return command\n' -1

Edit: Making the parameters into command arguments.

The first version above embedded the pattern and replacement text inside the gsub command itself. The second version made it easier to change these by (a) giving them names, and (b) declaring them at the head of the code.

The next stage in generalising the code would be to pass these from the shell. This is easy in awk. First, remove the line that defines reFix and txFix (convention is re for regular expression and tx for text, but call variables anything you like provided you are consistent).

To get shell strings into awk variables, there is a -v option. So your awk command becomes:

awk -v reFix="exit" -v txFix="return" "${AwkFull}" doFifo > doFifo.fix

and the final step to use shell variables is to use any form of shell substitution, like:

awk -v reFix="${1}" -v txFix="${myNew}" "${AwkFull}" doFifo > doFifo.fix

There are two (maybe more) downsides:

(1) awk knows that /exit/ is a pattern. In some cases, you may need to clarify the syntax: for example, a simple line-match like /exit/ will need to be rewritten as $0 ~ reFix. But awk knows the first arg to gsub() is a pattern, so that syntax works unchanged. (See https://www.gnu.org/software/gawk/manual/gawk.html#Strong-Regexp-Constants for more.)

(2) Patterns in variables do not get syntax-checked when the awk program is first read, only when they are used. So user-entered patterns may easily break in the middle of a run, and with obscure error messages.

4
  • Thank you for your comment about GNU/awk's -i flag. I'm wondering if a general question on "update-in-situ" would make a good U&L post? Or if one exists already, you could add a link? Nov 11, 2021 at 19:32
  • This is very pretty, thanks. Can I have one question? How do you pass variables to it, ie. how would you add them from outside? Eg. from the command line in your case.
    – My Work
    Nov 16, 2021 at 14:10
  • 1
    @MyWork Added a section on passing shell variables into GNU/awk (earlier awks may not have the -v option). Nov 17, 2021 at 22:27
  • @Paul_Pedant wow, thanks a lot, this is amazing!
    – My Work
    Nov 18, 2021 at 6:48
1

Using Raku (formerly known as Perl_6)

raku -pe 'state $k += s:g/ old_string /NEW_STRING/; END put $k;'

OR

raku -pe 'state $k; s:g/ old_string{++$k} /NEW_STRING/; END put $k;'

Above is a solution coded in Raku, a member of the Perl family of programming languages. The first solution is a rather direct translation of @terdon's Perl (5) code, posted elsewhere in this thread. The second solution demonstrates Raku's ability to execute code within {...} blocks, inside a regex m/.../ and/or substitution s/// matcher. Both examples write the number of substitutions performed on the last line of the file/output.

Briefly, raku is called at the command line with the -pe (autoprinting, linewise) flags. Variable $k is initialized with the state keyword, meaning it gets declared only once during execution. After the state declaration, $k is free to be incremented, either by the += operator (first example), or the ++$k pre-increment operator (second example). Note: within the left-half (but not right-half) of Raku's s/// matcher whitespace is not significant, so you can feel free to space out regex elements (to enhance readability).

Both Raku and Perl(5) are zero-indexed, so counting within the autoprinting loop starts from zero. However there is some additional flexibility with Raku if you discard the -pe autoprinting flags and use -e with a for lines() routine instead: you get to choose your counter's initial value. Let's say you want to perform 10 substitutions: you could take the code below, set $k=10 and decrement using --$k, checking the last line of the file to see if you've reached zero:

raku -e 'state $k=10; for lines() {S:g/ old_string{--$k} /NEW_STRING/.put; END put $k};'

[Addendum: It's not entirely clear to me within a raku -pe '' autoprinting loop why state $k=10; does not retain and then increment/decrement the initial value (currently the behavior is to start from zero regardless). I'll update this answer once I get feedback on this issue].

https://docs.raku.org/syntax/state
https://raku.org

2
  • 1
    Thanks for the explanation, I really appreciate it.
    – My Work
    Nov 16, 2021 at 14:09
  • @MyWork You're welcome. Writing just to say as a matter of daily practice I'll get counts and/or do replacements in vim with :%s/old_string//gn, wherein the trailing g means 'global' and the trailing n means 'count don't delete'. However the Raku approach is much more powerful because (if it's not obvious from above), you can count multiple substitutions in one go, as in ...s:g/ cat{++$m} | kitty{++$n} /feline/;... . Anyway, vim can help to serve as a quick estimate and raku can serve as a double-check on vim. Nov 17, 2021 at 15:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.