sed using double quotes with \!b leads to unknown command on back slash

Question

I currently have a sed command that I want to act on the following type of text:

user:
        ensure: 'present'
        uid: '666'
        gid: '100'
        home: '/home/example'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: ''

I can get the type of results I want with this command:

sed '/user:/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27\!\!\x27/g'

user:
        ensure: 'present'
        uid: '666'
        gid: '100'
        home: '/home/example'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '!!'

The problem with that command is that user: is static. I want to be able to iterate through a list of users and use a bash variable instead of a specific user in the sed command. To do that though, I need to use double quotes instead of single quotes. However, when I use this command:

sed "/user:/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27\!\!\x27/g"

It complains about the '!' in the !b sed command I'm using.(since bash is trying to interpret it) However, if I escape it like so:

sed "/user:/\!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27\!\!\x27/g"

Then I get this error:

sed: -e expression #1, char 6: unknown command: `\'

How can I get this to work?

Answer 1

Quoting does not delimit fields.

This is an important, but often forgotten, aspect of shell language syntax. The mental model that one "quotes arguments" is in fact simplistic and wrong. One quotes stuff that needs quoting, and that need only be part of what ends up as a single argument.

And that is exactly what you need to do here. Ironically, the first now-deleted answer was very close.

The final string that you want to give to sed as the actual single argument that it executes with is

/user:/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27!!\x27/g

with the user part varying according to the actual user name. But to get here you do not need to use one single quoting scheme for the entire argument. You can build up the argument from parts.

Use double quotes for the part where you need parameter expansion to occur, and single quotes for the parts where you need history expansion to not occur:

while read -r i
do
   sed -e '/'"$i"':/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27!!\x27/g' puppet-users.yaml > puppet-users{new}.yaml
   mv puppet-users{new}.yaml puppet-users.yaml
done < user_list.txt

This is:

1. The single-quoted character /.
2. The double-quoted characters $i which are subject to parameter expansion (and of course all other expansions and substitutions).
3. The single-quoted characters :/!b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27!!\x27/g which are not subject to history expansion.

Field splitting only occurs on unquoted characters; and there are none such here at all, so this is all one field, which becomes all one argument to the sed command.

Once you've done this, you'll discover that you've used too many TAB characters in what you are doing. ☺

On the gripping hand …

sed is not quite the right tool for this job. Nor indeed is awk, as in the second now-deleted answer. Parsing YAML using a proper YAML parser, such as Python's yaml module, will be better in the long run, if the task at hand becomes less trivial as it quite probably will.

And this is a simple one-liner with a proper tool for the job, such as various yq tools, whose usage will be roughly (since, alas, they all differ):

while read -r i
do
   yq < puppet-users.yaml "$i".password="'\!\!'" > puppet-users{new}.yaml
   mv puppet-users{new}.yaml puppet-users.yaml
done < user_list.txt

Further reading

• https://stackoverflow.com/questions/5014632/

Answer 2

Thanks to @steeldriver for the assist in the comments. I ended up resolving the issue by refactoring my command to be:

sed "/user:/{n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: \x27\!\!\x27/g}"

I was then able to replace user: with a bash script variable.

Answer 3

May I suggest you use ex, the predecessor and non-visual form of vi, for your multiple addressing requirements:

ex file.txt <<'EOF'
0/^user://password:/s/:.*/: '!!'/
0/^something else://subline to change:/s/:.*/: new value/
x
EOF

0 means line 0; that is, begin that search from the start of the file.

/^user:/ is an address. It works similarly to Sed or Awk addresses, except that in ex you can specify an additional address which is taken from the line specified by the first address. (You can even use backward addressing.)

/password:/ is another address. So this specifies the line found by searching forward for the pattern password: from the point where the pattern ^user: was found.

The s/old/new/ command works just as in Sed.

x means save and exit.

The whole thing is wrapped in a Bash "here doc."

---

As pointed out in the comments, the original purpose was to iterate through a list of such users rather than hardcoding a static user.

To do this, assuming a text file userlist.txt containing e.g.

user1
user2

And further assuming that you want to perform the same substitution for each user (namely, setting the "password" field to '!!'), simply use sed to convert the userlist into an ex command.

Note that since !! will be expanded in an interactive shell, to use this interactively, first run

set +H

Then:

< userlist.txt sed -e "s_.*_0/^&://password:/s/:.*/: '!!'/_" -e $'$a\\\nx' | ex file.txt

Easy as pie. ;)

---

Demonstration:

$ cat file.txt 
user1:
        ensure: 'present'
        uid: '666'
        gid: '100'
        home: '/home/example1'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '1'
user2:
        ensure: 'present'
        uid: '667'
        gid: '100'
        home: '/home/example2'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '2'
user3:
        ensure: 'present'
        uid: '668'
        gid: '100'
        home: '/home/example3'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '3'
user4:
        ensure: 'present'
        uid: '669'
        gid: '100'
        home: '/home/example4'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '4'
$ cat userlist.txt 
user3
user2
$ set +H
$ < userlist.txt sed -e "s_.*_0/^&://password:/s/:.*/: '!!'/_" -e $'$a\\\nx' | ex file.txt
$ cat file.txt 
user1:
        ensure: 'present'
        uid: '666'
        gid: '100'
        home: '/home/example1'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '1'
user2:
        ensure: 'present'
        uid: '667'
        gid: '100'
        home: '/home/example2'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '!!'
user3:
        ensure: 'present'
        uid: '668'
        gid: '100'
        home: '/home/example3'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '!!'
user4:
        ensure: 'present'
        uid: '669'
        gid: '100'
        home: '/home/example4'
        comment: ''
        password_max_age: '99999'
        password_min_age: '0'
        shell: '/bin/false'
        password: '4'
$ 

Voila.

Answer 4

The sed solution here will become awful as soon as you need to skip more lines. Here is a solution using awk:

awk '
/user:/ { 
   NR=0
   edit=1
} 
NR==9 && edit { 
   $0="\tpassword: '\'\!\!\''"  
} 
1' input_file

Or as a one-liner:

awk '/user:/{ NR=0; edit=1 } NR==9 && edit{ $0="\tpassword: '\'\!\!\''" } 1' input_file

Answer 5

! is a history expansion character. History expansion is a feature of bash (and a few other shells) when they're running interactively — in a script it has no special meaning (unless you explicitly enable it).

! retains its history expansion meaning inside double quotes. But inside double quotes, \! means backslash-bang. So there's no way to include an exclamation mark inside double quotes. You have to use some other form of quoting: either \! outside of any quotes, or ! inside single quotes. (Another solution would be to put the ! in a variable.)

Nothing prevents you from mixing forms of quoting in the same word, e.g..

sed "/$username:/"\!"b;n;n;n;n;n;n;n;n;n;s/.*/\t\tpassword: '"\!\!"'/g"

(But I have to agree with JdeBP, this is not a job for sed.)