3

I have 100+ jinja template files with 0-k occurrences of "value: ..." string in each of them. The problem is that some of the files are using:

value: something

some of them:

value: 'something'

and some of them:

value: "some other thing"

I need all of these to look the same, to use double quotes. I thought I'd do it with sed:

sed -i 's/value: ['"]?(.*)['"]?/value: "\1"/g' *.j2

but as you can see I'm quite horrible with sed and the past 2 hours only made me want to break my keyboard with the nonsense error messages I'm getting, like: unterminated `s' command and such.

Sample input:

- param:
  name: Command
  type: String
  value: '/bin/echo'
- param:
  name: Args
  type: String
  value: Hello World
- param:
  name: Something
  type: EnvVar
  value: "PATH"

from this I need to get:

- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"
4
  • Please edit your question and add an example of your files. When present, is the closing quote always the last thing on the line? Do you have other quotes that need to be unchanged?
    – terdon
    Commented Sep 21, 2020 at 10:37
  • Do you need the "value" in quotes as well? Best thing is give us a sample of input and expected output
    – GMaster
    Commented Sep 21, 2020 at 10:39
  • 1
    Welcome to the site. Please note that you have literal single quotes in a sed program enclosed in single quotes. This will not work. You can replace the literal single quotes in your regular expression with \x27 instead.
    – AdminBee
    Commented Sep 21, 2020 at 10:40
  • Thanks for the comments. I added a sample input and the expected output. @AdminBee: I already tried escaping pretty much everything... I'm really miserable with sed. Commented Sep 21, 2020 at 10:45

4 Answers 4

6

When you need to use the two forms of quotes ("') in the expression, things get tricky. For one, in your original attempt the shell identifies this 's/value: [' as a quoted string: the latter quote is not preserved.

In these cases, rather than having a headache, you can simply put the Sed commands in a file. Its contents won't be subject to the shell manipulation.

quotes.sed:

# (1) If line matches this regex (value: '), 
# (2) substitute the first ' with " and
# (3) substitute the ' in the end-of-line with ".
/value: '/{
  s/'/"/
  s/'$/"/
}
# (4) If line matches this regex (value: [^"]), 
# (5) substitute :<space> for :<space>" and
# (6) append a " in the end-of-line.
/value: [^"]/{
  s/: /: "/
  s/$/"/
}
$ sed -Ef quotes.sed file
- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"
0
4

The problem with your sed call is two-fold

  • You have literal single quotes as part of your regular expression ['"] in a sed program enclosed in single quotes. This will not work as single quotes inside single quotes cannot be escaped. You can represent them as \x27, however.
  • The ... (.*) ... syntax is extended regular expression syntax for having ( and ) mean "capture group" by default. Your should either use the -E option of sed, or define the capture group as \(.*\) to use basic regular expressions (but then you would need to replace the ? with something like \{0,1\}, too).

The following worked in my tests:

sed -E 's/value: [\x27"]?([^\x27"]*)[\x27"]?/value: "\1"/' input.j2
11
  • With this exact command I get: invalid reference \1 on `s' command's RHS. I tried a similar version with [\'"]? already which yield the same error. Commented Sep 21, 2020 at 10:54
  • Can you state the sed version in your question text? It works on GNU sed 4.2.2 with the example file you provided.
    – AdminBee
    Commented Sep 21, 2020 at 10:58
  • GNU sed 4.2.2. My OS is RHEL7 if that makes any difference. Commented Sep 21, 2020 at 11:00
  • What happens if you use sed 's/value: [\x27"]\{0,1\}\(.*\)[\x27"]\{0,1\}/value: "\1"/'?
    – AdminBee
    Commented Sep 21, 2020 at 11:02
  • 1
    @Displayname it's hard to believe you get the error message you say you do from the script in this answer. I can see how you'd get an error with the "similar version" you mentioned in your comment. Please copy/paste the script from this answer exactly as-is and try it again. I'm thinking you either removed the -E or changed it to -e or you escaped the round brackets.
    – Ed Morton
    Commented Sep 21, 2020 at 17:32
3

With GNU awk for the 3rd arg to match() and gensub():

$ awk 'match($0,/(\s*value:\s*)(.*)/,a){$0=a[1] "\"" gensub(/[\047"]/,"","g",a[2]) "\""} 1' file
- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"

otherwise Using any awk:

$ awk '/value:/{hd=$0; sub(/:.*/,": ",hd); gsub(/[^:]*: *[\047"]?|[\047"]$/,"");  $0=hd "\"" $0 "\""} 1' file
- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"
0

Using Raku (formerly known as Perl6)

A gentleman by the name of Larry Wall posted this neat quoting tip on the "perl6-users" mailing list ([email protected]). See Tip #8 at the NNTP link below:

~$ env re="\'" raku -ne 'say $0 if m/ <?after value\:\s >  \"? <{ %*ENV<re> }>?  (.+?) \"? <{ %*ENV<re> }>? $$ /;'  jinja.txt
「/bin/echo」
「Hello World」
「PATH」

The code above shows the correct match. So all that's needed is to run the code below:

~$ env re="\'" raku -pe 's/ <?after value\:\s >  \"? <{ %*ENV<re> }>?  (.+?) \"? <{ %*ENV<re> }>? $$ /"{$0}"/;'  jinja.txt
- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"

Want to make your escaping more readable? Don't want to play with ENV variables as above? Then just spell out your quoting characters (also from Larry Wall's Tip #8 at the NNTP link below):

~$ raku -ne 'say $0 if m/ <?after value\:\s >  \c[QUOTATION MARK]? \c[APOSTROPHE]?  (.+?)  \c[APOSTROPHE]? \c[QUOTATION MARK]? $$ /;'  jinja.txt
「/bin/echo」
「Hello World」
「PATH」

~$ raku -pe 's/ <?after value\:\s >  \c[QUOTATION MARK]? \c[APOSTROPHE]?  (.+?)  \c[APOSTROPHE]? \c[QUOTATION MARK]? $$ /"{$0}"/;'  jinja.txt
- param:
  name: Command
  type: String
  value: "/bin/echo"
- param:
  name: Args
  type: String
  value: "Hello World"
- param:
  name: Something
  type: EnvVar
  value: "PATH"

HTH.

https://raku.org/
https://www.nntp.perl.org/group/perl.perl6.users/2020/07/msg9004.html

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .