Struggling using sed command with variables

Question

So I have the following variables defined in a bash script:

new_commit="back:h3912kk"
old_commit="back:1.0.1"

file = docker-compose.yml    

Then I have this yml $file:

version: '3'


services:

  mongo:
    container_name: mongo
    image: mongo:3.4
    volumes:
      - /home/mongo_files:/data/db
    networks:
      my_network:


  back:
    container_name: back
    image: index.docker.io/gg/back:1.0.1
    networks:
      my_network

networks:
    my_network:

I would like to replace the line

image: index.docker.io/gg/back:1.0.1 

for

image: index.docker.io/gg/back:h3912kk

I am using the following sed command in my bash script:

echo sed -i "s/$old_commit/$new_commit/" $file

However the following error appears:

sed: -e expression #1, char 12: unterminated `s' command

Of course the docker-compose file does not update.

What am I doing wrong?

if I do an echo command this is displayed:

sed -i s/back:1.0.1
/back:h3912kk/ docker-compose.yml

Why the echo is not displaying the command on the same line?

EDIT:

As @Kusalananda♦ says, the old_commit variable is not a static file, and that is the main reason I guess why I am being unable to reproduce the answers you provide with the $old_commit involved ( I though it was not going to suppose a problem)

In order to obtain the old_commit parameter I use the following while that looks in the yml file for the line that satisfies the condition I am interested in, here is my full code:

output='back:h3912kk'

# here I split the name of the repo and the commit by ':'
readarray -d ':' -t array_new_commit <<< "$output"

repo="${array_new_commit[0]}"

file=docker-compose.yml

# here I loop over the lines of the yml file:
while read -r line; do
  # if a line contains the repo name str and image str I generate a line with the new commit
  if [[ $line == *$repo* && $line == *"image"* ]]; then

    # old_line=$line

    readarray -d '/' -t array_old_commit <<< "$line"
    old_commit=${array_old_commit[-1]}


    # new_line="${array_old_commit[0]}:${array_old_commit[1]}:$new_commit"
  fi
done<"$file"

Somehow ( dont know why, I am new into bash) generating this variable this way literally does not allows me to use it as if it was a static variable. (I would really want to understand this)

Could you please give me a workaround to replace the old_commit variable by the new_commit into the yml file?

Thank you again Kusalananda♦ for pointing out this issue

PD I am using a ubuntu 0.18.4 distro

EDIT: Using a second piece of code:

yq -Y --arg new "$new_commit" --arg reposit "$repo" \
'".services." + $reposit + ".image |=index.docker.io/gg/" + $new' docker-compose.yml 

my output was:

.services.back.image |=index.docker.io/gg/back:h3912kk 
...

Answer 1

You should not use sed to work with YAML files, or files written in any structured document format (YAML, JSON, XML, etc.) Instead, use a parser written for that document format.

For YAML, there are two good tools written for the shell, both confusingly called yq. The yq from https://kislyuk.github.io/yq/ is an easy to use jq wrapper (and jq is a JSON parser).

For part of this answer, I'm going to assume that you have defined the two shell variables old_commit and new_commit exactly as you show in your code in the question. The output that you show later tells me you're not actually assigning these as static strings but through some other process that you don't say anything about (in which $old_commit seems to get a trailing newline character), so I can't say much about it and will ignore that.

To change the value for the image key under back, under services, using this yq implementation, you would use

yq -Y '.services.back.image |= "index.docker.io/gg/back:h3912kk"' file.yml

or, using your $new_commit value:

new_commit='back:h3912kk'

yq -Y --arg new "$new_commit" \
    '.services.back.image |= 
        "index.docker.io/gg/" + $new' file.yml

Note that we import the shell variable's value into the expression using --arg, creating a yq variable $new, rather than injecting it directly into the expression string. This way we ensure that the value is properly encoded by yq, and it also ensures that we don't have any code injection vulnerabilities in our code.

Or, without having to mention the index.docker.io/gg/ string:

new_commit='back:h3912kk'

yq -Y --arg new "$new_commit" \
    '.services.back.image |=
        sub("[^/]*$"; "") + $new' file.yml

Or, using a separate variable for the repository:

new_commit='back:h3912kk'

yq -Y --arg new "$new_commit" --arg repo 'back' \
    '.services[$repo].image |=
        sub("[^/]*$"; "") + $new' file.yml

We could also use the $old_commit string to find the right image to replace instead of going by a static path:

old_commit='back:1.0.1'
new_commit='back:h3912kk'

yq -Y --arg old "$old_commit" --arg new "$new_commit" \
    '(.services[].image | select(endswith($old))) |=
        (rtrimstr($old) + $new)' file.yml

Here we select all .services.*.image paths that ends with whatever string is in $old_commit, and then we trim that off from the end and replace it with $new_commit.

Each of the commands above would, given the example YAML document in the question, produce

version: '3'
services:
  mongo:
    container_name: mongo
    image: mongo:3.4
    volumes:
      - /home/mongo_files:/data/db
    networks:
      my_network: null
  back:
    container_name: back
    image: index.docker.io/gg/back:h3912kk
    networks: my_network
networks:
  my_network: null

and you would easily be able to redirect the output to a new file (or edit the YAML document in-place using --in-place).

The yq tool that you get from snap on e.g. Ubuntu is different, and I tend to use it to convert YAML to JSON (and then back again):

yq -j e file.yml |
jq '.services.back.image |= "index.docker.io/gg/back:h3912kk"' |
yq -P e

Given the example YAML file, this produces the same output as above. The jq expression above would be exchangeable for any of the expressions at the top of this answer, with the same semantics.

Answer 2

I will not judge you about using sed on yaml file.. Everyone use what he wants to get things done.

I had the same issue some time ago, and it seems that if you 'brace' your vars in the sed regex, the whole thing starts to works:

$ cat $file
version: '3'


services:

  mongo:
    container_name: mongo
    image: mongo:3.4
    volumes:
      - /home/mongo_files:/data/db
    networks:
      my_network:


  back:
    container_name: back
    image: index.docker.io/gg/back:1.0.1
    networks:
      my_network

networks:
    my_network:

then :

$ sed s/${old_commit}/${new_commit}/ $file

Gives this result:

version: '3'


services:

  mongo:
    container_name: mongo
    image: mongo:3.4
    volumes:
      - /home/mongo_files:/data/db
    networks:
      my_network:


  back:
    container_name: back
    image: index.docker.io/gg/back:h3912kk
    networks:
      my_network

networks:
    my_network:

Hope this helps..

Tested on MacOS 10.13 (bash 5.1.4.1, gnu sed 4.8) and on Ubuntu 16.04 (bash 4.3.48, gnu sed 4.2.2)

Answer 3

yq -i -Y --arg new "$VERSION" '.tool.image.tag |= $new' values.yaml