18

This is how my docker-compose.yml looks like.

nginx:
  container_name: 'nginx'
  image: 'nginx:1.11'
  restart: 'always'
  ports:
    - '80:80'
    - '443:443'
  volumes:
    - '/opt/nginx/conf.d:/etc/nginx/conf.d:ro'
  links:
    - 'anything'

Now I need to add some content via shell script (on an ubuntu server). I am not quite sure if it is possible at all:

  1. Add new element to nginx/links, if it is not existing
  2. Append newthing block if no newthing-block is existing

The new content should look like this:

nginx:
  container_name: 'nginx'
  image: 'nginx:1.11'
  restart: 'always'
  ports:
    - '80:80'
    - '443:443'
  volumes:
    - '/opt/nginx/conf.d:/etc/nginx/conf.d:ro'
    - '/etc/letsencrypt:/etc/letsencrypt'
  links:
    - 'anything'
    - 'newthing'

newthing:
  container_name: foo
  image: 'newthing:1.2.3'
  restart: always
  hostname: 'example.com'
5
  • 1
    Shell provides you very powerful scripting language. You can easily write a script using sed, awk and regular expressions to update your file.
    – fossil
    Jan 20, 2017 at 5:26
  • Could you please provide a little example? Jan 20, 2017 at 5:28
  • 2
    While this is technically possible using shell, you are must better off using a language that has an actual yaml library.
    – jordanm
    Jan 20, 2017 at 5:32
  • I would advise you to check out the ruamel.yaml library for Python.
    – phk
    Jan 20, 2017 at 5:57
  • I think it's worth noting that links is a legacy feature which will be removed from docker in the future: docs.docker.com/compose/compose-file/#links Dec 24, 2017 at 3:39

7 Answers 7

15

I wrote https://github.com/kislyuk/yq, a wrapper around https://stedolan.github.io/jq/, to address this use case.

2
  • 1
    Can it do update?
    – weynhamz
    Nov 12, 2018 at 5:21
  • 3
    Yes: yq -y '.newthing=...' input.yml > output.yml. (If you are asking about an update in place, like sed -i, yq can't do it by itself yet, but you can use sponge: yq -y .newthing=... file.yml | sponge file.yml.)
    – weaver
    Nov 14, 2018 at 1:00
9

I wrote yaml_cli (https://github.com/Gallore/yaml_cli) to do exactly what you need. It's based on python. This would be the syntax for your example:

yaml_cli \
  -f docker-compose.yml \                # read from and save to file
  --list-append \                        # flag to append to lists instead of replacing existing values
  -s nginx:links newthing \              # add a value of type string; here you need --list-append
  -s newthing:container_name foo \       # key 'newthing' is created automatically
  -s newthing:image 'newthing:1.2.3' \   #
  -s newthing:restart always \           #
  -s newthing:hostname 'example.com'     #

Feedback about yaml_cli is appreciated.

2
  • your tool looked great but looks like maintained is abandoned i took the liberty of publishing python 3 compatible version to pypi.org/project/yaml-cli/0.0.8 i decided this because I see there are several python3 upgrade PR in your repo but you haven't merged them
    – DevZer0
    Sep 14, 2022 at 16:20
  • but unfortunately it doesn't work at the current state, so i cat Chart.yaml |sed -e "s/version.*$/version: 0.0.1/"
    – DevZer0
    Sep 14, 2022 at 16:25
6

There are a number of yaml libraries for Perl, Python etc. if it's ok to do it not directly from a shell script, but use another language.

Another option is to install a command-line yaml processor, and call it from your shell script.

3

Since other tool authors are tossing their hats in, I'll do the same. My project, yamlpath is based on ruamel.yaml (making it fully YAML standards-compliant, which includes fully supporting JSON and other YAML-compatible formats) and Python 3. It can directly and trivially answer your question from the command-line via some robust commands it installs, applying any changes directly to the input file or writing changes out to a new file at your discretion. As a cross-platform solution, you can install yamlpath via the package manager of some OS platforms or directly to Python 3 (or virtual Python 3 environments) via pip.

Using yamlpath's various command-line tools, you gain several ways to accomplish your specific goals (yaml-set and/or yaml-merge) and even verify the result (yaml-diff). For an all-in-one command, you could:

$ cat docker-compose.yml 
nginx:
  container_name: 'nginx'
  image: 'nginx:1.11'
  restart: 'always'
  ports:
    - '80:80'
    - '443:443'
  volumes:
    - '/opt/nginx/conf.d:/etc/nginx/conf.d:ro'
  links:
    - 'anything'

$ cat changes.yml 
nginx:
  links:
    - 'newthing'

newthing:
  container_name: foo
  image: 'newthing:1.2.3'
  restart: always
  hostname: 'example.com'

$ cat docker-compose.yml | yaml-merge - changes.yml 
---
nginx:
  container_name: 'nginx'
  image: 'nginx:1.11'
  restart: 'always'
  ports:
    - '80:80'
    - '443:443'
  volumes:
    - '/opt/nginx/conf.d:/etc/nginx/conf.d:ro'
  links:
    - 'anything'
    - 'newthing'
newthing:
  container_name: foo
  image: 'newthing:1.2.3'
  restart: always
  hostname: 'example.com'

In this example, I created a changes.yml file that can be used and reused to apply uniform standard changes to any number of files now and later. I used the input stream mode to demonstrate the changes directly to the console; you could just apply the changes directly to the input file using the --overwrite (-w) option, if that is your goal. You can see that every change you need has been applied.

Note that the yaml-merge command also accepts the change input from stream, so you don't have to use a static file to indicate the desired changes. This input stream can be in any YAML-Compatible format (like JSON or EYAML) for a more dynamic approach, like this:

$ echo '{"nginx": {"links": ["newthing"]}, "newthing": {"container_name": "foo", "image": "newthing:1.2.3", "restart": "always", "hostname": "example.com"}}' | yaml-merge docker-compose.yml 
---
nginx:
  container_name: 'nginx'
  image: 'nginx:1.11'
  restart: 'always'
  ports:
    - '80:80'
    - '443:443'
  volumes:
    - '/opt/nginx/conf.d:/etc/nginx/conf.d:ro'
  links:
    - 'anything'
    - "newthing"
"newthing":
  "container_name": "foo"
  "image": "newthing:1.2.3"
  "restart": "always"
  "hostname": "example.com"

As you can see, this demonstration allows the changes to be fed in via STDIN, which can come from any YAML/JSON generator. Note that yaml-merge is deeply customizable and fully supports advanced YAML capabilities like Anchors/Aliases, YAML Merge Keys, and more.

For smaller changes, you could use yaml-set, instead. Say, if you needed to just tweak a single line -- or even several with the same replacement value using a single command -- or create all parent nodes necessary to inject a new node, you could do something like:

$ cat docker-compose.yml | yaml-set --change='/newthing/container_name' --value='foo'
---
nginx:
  container_name: 'nginx'
  image: 'nginx:1.11'
  restart: 'always'
  ports:
    - '80:80'
    - '443:443'
  volumes:
    - '/opt/nginx/conf.d:/etc/nginx/conf.d:ro'
  links:
    - 'anything'

newthing:
  container_name: foo

I again used stream mode to demonstrate the change without affecting the input file; yaml-set can directly edit the input file in-place (just pass the file-name as a positional parameter rather than using cat to push the file into its STDIN). In this example, you can see that yaml-set created an entirely new data structure and all necessary child nodes to write the new value at its novel location within the YAML data.

Be sure to check out the yamlpath Wiki for a deep dive into these tools. YAML Paths are extremely powerful and intuitive, as defined and implemented by this project.

Disclaimer: I am the author of the yamlpath project.

1

Since the reason you want to do this is to modify a docker-compose file, another alternative is to use a JSON file. Docker-compose now supports JSON files. Support for command-line manipulation of JSONs is already very good (ex: jq)

0

First, you shouldn't do this. Hard code it and be done with it. Second, this is easily doable given 3 files:

Your shell script: /tmp/yo.sh

#!/usr/bin/env bash
set -x

myYoFile='/tmp/yo.out'

cat >> "$myYoFile" << EOF
I've been cat'd!
EOF


# expand variables too
# readFile is a template to read from
readFile='/tmp/yo.tmpl'
# targetFile is where we will write to
targetFile="$myYoFile"


export positive_vibes='best'
export negative_vibes='worst'

# code block to execute
envsubst < "$readFile" >> "$targetFile"

cat "$targetFile"

A Template File (optional; for expanding long scripts with lots of variables)

cat /tmp/yo.tmpl 
It was the $positive_vibes of times. It was the $negative_vibes of times.

And a blank output file: /tmp/yo.out.


When you run this script, /tmp/yo.sh, the cat command will simply drop anything between the EOFs into $myYoFile; great for a shorter number of lines.

This gets a little weird when the content between the EOFs gets too long. In this case, variables have been assigned values in the main script but they are expanded after the template file has been read into memory. Then the result is appended to the end of the $targetFile which has been assigned the value of $myYoFile.

$ cat /tmp/yo.out
I've been cat'd!
It was the best of times. It was the worst of times.

So, there are two ways to do the thing you should never do :-P

6
  • I can't really see what this has to do with the given question.
    – Kusalananda
    Apr 30, 2021 at 17:36
  • Subject: "Is it possible to modify a yml file via shell script?". Answer: yes. It doesn't matter what the file type is. You can use Bash to modify any ascii text file. In my example I appended /tmp/yo.out with text. You could just as easily append to a /tmp/yo.yaml file.
    – todd_dsm
    May 1, 2021 at 18:53
  • But how do you ensure that the contents that you add is added in the correct places and that it isn't added if it's already there? Also, the YAML format requires encoding of strings that contains certain characters. Your answer seems to assume some sort of home-grown templating system. The user in the question has a YAML file, not a template, and they need to ensure the integrity of the document structure.
    – Kusalananda
    May 1, 2021 at 19:36
  • There's only one way through any of this: setup an experiment and test; there's no other answer. As far as the home-grown bit, envsubst is GNU software, same as bash, sed and awk. Whatever you do with those tools is by nature DIY. There is no magic YAML templating system; you will note, I said: "First, you shouldn't do this". Just create your compose file and be done with it. The idea that this is being asked at all means an upstream decision has been made erroneously. Fix that and the real issue will go away.
    – todd_dsm
    May 3, 2021 at 19:00
  • I think the disconnect here is between the question's title and the question's body. The body has two requirements: "Add new element to nginx/links, if it is not existing" and "Append newthing block if no newthing-block is existing" (my emphasis on the if). The current answer is a simple variable/template subsitution, and doesn't take into account any existing data.
    – Jeff Schaller
    May 3, 2021 at 20:16
0

Using yq and jo:

# Create JSON variant of section that we might add
newsection=$( jo container_name=foo image=newthing:1.2.3 restart=always hostname=example.com )

# Modify the file (result to standard output)
yq -y \
    --argjson newsection "$newsection" \
    --arg newentry 'newthing' \
    '.newthing    = (.newthing // $newsection)   |
     .nginx.links = ([.nginx.links[], $newentry] | unique)' docker-compose.yml
  • jo (link) is a utility for creating JSON on the command line, and here we use it to create the newthing section that we will add if it doesn't already exist.

  • yq (link) is a YAML parser wrapped around jq, the JSON parser.

  • jq (link) is a well known JSON parser.

The jq expression that we use with yq here adds the .newthing section if it doesn't already exist. It does this by simply setting the section's contents to the old contents if it exists, or to the new contents in $newsection, if it doesn't exist.

The addition of the newthing string to the nginx.links array is done through rewriting the array by adding the element and then removing duplicates using the unique filter.

The output, given the data in the question:

nginx:
  container_name: nginx
  image: nginx:1.11
  restart: always
  ports:
    - 80:80
    - 443:443
  volumes:
    - /opt/nginx/conf.d:/etc/nginx/conf.d:ro
  links:
    - anything
    - newthing
newthing:
  container_name: foo
  image: newthing:1.2.3
  restart: always
  hostname: example.com

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .