0

I have a directory with several folders (only interested in folders one level deep) and each one of these folders have files inside (values.yaml and example.yaml)

folder_1
 -values.yaml
 -example.yaml

folder_2
 -values.yaml
 -example.yaml
...

values.yaml is a valid yaml file with several values

example of values.yaml for folder 1

name: John
age: 66
country:
  name: Spain
  lenght: 32
...

example of values.yaml for folder 2

name: Simon
age: 90
country:
  name: France
  lenght: 33
...

I have to iterate through every single one of these directories and build an yaml file where I put the folder name concatenated with a certain string (eg:-batch) and put all values from values.yaml from this folder inside this field

Example of final output

folder_1-batch
  name: John
  age: 66
  country
    name:Spain
    lenght: 32
  ...

folder_2-batch
  name: Simon
  age: 90
  country:
    name: France
    lenght: 33
  ...
 

Any ideia how I could do this? I am able to copy all the content from every single values.yaml to a new file, but I am not understanding how I can put this content inside the main field (folder concatenated with -batch) with the correct indentation of a yaml file


small update: folders can have any name, and do not follow any pattern (folder_1 and folder_2 was just an example)

1
  • Your expected output does not seem to be valid YAML. There are missing colons, and each colon after a key needs a space after it.
    – Kusalananda
    Commented Aug 27, 2022 at 15:04

2 Answers 2

2

Using yq from https://kislyuk.github.io/yq/

shopt -s nullglob
for pathname in */values.yaml; do
    yq -y --arg name "${pathname%/*}-batch" '{ ($name): . }' "$pathname"
done >output.yaml

This first enables nullglob to avoid running the subsequent loop at all if there are no matching names.

The loop iterates over all values.yaml files in all the 1st-level subdirectories of the current directory.

For each file, yq is called to merge in the file's contents under the key given by the internal variable $name, which gets its value from --arg on the command line. The variable's value will be the directory's name followed by the string -batch.

All output is written to the file output.yaml.

Testing:

$ tree
.
├── folder1
│   └── values.yaml
└── folder2
    └── values.yaml

2 directories, 2 files
$ shopt -s nullglob
$ for pathname in */values.yaml; do yq -y --arg name "${pathname%/*}-batch" '{ ($name): . }' "$pathname"; done >output.yaml
$ cat output.yaml
folder1-batch:
  name: John
  age: 66
  country:
    name: Spain
    lenght: 32
folder2-batch:
  name: Simon
  age: 90
  country:
    name: France
    lenght: 33
4
  • Hey @Kusalananda. Can the same be achieved with mikefarah's tool ? Commented Aug 30, 2022 at 12:57
  • @AndreSilva The issue with Mike's yq is that I haven't found an obvious way to safely import the value of a shell variable into an expression, i.e., an equivalent to Andrey's --arg key value option. This way of importing a value ensures that the imported string is properly JSON-encoded.
    – Kusalananda
    Commented Aug 30, 2022 at 12:59
  • @AndreSilva You could possibly just inject the shell variable into yq eval "{ \"${pathname%/*}-batch\": . }" "$pathname" (or something similar, this is untested), but that would break as soon as any pathname contained a character that needs special JSON encoding, so I won't include that into my answer.
    – Kusalananda
    Commented Aug 30, 2022 at 13:02
  • Understood. Thankyou @Kusalananda Commented Aug 30, 2022 at 20:29
1

Loop over the folders, use sed to indent each YAML file.

#!/bin/bash
for d in folder_* ; do
    echo "$d"-batch
    sed 's/^/  /' "$d"/values.yaml
done > output.yaml
5
  • Hey @choroba. This goes well only for the first element of the iteration, but not for the remaining ones Commented Aug 26, 2022 at 21:38
  • @AndreSilva: Please provide more details. What's wrong with the second or third iteration?
    – choroba
    Commented Aug 26, 2022 at 21:58
  • second folder name is written right after the last value of the first folder code folder_1-batch: name: John age: 66 country name:Spain lenght: 32folder_2-batch Commented Aug 26, 2022 at 22:15
  • @AndreSilva: So add an echo after the sed line.
    – choroba
    Commented Aug 26, 2022 at 22:21
  • Hey @choroba yes, I already did it. Thank you. Commented Aug 27, 2022 at 14:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .