bash iterate through several directories to build one single yaml file

Question

I have a directory with several folders (only interested in folders one level deep) and each one of these folders have files inside (values.yaml and example.yaml)

folder_1
 -values.yaml
 -example.yaml

folder_2
 -values.yaml
 -example.yaml
...

values.yaml is a valid yaml file with several values

example of values.yaml for folder 1

name: John
age: 66
country:
  name: Spain
  lenght: 32
...

example of values.yaml for folder 2

name: Simon
age: 90
country:
  name: France
  lenght: 33
...

I have to iterate through every single one of these directories and build an yaml file where I put the folder name concatenated with a certain string (eg:-batch) and put all values from values.yaml from this folder inside this field

Example of final output

folder_1-batch
  name: John
  age: 66
  country
    name:Spain
    lenght: 32
  ...

folder_2-batch
  name: Simon
  age: 90
  country:
    name: France
    lenght: 33
  ...
 

Any ideia how I could do this? I am able to copy all the content from every single values.yaml to a new file, but I am not understanding how I can put this content inside the main field (folder concatenated with -batch) with the correct indentation of a yaml file

---

small update: folders can have any name, and do not follow any pattern (folder_1 and folder_2 was just an example)

Answer 1

Using yq from https://kislyuk.github.io/yq/

shopt -s nullglob
for pathname in */values.yaml; do
    yq -y --arg name "${pathname%/*}-batch" '{ ($name): . }' "$pathname"
done >output.yaml

This first enables nullglob to avoid running the subsequent loop at all if there are no matching names.

The loop iterates over all values.yaml files in all the 1st-level subdirectories of the current directory.

For each file, yq is called to merge in the file's contents under the key given by the internal variable $name, which gets its value from --arg on the command line. The variable's value will be the directory's name followed by the string -batch.

All output is written to the file output.yaml.

Testing:

$ tree
.
├── folder1
│   └── values.yaml
└── folder2
    └── values.yaml

2 directories, 2 files

$ shopt -s nullglob
$ for pathname in */values.yaml; do yq -y --arg name "${pathname%/*}-batch" '{ ($name): . }' "$pathname"; done >output.yaml

$ cat output.yaml
folder1-batch:
  name: John
  age: 66
  country:
    name: Spain
    lenght: 32
folder2-batch:
  name: Simon
  age: 90
  country:
    name: France
    lenght: 33

Answer 2

Loop over the folders, use sed to indent each YAML file.

#!/bin/bash
for d in folder_* ; do
    echo "$d"-batch
    sed 's/^/  /' "$d"/values.yaml
done > output.yaml