1

I am trying to create yaml files from a template using my variables. My yaml template look like this

number: {{NUMBER}}
  name: {{NAME}}
  region: {{REGION}}
  storenum: {{STORENUM}}
  clients: {{CLIENTS}}
  tags: {{TAGS}}


storename: {{STORENAME}}
employee: {{EMPLOYEE}}
products: {{PRODUCTS}}

But my variables are in a CSV file; the structure is the variables.

Number - Name - Region - Storenum  
StoreX - StoreX - New York - 30  

I now have a little script to create from a template with the variable parameters and the template like this: script.sh template.yml -f variables.txt. And my result looks like this

number: 37579922
  name: Store1
  region: New York
  storenum: 32
  clients: 100
  tags: stores


storename: Store newyork
employee: 10
products: 200

But I can only do one at a time. Is there any way to read the CSV parameters and send to the program and generate, for example, Template1,Template2,etc from the CSV parameters?

#!/bin/bash
readonly PROGNAME=$(basename $0)

config_file="<none>"
print_only="false"
silent="false"

usage="${PROGNAME} [-h] [-d] [-f] [-s] -- 

where:
    -h, --help
        Show this help text
    -p, --print
        Don't do anything, just print the result of the variable expansion(s)
    -f, --file
        Specify a file to read variables from
    -s, --silent
        Don't print warning messages (for example if no variables are found)

examples:
    VAR1=Something VAR2=1.2.3 ${PROGNAME} test.txt 
    ${PROGNAME} test.txt -f my-variables.txt
    ${PROGNAME} test.txt -f my-variables.txt > new-test.txt"

if [ $# -eq 0 ]; then
  echo "$usage"
  exit 1    
fi

if [[ ! -f "${1}" ]]; then
    echo "You need to specify a template file" >&2
    echo "$usage"
    exit 1
fi

template="${1}"

if [ "$#" -ne 0 ]; then
    while [ "$#" -gt 0 ]
    do
        case "$1" in
        -h|--help)
            echo "$usage"
            exit 0
            ;;        
        -p|--print)
            print_only="true"
            ;;
        -f|--file)
            config_file="$2"
            ;;
        -s|--silent)
            silent="true"
            ;;
        --)
            break
            ;;
        -*)
            echo "Invalid option '$1'. Use --help to see the valid options" >&2
            exit 1
            ;;
        # an option argument, continue
        *)  ;;
        esac
        shift
    done
fi

vars=$(grep -oE '\{\{[A-Za-z0-9_]+\}\}' "${template}" | sort | uniq | sed -e 's/^{{//' -e 's/}}$//')

if [[ -z "$vars" ]]; then
    if [ "$silent" == "false" ]; then
        echo "Warning: No variable was found in ${template}, syntax is {{VAR}}" >&2
    fi
fi

# Load variables from file if needed
if [ "${config_file}" != "<none>" ]; then
    if [[ ! -f "${config_file}" ]]; then
      echo "The file ${config_file} does not exists" >&2
      echo "$usage"      
      exit 1
    fi

    source "${config_file}"
fi    

var_value() {
    eval echo \$$1
}

replaces=""

# Reads default values defined as {{VAR=value}} and delete those lines
# There are evaluated, so you can do {{PATH=$HOME}} or {{PATH=`pwd`}}
# You can even reference variables defined in the template before
defaults=$(grep -oE '^\{\{[A-Za-z0-9_]+=.+\}\}' "${template}" | sed -e 's/^{{//' -e 's/}}$//')

for default in $defaults; do
    var=$(echo "$default" | grep -oE "^[A-Za-z0-9_]+")
    current=`var_value $var`

    # Replace only if var is not set
    if [[ -z "$current" ]]; then
        eval $default
    fi

    # remove define line
    replaces="-e '/^{{$var=/d' $replaces"
    vars="$vars
$current"
done

vars=$(echo $vars | sort | uniq)

if [[ "$print_only" == "true" ]]; then
    for var in $vars; do
        value=`var_value $var`
        echo "$var = $value"
    done
    exit 0
fi

# Replace all {{VAR}} by $VAR value
for var in $vars; do
    value=$(var_value $var | sed -e "s;\&;\\\&;g" -e "s;\ ;\\\ ;g") # '&' and <space> is escaped 
    if [[ -z "$value" ]]; then
        if [ $silent == "false" ]; then
            echo "Warning: $var is not defined and no default is set, replacing by empty" >&2
        fi
    fi

    # Escape slashes
    value=$(echo "$value" | sed 's/\//\\\//g');
    replaces="-e 's/{{$var}}/${value}/g' $replaces"    
done

escaped_template_path=$(echo $template | sed 's/ /\\ /g')
eval sed $replaces "$escaped_template_path"
2
  • Yes. That's what loops like while read are for. However, shell is a terrible language for doing text processing. What you're doing can and should be done in another language - any other language: awk, perl, or python, for example. perl is particularly suited to tasks like this because it has library modules for reading and writing CSV and YAML (e.g. Text::CSV and YAML). Option processing is included with Getopt::Std and Getopt::Long modules. Your ~140 line script would be about 10-20 lines in perl.
    – cas
    Oct 3, 2021 at 1:58
  • Your templating needs seem pretty simple and should be doable with just a heredoc, but perl also has an excellent module for templating called Text::Template.
    – cas
    Oct 3, 2021 at 1:59

3 Answers 3

1

Here's a very simple example of doing this in perl using the Text::CSV module to parse the CSV.

No command-line option processing is done (although this could easily be done using Getopt::Std or Getopt::Long which are adequate but basic (but are included with perl), or fancier modules like Getopt::Lucid, which need to be installed but can do pretty much anything you might want to do with options.

This just embeds the template in the script as a heredoc. For more complex templating needs, use the Text::Template library module.

It also just prints the output to stdout. You can redirect that in the shell as usual or, if you need the output from each csv input line to be stored in a separate file, it's easy enough to get perl to open a file for write and print the output to that file.

Compared to your ~ 140 lines of bash script (about a third of which is comments, blank lines and the usage message), this perl script has 35 lines total, 12 of which are the template, 6 are comments, and 8 are blank lines to improve readability. i.e. 9 lines of actual code vs about 90 lines for bash.

Unlike a bash script, this won't have any issues with quoting or white space that you'd need to deal with, and will run much faster because it doesn't have to repeatedly fork external programs like sed (features equivalent to grep, sed, tr, and more are built-in to perl). Note also that the Text::CSV module easily handles fields that contain embedded commas (the TAGS field) - which is one of the benefits of using an actual CSV parser instead of faking it with regular expressions.

#!/usr/bin/perl

use strict;
use Text::CSV;

# open the CSV file for read
my $file = 'data.csv';
open(my $fh, "<", $file) or die "Couldn't open $file: $!\n";

# initialise a csv object
my $csv = Text::CSV->new();

# read the header line
my @headers = $csv->getline($fh);
$csv->column_names(@headers);

# iterate over each line of the CSV file, reading
# each line into a hash (associative array) reference.
while (my $row = $csv->getline_hr($fh)) {
print <<__EOF__;
number: $row->{NUMBER}
  name: $row->{NAME}
  region: $row->{REGION}
  storenum: $row->{STORENUM}
  clients: $row->{CLIENTS}
  tags: $row->{TAGS}


storename: $row->{STORENAME}
employee: $row->{EMPLOYEE}
products: $row->{PRODUCTS}

__EOF__
}
close($fh);

If data.csv contains the following:

NUMBER,NAME,REGION,STORENUM,CLIENTS,TAGS,STORENAME,EMPLOYEE,PRODUCTS
37579922,Store1,New York,32,100,stores,Store newyork,10,200
2,Store2,Somewhere,2,100,"tag1,tag2,tag3",Somewhere Store,5,10
3,Store3,Elsewhere,3,100,"tag1,tag3",Elsewhere Store,3,100

Then running it produces the following output:

$ ./template-example.pl 
number: 37579922
  name: Store1
  region: New York
  storenum: 32
  clients: 100
  tags: stores


storename: Store newyork
employee: 10
products: 200

number: 2
  name: Store2
  region: Somewhere
  storenum: 2
  clients: 100
  tags: tag1,tag2,tag3


storename: Somewhere Store
employee: 5
products: 10

number: 3
  name: Store3
  region: Elsewhere
  storenum: 3
  clients: 100
  tags: tag1,tag3


storename: Elsewhere Store
employee: 3
products: 100

BTW, if you prefer python, this would be about as easy to write in python as it is in perl.

0

This gawk solution was tested on GNU Awk v5.1.0.

It consists in a Bash script which accepts two input files: yaml_template (given in OP):

$ cat yaml_template
number: {{NUMBER}}
  name: {{NAME}}
  region: {{REGION}}
  storenum: {{STORENUM}}
  clients: {{CLIENTS}}
  tags: {{TAGS}}


storename: {{STORENAME}}
employee: {{EMPLOYEE}}
products: {{PRODUCTS}}

and data.csv (proposed by @cas in his answer):

$ cat data.csv
NUMBER,NAME,REGION,STORENUM,CLIENTS,TAGS,STORENAME,EMPLOYEE,PRODUCTS
37579922,Store1,New York,32,100,stores,Store newyork,10,200
2,Store2,Somewhere,2,100,"tag1,tag2,tag3",Somewhere Store,5,10
3,Store3,Elsewhere,3,100,"tag1,tag3",Elsewhere Store,3,100

The Bash script, yamlit.sh is made executable (with cmd $ chmod ug+x yamlit.sh):

$ cat yamlit.sh
#!/usr/bin/env bash
gawk -F"[,:]" '
    FNR==NR {
        match($0,/[^[:blank:]]+/); i++;
        if (RSTART-1 < 0) {$1="";null++; teenar[i] = ""} else {$1 = substr($1,RSTART); teenar[i] = $1};
        teenof[$1] = RSTART - 1;
        next;
    }
    FNR==1 {nteen=i; ncol=split(tolower($0), colhead, ",");next;}
    {
    for (i=1; i<=nteen; i++) {
        offset=teenof[teenar[i]];
        if (offset >= 0) {
            patsplit($0,datafield,"([^,]*)|(\"[^\"]*\")")
            for (j=1; j<=ncol; j++) {
                if (tolower(teenar[i]) == colhead[j]) {
                    printf "%*s: %s\n", length(teenar[i]) + offset, teenar[i], datafield[j];
                    }
                }
            }
        else {print ""}
        }
    printf "\n%s\n", "=========================="
    }' "$1" "$2"

The script when run from terminal yields:

$ yamlit.sh yaml_template data.csv
number: 37579922
  name: Store1
  region: New York
  storenum: 32
  clients: 100
  tags: stores


storename: Store newyork
employee: 10
products: 200

==========================
number: 2
  name: Store2
  region: Somewhere
  storenum: 2
  clients: 100
  tags: "tag1,tag2,tag3"


storename: Somewhere Store
employee: 5
products: 10

==========================
number: 3
  name: Store3
  region: Elsewhere
  storenum: 3
  clients: 100
  tags: "tag1,tag3"


storename: Elsewhere Store
employee: 3
products: 100

==========================

A word of explanation:

  • the script prints its output to terminal, but output can be easily redirected toward a file, e.g. from CLI: $ yamlit.sh yaml_template data.csv >| yaml_out,
  • it respects the format and structure provided by the yaml template scrupulously. Any formatting error in the template will show in the output. That includes any faulty blank spaces (i.e. faulty indentations) and blank lines,
  • it respects the order in which the template entries are provided, irrespective of column ordering in the data file data.csv,
  • templates can be as complex and with as many nested levels as required,
  • matching template entry keys and data column headers is case-insensitive.

What could be added ?
Features that would be trivial to add to the script include:

  • redirecting the "yaml" output of each data file record to either one file on disk or to as many distinct output files as there are data records (rows), not counting the file header.
  • running checks on whether there is a data mismatch between data.csv and yaml template entry keys in the template file

The code:

  • the first block FNR==NR {...} computes:
    • the number of (blank and non-blank) lines in the template file,
    • the indentation of each line, saved in the array: teenof, short for "TEmplate ENtry OFfset". Negative values indicate a blank line in the template file.
    • Template entry key (the first field of the record) in another array: teenar, short for "TEmplate ENtry ARray".
  • the second block FNR==1 {...} places each column header of the data.csv file in a third array colhead.
  • the third and last block {...} does several things:
    • loops over template entry keys (in lower case),
    • for positive or null indentations:
      • it splits each data.csv file record according to a field regex capable of discerning whether any field in the record is made of a quoted string containing other field delimiters (here ,) or not. Components that results from the split are placed in a fourth array: datafield.
      • it performs a nested-loop over data file column headers to find a match with the template entry; If found it prints the corresponding line, respecting the indentation specified by the yaml template.
    • for negative indentations, it prints a blank line, thus respecting the template's original sequence where it includes one or more blank lines.

Critique:

  • Two nested loops are used, which even for awk is far from ideal from a compute standpoint. Complexity is O(n^2). This however made generalizing the solution in terms of data.csv files having arbitrary column ordering relative to the order of entries of the yaml template file possible. I did not spend time to look for another solution. Not sure there is a better one with awk...
  • A total of 4 GNU Awk arrays are used, which for files "very large" or "massive, say with many columns and many many records (or rows) might become memory intensive. That would be compounded by the corresponding arrays (teenof and teenar) built from the yaml templates, presumably with as many entries as there data columns. Of those 4 arrays, only one (datafield) is emptied and rebuilt for each data.csv file record. The others are static. That said, I am not certain that squashing a Gawk array with a new one reallocates memory as efficiently as C would for a deleted/redeclared array variable. I leave that for others more knowledgeable than I am to comment.

HTH.

0

Here's a weird way to do it using GoCSV, and the fact that it leverages Go's template engine.

I made a mock CSV based on your template and the sample data you provided:

store-x.csv

Number,Name,Region,Storenum,Clients,Tags,Storename,Employee,Products
StoreX,StoreX,New York,30,Foo,store-x,Store X,Alice,X stuff

Using templates with GoCSV is almost identical to your original YAML template:

  • the field names are title-cased to match the CSV
  • the field names have a leading . (Go/GoCSV's notation for field names)

template.yaml

number: {{.Number}}
  name: {{.Name}}
  region: {{.Region}}
  storenum: {{.Storenum}}
  clients: {{.Clients}}
  tags: {{.Tags}}


storename: {{.Storename}}
employee: {{.Employee}}
products: {{.Products}}

And, finally, a short shell script with a GoCSV pipeline:

make_yaml.sh

#!/bin/sh

csv_data=$1

yaml_tmpl=$(cat template.yaml)

cat "$csv_data"                                   \
| gocsv add --name 'YAML' --template "$yaml_tmpl" \  # 1.
| gocsv select -c 'YAML'                          \  # 2.
| gocsv behead                                       # 3.
  1. add a column named YAML with your template populated with the CSV's data
  2. select just the new YAML column
  3. take away the header

...and you're left with just your YAML:

% sh make_yaml.sh data.csv

"number: StoreX
  name: StoreX
  region: New York
  storenum: 30
  clients: Foo
  tags: store-x


storename: Store X
employee: Alice
products: X stuff"

Almost 😣 Downside of doing this inside the CSV is the quote chars for embedded newlines.

Piping that through these two sed commands gets rid of just the first char from the first line, and just the last char from the last line:

... | sed '1 s/^.//' | sed '$ s/.$//'

And, now, finally:

number: StoreX
  name: StoreX
  region: New York
  storenum: 30
  clients: Foo
  tags: store-x


storename: Store X
employee: Alice
products: X stuff

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.