Replace environment variables in a file with their actual values?

Question

Is there an easy way to substitute/evaluate environment variables in a file? Like let's say I have a file config.xml that contains:

<property>
    <name>instanceId</name>
    <value>$INSTANCE_ID</value>
</property>
<property>
    <name>rootPath</name>
    <value>/services/$SERVICE_NAME</value>
</property>

...etc. I want to replace $INSTANCE_ID in the file with the value of the INSTANCE_ID environment variable, $SERVICE_NAME with the value of the SERVICE_NAME env var. I won't know a priori which environment vars are needed (or rather, I don't want to have to update the script if someone adds a new environment variable to the config file). Thanks!

Answer 1

You could use envsubst (part of gnu gettext):

envsubst < infile

will replace the environment variables in your file with their corresponding value. The variable names must consist solely of alphanumeric or underscore ASCII characters, not start with a digit and be nonempty; otherwise such a variable reference is ignored.

---

To replace only certain environment variables, see this question.

Answer 2

This is not very nice but it works

( echo "cat <<EOF" ; cat config.xml ; echo EOF ) | sh

If it was in a shell script it would look like:

#! /bin/sh
cat <<EOF
<property>
    <name>instanceId</name>
    <value>$INSTANCE_ID</value>
</property>
EOF

Edit, second proposal:

eval "echo \"$(cat config.xml)\""

Edit, not strictly related to question, but in case of variables read from file:

(. .env && eval "echo \"$(cat config.xml)\"")

Answer 3

If you happen to have Perl (but not gettext and envsubst) you can do the simple replacement with a short script:

$ export INSTANCE_ID=foo; export SERVICE_NAME=bar;
$ perl -pe 's/\$([_A-Z]+)/$ENV{$1}/g'  < config.xml
<property>
    <name>instanceId</name>
    <value>foo</value>
</property>
<property>
    <name>rootPath</name>
    <value>/services/bar</value>
</property>

I assumed the variable names will only have uppercase letters and underscores, but the first pattern should be easy to alter as needed. $ENV{...} references the environment Perl sees.

If you want to support the ${...} syntax or throw an error on unset variables, you'll need some more work. A close equivalent of gettext's envsubst would be:

perl -pe 's/\$(\{)?([a-zA-Z_]\w*)(?(1)\})/$ENV{$2}/g'

Though I feel that feeding variables like that via the process environment seems a bit iffy in general: you can't use arbitrary variables in the files (since they may have special meanings), and some of the values could possibly have at least semi-sensitive data in them.

Answer 4

May I suggest my own script for this?

https://github.com/rydnr/set-square/blob/master/.templates/common-files/process-file.sh

#!/bin/bash /usr/local/bin/dry-wit
# Copyright 2016-today Automated Computing Machinery S.L.
# Distributed under the terms of the GNU General Public License v3

function usage() {
cat <<EOF
$SCRIPT_NAME -o|--output output input
$SCRIPT_NAME [-h|--help]
(c) 2016-today Automated Computing Machinery S.L.
    Distributed under the terms of the GNU General Public License v3

Processes a file, replacing any placeholders with the contents of the
environment variables, and stores the result in the specified output file.

Where:
    * input: the input file.
    * output: the output file.
Common flags:
    * -h | --help: Display this message.
    * -v: Increase the verbosity.
    * -vv: Increase the verbosity further.
    * -q | --quiet: Be silent.
EOF
}

# Requirements
function checkRequirements() {
  checkReq envsubst ENVSUBST_NOT_INSTALLED;
}

# Error messages
function defineErrors() {
  export INVALID_OPTION="Unrecognized option";
  export ENVSUBST_NOT_INSTALLED="envsubst is not installed";
  export NO_INPUT_FILE_SPECIFIED="The input file is mandatory";
  export NO_OUTPUT_FILE_SPECIFIED="The output file is mandatory";

  ERROR_MESSAGES=(\
    INVALID_OPTION \
    ENVSUBST_NOT_INSTALLED \
    NO_INPUT_FILE_SPECIFIED \
    NO_OUTPUT_FILE_SPECIFIED \
  );

  export ERROR_MESSAGES;
}

## Parses the input
## dry-wit hook
function parseInput() {

  local _flags=$(extractFlags $@);
  local _flagCount;
  local _currentCount;

  # Flags
  for _flag in ${_flags}; do
    _flagCount=$((_flagCount+1));
    case ${_flag} in
      -h | --help | -v | -vv | -q)
         shift;
         ;;
      -o | --output)
         shift;
         OUTPUT_FILE="${1}";
         shift;
         ;;
    esac
  done

  # Parameters
  if [[ -z ${INPUT_FILE} ]]; then
    INPUT_FILE="$1";
    shift;
  fi
}

## Checking input
## dry-wit hook
function checkInput() {

  local _flags=$(extractFlags $@);
  local _flagCount;
  local _currentCount;
  logDebug -n "Checking input";

  # Flags
  for _flag in ${_flags}; do
    _flagCount=$((_flagCount+1));
    case ${_flag} in
      -h | --help | -v | -vv | -q | --quiet)
         ;;
      -o | --output)
         ;;
      *) logDebugResult FAILURE "fail";
         exitWithErrorCode INVALID_OPTION ${_flag};
         ;;
    esac
  done

  if [[ -z ${INPUT_FILE} ]]; then
    logDebugResult FAILURE "fail";
    exitWithErrorCode NO_INPUT_FILE_SPECIFIED;
  fi

  if [[ -z ${OUTPUT_FILE} ]]; then
      logDebugResult FAILURE "fail";
      exitWithErrorCode NO_OUTPUT_FILE_SPECIFIED;
  fi
}

## Replaces any placeholders in given file.
## -> 1: The file to process.
## -> 2: The output file.
## <- 0 if the file is processed, 1 otherwise.
## <- RESULT: the path of the processed file.
function replace_placeholders() {
  local _file="${1}";
  local _output="${2}";
  local _rescode;
  local _env="$(IFS=" \t" env | awk -F'=' '{printf("%s=\"%s\" ", $1, $2);}')";
  local _envsubstDecl=$(echo -n "'"; IFS=" \t" env | cut -d'=' -f 1 | awk '{printf("${%s} ", $0);}'; echo -n "'";);

  echo "${_env} envsubst ${_envsubstDecl} < ${_file} > ${_output}" | sh;
  _rescode=$?;
  export RESULT="${_output}";
  return ${_rescode};
}

## Main logic
## dry-wit hook
function main() {
  replace_placeholders "${INPUT_FILE}" "${OUTPUT_FILE}";
}
# vim: syntax=sh ts=2 sw=2 sts=4 sr noet

Answer 5

Use cmake's configure_file function.

It copies an <input> file to an <output> file and substitutes variable values referenced as @VAR@ or ${VAR} in the input file content. Each variable reference will be replaced with the current value of the variable, or the empty string if the variable is not defined.

Answer 6

Similarly to the Perl answer, environment variable substitution can be delegated to the PHP CLI. Dependency on PHP may or may not be acceptable depending on the tech stack in use.

php -r 'echo preg_replace_callback("/\\$([a-z0-9_]+)/i", function ($matches) { return getenv($matches[1]); }, fread(STDIN, 8192));' < input.file > output.file

You can go further and put it in a reusable script, for example, envsubst:

#!/usr/bin/env php
<?php

echo preg_replace_callback(
    '/\$(?<name>[a-z0-9_]+)/i',
    function ($matches) {
        return getenv($matches['name']);
    },
    file_get_contents('php://stdin')
);

The usage would be:

envsubst < input.file > output.file