Replace environment variables in a file with their actual values?

Question

Is there an easy way to substitute/evaluate environment variables in a file? Like let's say I have a file config.xml that contains:

<property>
    <name>instanceId</name>
    <value>$INSTANCE_ID</value>
</property>
<property>
    <name>rootPath</name>
    <value>/services/$SERVICE_NAME</value>
</property>

...etc. I want to replace $INSTANCE_ID in the file with the value of the INSTANCE_ID environment variable, $SERVICE_NAME with the value of the SERVICE_NAME env var. I won't know a priori which environment vars are needed (or rather, I don't want to have to update the script if someone adds a new environment variable to the config file). Thanks!

When will you do something with file (cat, echo, source,…) the variable will subtitute by its value — Costas, Jul 9 '16 at 14:30
Is the contents of this xml file up to you? If so, parameterized xslt offers another way to inject values and (unlike envsubst and its ilk) guarantees well formed xml as a result. — kojiro, Jul 9 '16 at 17:28

don_crissti · Accepted Answer · 2017-07-28 10:32:56Z

58

You could use envsubst (part of gnu gettext):

envsubst < infile

will replace the environment variables in your file with their corresponding value. The variable names must consist solely of alphanumeric or underscore ASCII characters, not start with a digit and be nonempty; otherwise such a variable reference is ignored.

To replace only certain environment variables, see this question.

edited Jul 28 '17 at 10:32

answered Jul 9 '16 at 14:38

don_crissti

54.5k18 gold badges152 silver badges178 bronze badges

1

...except it's not installed by default in my docker image :'-( – Robert Fraser Jul 9 '16 at 14:57
4

That's good. Docker images should be lightweight and tailor made. Of course you could always add envsubst to it, though. – kojiro Jul 9 '16 at 17:26
Or go full container on it and put envsubst in a container all by itself. It's a common pattern and a way of life if you make use of an OS like Atomic Host, CoreOS, or RancherOS. Atomic specifically won't even let root mess with the file system or what is installed you have to use a container. – Kuberchaun Aug 22 '16 at 1:26
1

Note that it won't replace "all" environment variables, only those whose name matches ^[[:alpha:]_][[:alnum:]_]*$ in the POSIX locale. – Stéphane Chazelas Jul 27 '17 at 12:26
Seems to be very succinct, however not necessarily correct with all substitution values. It does not seem to respect XML special characters. – EFraim Dec 19 '17 at 10:00

add a comment |

Paolo · Accepted Answer · 2017-08-11 00:27:17Z

12

This is not very nice but it works

( echo "cat <<EOF" ; cat config.xml ; echo EOF ) | sh

If it was in a shell script it would look like:

#! /bin/sh
cat <<EOF
<property>
    <name>instanceId</name>
    <value>$INSTANCE_ID</value>
</property>
EOF

Edit, second proposal:

eval "echo \"$(cat config.xml)\""

Edit, not strictly related to question, but in case of variables read from file:

(. .env && eval "echo \"$(cat config.xml)\"")

edited Aug 11 '17 at 0:27

Paolo

5,7307 gold badges23 silver badges36 bronze badges

answered Jul 9 '16 at 14:25

hschou

2,2566 silver badges13 bronze badges

The problem with this is that if the file contains a line with EOF, the remaining lines will be executed as commands by the shell. We could change the separator to something longer or more complicated, but there's still a theoretical possibility of colliding. And someone could deliberately make a file with the separator to execute commands. – ilkkachu Jul 9 '16 at 14:32
OK, try this: eval "echo \"$(cat config.xml)\"" – hschou Jul 9 '16 at 14:34
3

Try putting something like "; ls ;" inside the file and do that eval command again :) This is pretty much the same problem as with SQL injection attacks. You have to be really careful when mixing data with code (and that's what shell commands are), unless you're really, really sure nobody is trying to do anything to mess up your day. – ilkkachu Jul 9 '16 at 14:43
No. "; ls ;" won't do any harm. – hschou Jul 9 '16 at 16:32
3

@hschou I think ilkkachu meant `"; ls ;"` — the comment formatting ate the backticks. But actually that shoule be just `ls` here. The point is that the content of the file leads to arbitrary code execution and there's nothing you can do about it. – Gilles Jul 9 '16 at 22:44

| show 2 more comments

Stéphane Chazelas · Accepted Answer · 2017-07-27 12:28:58Z

If you happen to have Perl (but not gettext and envsubst) you can do the simple replacement with a short script:

$ export INSTANCE_ID=foo; export SERVICE_NAME=bar;
$ perl -pe 's/\$([_A-Z]+)/$ENV{$1}/g'  < config.xml
<property>
    <name>instanceId</name>
    <value>foo</value>
</property>
<property>
    <name>rootPath</name>
    <value>/services/bar</value>
</property>

I assumed the variable names will only have uppercase letters and underscores, but the first pattern should be easy to alter as needed. $ENV{...} references the environment Perl sees.

If you want to support the ${...} syntax or throw an error on unset variables, you'll need some more work. A close equivalent of gettext's envsubst would be:

perl -pe 's/\$(\{)?([a-zA-Z_]\w*)(?(1)\})/$ENV{$2}/g'

Though I feel that feeding variables like that via the process environment seems a bit iffy in general: you can't use arbitrary variables in the files (since they may have special meanings), and some of the values could possibly have at least semi-sensitive data in them.

Would rather not use Perl since it's supposed to be a docker container, but that looks like the best solution. — Robert Fraser, Jul 9 '16 at 14:30
See also perl -pe 's{\$(\{)?(\w+)(?(1)\})}{$ENV{$2} // $&}ge' to only substitute the variables that are defined. — Stéphane Chazelas, Jul 27 '17 at 12:40

Zanna · Accepted Answer · 2017-04-02 09:06:54Z

May I suggest my own script for this?

https://github.com/rydnr/set-square/blob/master/.templates/common-files/process-file.sh

#!/bin/bash /usr/local/bin/dry-wit
# Copyright 2016-today Automated Computing Machinery S.L.
# Distributed under the terms of the GNU General Public License v3

function usage() {
cat <<EOF
$SCRIPT_NAME -o|--output output input
$SCRIPT_NAME [-h|--help]
(c) 2016-today Automated Computing Machinery S.L.
    Distributed under the terms of the GNU General Public License v3

Processes a file, replacing any placeholders with the contents of the
environment variables, and stores the result in the specified output file.

Where:
    * input: the input file.
    * output: the output file.
Common flags:
    * -h | --help: Display this message.
    * -v: Increase the verbosity.
    * -vv: Increase the verbosity further.
    * -q | --quiet: Be silent.
EOF
}

# Requirements
function checkRequirements() {
  checkReq envsubst ENVSUBST_NOT_INSTALLED;
}

# Error messages
function defineErrors() {
  export INVALID_OPTION="Unrecognized option";
  export ENVSUBST_NOT_INSTALLED="envsubst is not installed";
  export NO_INPUT_FILE_SPECIFIED="The input file is mandatory";
  export NO_OUTPUT_FILE_SPECIFIED="The output file is mandatory";

  ERROR_MESSAGES=(\
    INVALID_OPTION \
    ENVSUBST_NOT_INSTALLED \
    NO_INPUT_FILE_SPECIFIED \
    NO_OUTPUT_FILE_SPECIFIED \
  );

  export ERROR_MESSAGES;
}

## Parses the input
## dry-wit hook
function parseInput() {

  local _flags=$(extractFlags $@);
  local _flagCount;
  local _currentCount;

  # Flags
  for _flag in ${_flags}; do
    _flagCount=$((_flagCount+1));
    case ${_flag} in
      -h | --help | -v | -vv | -q)
         shift;
         ;;
      -o | --output)
         shift;
         OUTPUT_FILE="${1}";
         shift;
         ;;
    esac
  done

  # Parameters
  if [[ -z ${INPUT_FILE} ]]; then
    INPUT_FILE="$1";
    shift;
  fi
}

## Checking input
## dry-wit hook
function checkInput() {

  local _flags=$(extractFlags $@);
  local _flagCount;
  local _currentCount;
  logDebug -n "Checking input";

  # Flags
  for _flag in ${_flags}; do
    _flagCount=$((_flagCount+1));
    case ${_flag} in
      -h | --help | -v | -vv | -q | --quiet)
         ;;
      -o | --output)
         ;;
      *) logDebugResult FAILURE "fail";
         exitWithErrorCode INVALID_OPTION ${_flag};
         ;;
    esac
  done

  if [[ -z ${INPUT_FILE} ]]; then
    logDebugResult FAILURE "fail";
    exitWithErrorCode NO_INPUT_FILE_SPECIFIED;
  fi

  if [[ -z ${OUTPUT_FILE} ]]; then
      logDebugResult FAILURE "fail";
      exitWithErrorCode NO_OUTPUT_FILE_SPECIFIED;
  fi
}

## Replaces any placeholders in given file.
## -> 1: The file to process.
## -> 2: The output file.
## <- 0 if the file is processed, 1 otherwise.
## <- RESULT: the path of the processed file.
function replace_placeholders() {
  local _file="${1}";
  local _output="${2}";
  local _rescode;
  local _env="$(IFS=" \t" env | awk -F'=' '{printf("%s=\"%s\" ", $1, $2);}')";
  local _envsubstDecl=$(echo -n "'"; IFS=" \t" env | cut -d'=' -f 1 | awk '{printf("${%s} ", $0);}'; echo -n "'";);

  echo "${_env} envsubst ${_envsubstDecl} < ${_file} > ${_output}" | sh;
  _rescode=$?;
  export RESULT="${_output}";
  return ${_rescode};
}

## Main logic
## dry-wit hook
function main() {
  replace_placeholders "${INPUT_FILE}" "${OUTPUT_FILE}";
}
# vim: syntax=sh ts=2 sw=2 sts=4 sr noet

Stack Exchange Network

current community

your communities

more stack exchange communities

Replace environment variables in a file with their actual values?

4 Answers 4

Your Answer

Not the answer you're looking for? Browse other questions tagged shell-script text-processing scripting environment-variables or ask your own question.

Linked

Hot Network Questions

Replace environment variables in a file with their actual values?

4 Answers 4

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged shell-script text-processing scripting environment-variables or ask your own question.

Linked

Related

Hot Network Questions