Use config file for my shell script

Question

I need to create a config file for my own script:
Here is an example:

script:

#!/bin/bash
source /home/myuser/test/config
echo "Name=$nam" >&2
echo "Surname=$sur" >&2

Content of /home/myuser/test/config:

nam="Mark"
sur="Brown"

that works!

My question: is this the correct way to do this or there're other ways?

Answer 1

source is not secure as it will execute arbitrary code. This may not be a concern for you, but if file permissions are incorrect, it may be possible for an attacker with filesystem access to execute code as a privileged user by injecting code into a config file loaded by an otherwise-secured script such as an init script.

So far, the best solution I've been able to identify is the clumsy reinventing-the-wheel solution:

myscript.conf

password=bar
echo rm -rf /
PROMPT_COMMAND='echo "Sending your last command $(history 1) to my email"'
hostname=localhost; echo rm -rf /

Using source, this would run echo rm -rf / twice, as well as change the running user's $PROMPT_COMMAND. Instead, do this:

myscript.sh (Bash 4)

#!/bin/bash
typeset -A config # init array
config=( # set default values in config array
    [username]="root"
    [password]=""
    [hostname]="localhost"
)

while read line
do
    if echo $line | grep -F = &>/dev/null
    then
        varname=$(echo "$line" | cut -d '=' -f 1)
        config[$varname]=$(echo "$line" | cut -d '=' -f 2-)
    fi
done < myscript.conf

echo ${config[username]} # should be loaded from defaults
echo ${config[password]} # should be loaded from config file
echo ${config[hostname]} # includes the "injected" code, but it's fine here
echo ${config[PROMPT_COMMAND]} # also respects variables that you may not have
               # been looking for, but they're sandboxed inside the $config array

myscript.sh (Mac/Bash 3-compatible)

#!/bin/bash
config() {
    val=$(grep -E "^$1=" myscript.conf 2>/dev/null || echo "$1=__DEFAULT__" | head -n 1 | cut -d '=' -f 2-)

    if [[ $val == __DEFAULT__ ]]
    then
        case $1 in
            username)
                echo -n "root"
                ;;
            password)
                echo -n ""
                ;;
            hostname)
                echo -n "localhost"
                ;;
        esac
    else
        echo -n $val
    fi
}

echo $(config username) # should be loaded from defaults
echo $(config password) # should be loaded from config file
echo $(config hostname) # includes the "injected" code, but it's fine here
echo $(config PROMPT_COMMAND) # also respects variables that you may not have
               # been looking for, but they're sandboxed inside the $config array

Please reply if you find a security exploit in my code.

Answer 2

Parse the configuration file, don't execute it.

I'm currently writing an application at work that uses an extremely simple XML configuration:

<config>
    <username>username-or-email</username>
    <password>the-password</password>
</config>

In the shell script (the "application"), this is what I do to get at the username (more or less, I've put it in a shell function):

username=$( xmlstarlet sel -t -v '/config/username' "$config_file" )

The xmlstarlet command is XMLStarlet, which is available for most Unices. On some systems, it's installed as xml.

I'm using XML since other parts of the application also deals with data encoded in XML files, so it was easiest.

If you prefer JSON, there's jq which is an easy to use shell JSON parser.

My configuration file would look something like this in JSON:

{                                 
  "username": "username-or-email",
  "password": "the-password"      
}         

And then I'd be getting the username in the script:

username=$( jq -r .username "$config_file" )

There is also TOML ("Tom's Obvious Minimal Language"), with parsers for several languages. My personal favorite parser, tomlq, is part of the yq distribution from https://kislyuk.github.io/yq/ (uses jq behind the scenes).

The TOML version of your configuration file would look like

username = "username-or-email"
password = "the-password"

(the strings would be JSON-encoded)... and getting data out from it would be as simple as in the JSON case (since tomlq is built on top of jq):

username=$( tomlq -r .username "$config_file" )

Answer 3

Here is a clean and portable version which is compatible with Bash 3 and up, on both Mac and Linux.

It specifies all defaults in a separate file, to avoid the need for a huge, cluttered, duplicated "defaults" config function in all of your shell scripts. And it lets you choose between reading with or without default fallbacks:

config.cfg:

myvar=Hello World

config.cfg.defaults:

myvar=Default Value
othervar=Another Variable

config.shlib (this is a library, so there is no shebang-line):

config_read_file() {
    (grep -E "^${2}=" -m 1 "${1}" 2>/dev/null || echo "VAR=__UNDEFINED__") | head -n 1 | cut -d '=' -f 2-;
}

config_get() {
    val="$(config_read_file config.cfg "${1}")";
    if [ "${val}" = "__UNDEFINED__" ]; then
        val="$(config_read_file config.cfg.defaults "${1}")";
    fi
    printf -- "%s" "${val}";
}

test.sh (or any scripts where you want to read config values):

#!/usr/bin/env bash
source config.shlib; # load the config library functions
echo "$(config_get myvar)"; # will be found in user-cfg
printf -- "%s\n" "$(config_get myvar)"; # safer way of echoing!
myvar="$(config_get myvar)"; # how to just read a value without echoing
echo "$(config_get othervar)"; # will fall back to defaults
echo "$(config_get bleh)"; # "__UNDEFINED__" since it isn't set anywhere

Explanation of the test script:

• Note that all usages of config_get in test.sh are wrapped in double quotes. By wrapping every config_get in double quotes, we ensure that text in the variable value will never be misinterpreted as flags. And it ensures that we preserve whitespace properly, such as multiple spaces in a row in the config value.
• And what's that printf line? Well, it's something you should be aware of: echo is a bad command for printing text that you have no control over. Even if you use double quotes, it will interpret flags. Try setting myvar (in config.cfg) to -e and you will see an empty line, because echo will think that it's a flag. But printf doesn't have that problem. The printf -- says "print this, and don't interpret anything as flags", and the "%s\n" says "format the output as a string with a trailing newline, and lastly the final parameter is the value for printf to format.
• If you aren't going to be echoing values to the screen, then you'd simply assign them normally, like myvar="$(config_get myvar)";. If you're going to print them to the screen, I suggest using printf to be totally safe against any echo-incompatible strings that may be in the user config. But echo is fine if the user-provided variable isn't the first character of the string you are echoing, since that's the only situation where "flags" could be interpreted, so something like echo "foo: $(config_get myvar)"; is safe, since the "foo" doesn't begin with a dash and therefore tells echo that the rest of the string isn't flags for it either. :-)

Answer 4

Most users, although not many containers, already have the git binary. Why not therefore use git config for application configuration management using a dedicated non-conflicting configuration file as in the examples below?

# Set
$ git config -f ~/.myapp core.mykey myval

# Get
$ git config -f ~/.myapp core.mykey
myval

# Get invalid
$ git config -f ~/.myapp core.mykey
$ echo $?
1

# List
git config -f ~/.myapp -l
core.mykey=myval

# View
$ cat ~/.myapp 
[core]
    mykey = myval

For additional commands, see its man page. It is wise to first ensure the configuration file exists:

touch -a ~/.myapp

Answer 5

I use this in my scripts:

sed_escape() {
  sed -e 's/[]\/$*.^[]/\\&/g'
}

cfg_write() { # path, key, value
  cfg_delete "$1" "$2"
  echo "$2=$3" >> "$1"
}

cfg_read() { # path, key -> value
  test -f "$1" && grep "^$(echo "$2" | sed_escape)=" "$1" | sed "s/^$(echo "$2" | sed_escape)=//" | tail -1
}

cfg_delete() { # path, key
  test -f "$1" && sed -i "/^$(echo $2 | sed_escape).*$/d" "$1"
}

cfg_haskey() { # path, key
  test -f "$1" && grep "^$(echo "$2" | sed_escape)=" "$1" > /dev/null
}

Should support every character combination, except keys can't have = in them, since that's the separator. Anything else works.

% cfg_write test.conf mykey myvalue
% cfg_read test.conf mykey
myvalue
% cfg_delete test.conf mykey
% cfg_haskey test.conf mykey || echo "It's not here anymore"
It's not here anymore

Also, this is completely safe since it doesn't use source or eval.

Answer 6

The most common, efficient and correct way is to use source, or . as a shorthand form. For example:

source /home/myuser/test/config

or

. /home/myuser/test/config

Something to consider, however, is the security issues that using an additional externally-sourced configuration file can raise, given that additional code can be inserted. For more information, including on how to detect and resolve this issue, I would recommend taking a look at the 'Secure it' section of http://wiki.bash-hackers.org/howto/conffile#secure_it

Answer 7

This is succinct and secure:

# Read common vars from common.vars
# the incantation here ensures (by env) that only key=value pairs are present
# then declare-ing the result puts those vars in our environment 
declare $(env -i `cat common.vars`)

The -i ensures you only get the variables from common.vars

Update: An illustration of the security is that

env -i 'touch evil1 foo=omg boo=$(touch evil2)'

Will not produce any touched files. Tested on mac with bash, i.e. using bsd env.

Answer 8

This one seems safe and short. Feel free to crack this ruthlessly. I'd like to know of a better way.

TL;DR;

while read LINE; do declare "$LINE"; done < evil.conf

I'm using bash 4.3.48.

It's also compliant with bash --posix. See bottom for test.

But sh doesn't support it because of declare.

---

Basic test for those who want proof

Create file evil.conf

echo > evil.conf '
A=1
B=2
C=$(echo hello)

# Could produce side-effect
D=`touch evil`
C=$((1+2))
E=$(ping 8.8.8.8 -n 3)
echo hello

# Could produce visible side-effect
touch evil2
ping 8.8.8.8 -n 3
F=ok'

Load the config with the snippet

while read LINE; do declare "$LINE"; done < evil.conf

Output (see sanitizer in action)

bash: declare: `': not a valid identifier
bash: declare: `': not a valid identifier
bash: declare: `# Could produce side-effect': not a valid identifier
bash: declare: `echo hello': not a valid identifier
bash: declare: `': not a valid identifier
bash: declare: `# Could produce visible side-effect': not a valid identifier
bash: declare: `touch evil2': not a valid identifier
bash: declare: `ping 8.8.8.8 -n 3': not a valid identifier

Let's check values now

for V in A B C D E F; do declare -p $V; done

declare -- A="1"
declare -- B="2"
declare -- C="\$((1+2))"
declare -- D="\`touch evil\`"
declare -- E="\$(ping 8.8.8.8 -n 3)"
declare -- F="ok"

Check side effects (no side effects):

ls evil evil2

ls: cannot access 'evil': No such file or directory
ls: cannot access 'evil2': No such file or directory

Appendix. Test bash --posix

bash -c 'while read LINE; do declare "$LINE"; done < evil.conf; for V in A B C D E F; do declare -p $V; done' --posix

Answer 9

A bit late to the party here, but a simple way of reading env-style files as configuration for a script while preventing execution of arbitrary and potentially malicious code (that the source approach is vulnerable to) is to combine the envsubst utility, which only substitutes variables and not processes, with the export command, which only allows assignment to environment variables and not execution of arbitrary code. A configuration file can therefore be loaded in a source /path/to/config.env-fashion with the following one-liner:

eval export $(envsubst < /path/to/config.env \      # Substitute only variables, not processes
| grep --perl-regexp --invert-match '^\s*(#|$)' \   # Ignore comments and empty lines
| xargs --delimiter '\n' printf " %q")              # Escape and concat strings

(Note: the placement of comments in the above snippet is not valid bash; they must be removed for correct execution.)

For example:

# ==== File /path/to/config.env ==== #

# This comment and the following blank line are ignored

TEST1=abc
TEST2="Sweet ${HOME}"
TEST3=$(echo malicious!)

# The following command would produce a parse error:
# ./script.sh: line 6: export: `touch malicious': not a valid identifier
#touch malicious

# ==== File ./script.sh ==== #

#!/bin/bash

set -euo pipefail

function load_config {
    eval export $(envsubst < "$1" \
    | grep -Pv '^\s*(#|$)' \
    | xargs -d '\n' printf " %q")
}

load_config /path/to/config.env
env | grep TEST

# ==== Terminal ==== #

$ ./script.sh /path/to/config.env
TEST3=$(echo malicious!)
TEST2="Sweet /home/me"
TEST1=abc

ADDENDUM #1:

Caveats: The envsubst utility is not syntax-aware — something I had initially overlooked — and will substitute all occurrences of substitution-looking strings anywhere in the config file, such as ${USER}=xyz (normally invalid variable assignment) and TEST='Sweet ${HOME}' (normally not substituted). Furthermore, it will invariably substitute undefined identifiers with empty strings, and it does not recognize conditional substitutions (such as ${VAR:-val}) and Bash associative arrays (e.g. ${VAR[KEY]}). All these are not necessarily problems, depending on the use case, but limitations to keep in mind.

ADDENDUM #2:

This variant on the previous form exports variables one by one, allowing definitions lower in the configuration file to be aware of previously defined variables:

while read -r LINE; do
    eval export $(envsubst <<< "$LINE" | xargs --delimiter '\n' printf "%q")
done < <(grep --perl-regexp --invert-match '^\s*(#|$)' /path/to/config.env)

Answer 10

I filter for lines that look like an expected parameter definition, then use export to bring them in (other variable-defining keywords don't seem to work with this method, unfortunately).

# Define the full set of parameters that we will read from the config file
params="BASENAME SOURCE SHORT_LIFE LONG_LIFE LONG_FREQ TS_FORMAT INDEX_FILE COUNTER_FILE"

# Load in the profile (loop through all possible legal params)
for p in $params
do
        export "`grep ^$p= $config || echo $p=`"
done

Ignores all lines that don't match what it's looking for, is compact, easily extended and (I think) secure against malicious config files.

Config file looks like this:

BASENAME=foo
SOURCE=bar
this is a comment
# this is also a comment
BASENAME this will be ignored because there`s no = sign
 BASENAME= this will be ignored too (line doesn`t start with param name)
...
COUNTER_FILE=baz

Answer 11

For my scenario, source or . was fine, but I wanted to support local environment variables (ie, FOO=bar myscript.sh) taking precedence over configured variables. I also wanted the config file to be user editable and comfortable to someone used to sourced config files, and to keep it as small/simple as possible, to not distract from the main thrust of my very small script.

This is what I came up with:

CONF=${XDG_CONFIG_HOME:-~/config}/myscript.sh
if [ ! -f $CONF ]; then
    cat > $CONF << CONF
VAR1="default value"
CONF
fi
. <(sed 's/^\([^=]\+\) *= *\(.*\)$/\1=${\1:-\2}/' < $CONF)

Essentially - it checks for variable definitions (without being very flexible about whitespace) and rewrites those lines so that the value is converted to a default for that variable, and the variable is unmodified if found, like the XDG_CONFIG_HOME variable above. It sources this altered version of the config file and continues on.

Future work could make the sed script more robust, filter out lines that look weird or aren't definitions, etc, not break on end of line comments - but this is good enough for me for now.

Answer 12

Inspired by this article mentioned here in this response I came up with my own solution.

The idea is the same, make a filtered copy of the config file in /tmp and use this one instead of the original.

In addition, emty lines will be ignored and leading spaces will be removed.

Original config file with malicious code in it:

# Comments are allowed
TEXT_WITH_SINGLE_QUOTES='hello'
TEXT_WITH_DOUBLE_QUOTES="world"

INTEGER_VALUE=2357
DECIMAL_VALUE=0.0011
    SPACES_BEFORE='will be removed'

    # Examples with malicious code
MALICIOUS_EXAMPLE_1='var1' ; echo 'foo'
MALICIOUS_EXAMPLE_2='var2' & echo 'bar'

MALICIOUS_EXAMPLE_3='var3' && echo 'fuu'

The code to filter the input file:

CONFIG='./my.conf'
CONFIG_SECURED='/tmp/my.conf'

rm -f $CONFIG_SECURED

while IFS= read -r line; do
  echo $(sed -E "s/^(\s)?+(#.+$|[a-zA-Z0-9_]+=['\"]?[a-zA-Z0-9_~\.\`-]+['\"]).+$/\2/g" <<< $line) >> $CONFIG_SECURED
  CONFIG=$CONFIG_SECURED
done <<EOF
`cat $CONFIG | grep -v "^$"`
EOF

. $CONFIG

The filtered config file without malicious code in it:

# Comments are allowed
TEXT_WITH_SINGLE_QUOTES='hello'
TEXT_WITH_DOUBLE_QUOTES="world"
INTEGER_VALUE=2357
DECIMAL_VALUE=0.0011
SPACES_BEFORE='will be removed'
# Examples with malicious code
MALICIOUS_EXAMPLE_1='var1'
MALICIOUS_EXAMPLE_2='var2'
MALICIOUS_EXAMPLE_3='var3'

The filtering part is certainly perfectible, but it think it does most of the important stuff.

Answer 13

my favourite approach is to use crudini
on Ubuntu you can install it with sudo apt install crudini
example:

$ crudini --set example.ini "section name" "value name" 'malicious; echo $HOME'
$ cat example.ini 
[section name]
value name = malicious; echo $HOME
$ value_name=$(crudini --get example.ini "section name" "value name")
$ echo "$value_name"
malicious; echo $HOME
$