9

I'd like to read the properties in a properties file into an associative array. How can I do it?

Specifics on what to parse: hash and equal signs. Everything else is a bonus.

Properties file content example:

# comment
a=value-a
b=http://prefix.suffix:8080/?key=value
c=password_with\\backslash-and=equals

I'd like this bash associative array to be constructed out of that file:

 declare -A props='(
  [a]="value-a"
  [b]="http://prefix.suffix:8080/?key=value"
  [c]="password_with\\backslash-and=equals" )'

(expected output of declare -p on that associative array, note that ${props[c]} contains only one backslash, "\\" is '\').

8
  • augeas has a property file lens so using augtool to retrieve whatever values you're interested in was another possible. IMHO putting some sort of third party parsing logic into your script is usually better than cobbling together yourself. You might be able to find something that seems to work for the data you've seen so far but it's generally safer to use logic someone else has vetted. You'd definitely want to limit the lenses you load if you were going to use augtool in a bash script or something.
    – Bratchley
    Jul 18, 2016 at 15:48
  • 1
    does it need to handle comments (of the form ! and #) as well as the key/value formats: key=value, key = value, key:value, and key value?
    – Jeff Schaller
    Jul 18, 2016 at 16:44
  • @jeff - I don't know. Most of the properties files I saw use equals for separation and hash for comments. Jul 18, 2016 at 18:40
  • 1
    I try to ask questions that clarify the OP so that we can get good answers. If you link to an example properties file but only want to parse some of them, then you need to clarify that. In either case, I think you should import the linked properties file into your question so that we don't have to leave the site to determine the requirements.
    – Jeff Schaller
    Jul 18, 2016 at 20:42
  • 1
    Sorry if I was unclear, @StéphaneChazelas -- my main point was that AlikElzin-kilaka linked to a sample file but then "didn't know" what syntax they wanted to support. I don't personally care what syntax that is, just that this Question should be clear about it.
    – Jeff Schaller
    Jul 19, 2016 at 12:34

4 Answers 4

6

Here is howto in Bash4+

#!/usr/bin/env bash

declare -A properties

# Read with:
# IFS (Field Separator) =
# -d (Record separator) newline
# first field before separator as k (key)
# second field after separator and reminder of record as v (value)
while IFS='=' read -d $'\n' -r k v; do
  # Skip lines starting with sharp
  # or lines containing only space or empty lines
  [[ "$k" =~ ^([[:space:]]*|[[:space:]]*#.*)$ ]] && continue
  # Store key value into assoc array
  properties[$k]="$v"
  # stdin the properties file
done < file.properties

# display the array for testing
typeset -p properties

file.properties:

# comment
a=value-a
b=http://prefix.suffix:8080/?key=value
c=password_with\\backslash-and=equals

d e=the d e value
  # comment

The output of this script from the supplied data sample:

declare -A properties=(["d e"]="the d e value" [c]="password_with\\\\backslash-and=equals" [b]="http://prefix.suffix:8080/?key=value" [a]="value-a" )
2
  • This version worked for me with regular property files (NAME=API_KEY). The version mentionned in this thread, where ${line% =*} is the key : I couldn't reaccess the value via the array.
    – Thomas
    May 7, 2020 at 13:44
  • This is in my opinion the best answer. Simple and straightforward, using pure bash (perl is great, but the question wanted bash; plus this is easier to read unless you're really familiar with perl). Plus is skips comments and empty lines, which is a nice perk.
    – Alacritas
    Aug 1, 2020 at 13:00
6

Use a real parser like perl's Config::Properties module. I would do the whole script in perl, but if you have to use bash, you could do something like:

typeset -A props
while IFS= read -rd '' key && IFS= read -rd '' value; do
  props[$key]=$value
done < <(
  perl -MConfig::Properties -l0 -e '
   $p = Config::Properties->new();
   $p->load(STDIN);
   print for $p->properties' < file.properties
)

(also works with zsh).

Implementing a full parser in bash would be a lot of work and mean re-inventing the wheel. You can implement a good subset with a simple while read loop though as the read builtin expects an input format that is very similar to those properties files:

typeset -A props
while IFS=$':= \t' read key value; do
  [[ $key = [#!]* ]] || [[ $key = "" ]] || props[$key]=$value
done < file.properties

(also works with ksh93 and zsh, the two other Bourne-like shell supporting associative arrays).

That handles:

  • prop = value
  • prop: value
  • prop value
  • comments at the start of the line (! and # with optional leading blanks)
  • backslash escaping (as in foo\:\:bar=value for keys containing delimiters or foo=\ bar or the password_with\\backslash-and=equals in your sample).
  • line continuation with backslash

However, if we check against the specification

  • That doesn't handle \n, \r, \uXXXX... sequences

  • LF is the only recognised line delimiter (not CR nor CRLF).

  • FF is not recognised as a whitespace (we can't just add it to $IFS as depending on the shell and version, \f will not necessarily be recognised as an IFS-whitespace character¹).

  • for an input like foo: bar = , that stores bar in ${props[foo]} instead of bar = (foo: bar:baz: is OK though). That's only a problem when the value of the property contains one (unescaped) delimiter (: optionally surrounded by SPC/TAB characters, = optionally surrounded by SPC/TAB characters or sequence of one or more SPC/TAB characters) and it is at the end.

  • it treats as comments lines that start with \! or \#. Only a problem for properties whose name starts with ! or #.

  • in

      prop=1\
       2\
       3
    

we get 1 2 3 instead of 123: the leading spaces are not ignored in the continuation lines as they should be.


² IFS whitespace characters, per POSIX are the characters classified as [:space:] in the locale (which generally includes \f but doesn't have to) and that happen to be in $IFS though in ksh88 (on which the POSIX specification is based) and in most shells, that's still limited to SPC, TAB and NL. The only POSIX compliant shell in that regard I found was yash. ksh93 and bash (since 5.0) also include other whitespace (such as CR, FF, VT...), but limited to the single-byte ones (beware on some systems like Solaris, that includes the non-breaking-space which is single byte in some locales)

6
  • Can't locate Config/Properties.pm in @INC (you may need to install the Config::Properties module). Missing perl module :( Jul 19, 2016 at 11:50
  • @AlikElzin-kilaka, hence the link to where to find it in my answer. You can do install Config::Properties in cpan. Jul 19, 2016 at 12:08
  • 1
    installing modules from CPAN isn't hard, but (depending on your unix or linux distribution) you may be able to install it even easier with your system's package manager. e.g. on debian apt-get install libconfig-properties-perl.
    – cas
    Jul 19, 2016 at 12:16
  • @cas I'm not able to find this package in Ubuntu 16.04 Is there any alternative for this in Xenial ? packages.ubuntu.com/xenial/perl Nov 10, 2017 at 19:31
  • there are dozens of perl config file parsers. Config::Simple (packaged as libconfig-simple-perl) will probably work with this file format. See packages.ubuntu.com/xenial/all/perl for a list of all packaged perl modules in xenial, and search that page for libconfig-.
    – cas
    Nov 11, 2017 at 2:48
4

For the most common subset of that dataformat, you can use a short function, using bash variable expansion and regexp matching.

Note: This expects lines to be in ^key = value$ format, or ^#.*$ and ^!.*$ for comments. Adapt the code, or pre-process your data otherwise

$ cat /tmp/propdata 
k1 = v1
# A comment
k2 = v2and some s=t=u=f=f
! Another comment
k3 = v3

$ unset DATA
$ declare -A DATA

$ props(){ while read line || [[ -n $line ]]; do
[[ "$line" =~ ^#|^! ]] && continue;
if [[ "${line% =*}" ]]; then DATA[${line% =*}]="${line#*= }" ; fi ;
done < $1 ; }

$ props /tmp/propdata

$ echo "${DATA[k3]}"
v3
$ echo "${DATA[k2]}"
v2and some s=t=u=f=f

Edit: Updated to trim the spaces around the "=" for key and value

Edit2: Filters comments now too.

2
declare -A properties
function readPopertyFile
{
    while read line || [[ -n $line ]]; do
        key=`echo $line | cut -s -d'=' -f1`
        if [ -n "$key" ]; then
            value=`echo $line | cut -d'=' -f2-`
            properties["$key"]="$value"
        fi
    done < $1
}

Usage:

readPopertyFile "file.properties"

Will read the properties into an associative array variable named properties.

* Works in bash. Don't know about other shells.

* Won't handle multi-line properties.

6
  • 2
    Using bash parameter expansion to set the variables will save you 2 sub shells, 2 echo, 2 pipes and two cut commands with key=${line%=*} and value=${line#*=}. that's less clear but much faster if the file is big.
    – Emmanuel
    Jul 18, 2016 at 15:58
  • @Emmanuel - Will ${line#*=} "eat" the whole value? Even when there are several = (equal) chars? Jul 18, 2016 at 16:16
  • 1
    Always double quote variable expansions. Your script breaks on lines containing spaces and wildcards due to the missing double quotes. But as Emmanuel advises you should use parameter expansion instead of echo | cut, it's easier to get it right and faster. ${line#*=} cuts off the part up to the first =, and ${line%%*=} (note double %) cuts off the part from the first =. Jul 18, 2016 at 23:40
  • I noted the cut -s that is followed by a test on empty strings. This is to deal with lines containing no "=". Parameter expansion will not help here but you can filter the input at the beginning of the script : grep = $1 | while read line. It adds one grep to the script but as it is outside the loop, that's ok.
    – Emmanuel
    Jul 19, 2016 at 8:51
  • @AlikElzin-kilaka I assumed values would not contain an =, if thats the case use the %% (longest matching string) as advised by Giles.
    – Emmanuel
    Jul 19, 2016 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.