Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

My bash script needs to read the variables from a text file that contains hundreds of variables, most of which conform to standard bash variable sytax (e.g., VAR1=VALUE1). However, a few lines in this file can be problematic and I hope to find a simple solution to reading them into bash.

Here's what the file looks like:

#comment
VAR1=VALUE1
VAR_2=VALUE_2
...
VAR5=VALUE 5 (ASDF)
VAR6=VALUE 6 #a comment
VAR7=/path/with/slashes/and,commas
VAR8=2.1
VAR9=a name with spaces
VAR_10=true
...
VAR_N=VALUE_N

The rules about the file structure include :

  • one variable (with value) per line
  • the assignment (=) has no spaces around it
  • the variable name is in the first column (unless it is a comment line)
  • comments can follow the value (after #)
  • the values can include spaces, parens, slashes, commas and other chars
  • the values can be floating point numbers (2.1) or integers or true/false or strings.
  • string values are not quoted, and they can be a thousand chars long or longer
  • the variable name contains only letters and underscores UPDATE: and numbers.

Most of the variables are of a type that would just allow me to source the file into my bash script. But those few problematic ones dictate a different solution. I'm not sure how to read them.

share|improve this question
    
Can you just put the RHS into single quotes? –  Ketan Feb 22 at 2:05
    
@Ketan - how do you mean? I can't change the text file with the variables. –  MountainX Feb 22 at 2:15
    
You don't have to change the original text file, run it through sed and put the output in a new file and source it. –  Ketan Feb 22 at 2:17
    
@Ketan - that sounds like it should work... can you provide an answer with example code? The devil is often in the details, especially with bash. Thank you. –  MountainX Feb 22 at 2:20

2 Answers 2

up vote 1 down vote accepted

While you can transform this file to be a shell snippet, it's tricky. You need to make sure that all shell special characters are properly quoted.

The easiest way to do that is to put single quotes around the value and replace single quotes inside the value by '\''. You can then put the result into a temporary file and source that file.

script=$(mktemp)
sed <"config-file" >"$script" \
  -e '/^[A-Z_a-z][A-Z_a-z]*=/ !d' \
  -e s/\'/\'\\\\\'\'/g \
  -e s/=/=\'/ -e s/\$/\'/

I recommend doing the parsing directly in the shell instead. The complexity of the code is about the same, but there are two major benefits: you avoid the need for a temporary file, and the risk that you accidentally got the quoting wrong and end up executing a part of a line as a shell snippet (something like dangerous='$(run me)'). You also get a better chance at validating potential errors.

while IFS= read -r line; do
  line=${line%%#*}  # strip comment (if any)
  case $line in
    *=*)
      var=${line%%=*}
      case $var in
        *[!A-Z_a-z]*)
          echo "Warning: invalid variable name $var ignored" >&2
          continue;;
      esac
      if eval '[ -n "${'$var'+1}" ]'; then
        echo "Warning: variable $var already set, redefinition ignored" >&2
        continue
      fi
      line=${line#*=}
      eval $var='"$line"'
  esac
done <"config-file"
share|improve this answer
    
I'm using your recommended (2nd) method. Works great. Is there an easy way to check for a variable naming collision during this while-loop? With this approach it seems like that check might be appropriately incorporated. –  MountainX Feb 23 at 0:43
    
Regarding name collisions, I guess I could use eval IMPORTED_$var='"$line"' but I would rather first check for collisions and not change/prefix the variable names. –  MountainX Feb 23 at 0:53
    
@MountainX See my edit. –  Gilles Feb 23 at 0:55
    
BTW, I made a mistake regarding the variable name convention. The standard letters, numbers and underscores are allowed. So the char class expression should be [!A-Z_a-z0-9]. –  MountainX Feb 23 at 0:57
    
Thanks for the edit. This is an awesome solution! –  MountainX Feb 23 at 0:59

Assuming your contents are in a file x.txt, you can use a sed expression:

sed -e 's/#.*//' -e 's/=.*$/="&"/g' -e 's/=//2' x.txt

First expression puts everything after the = sign in quotes. Second expression removes extra = sign and the third expression removes everything between # and " as they are comments.

Following will quote RHS into single quotes:

sed -e "s/#.*//" -e "s/=.*$/='&'/g" -e "s/=//2" x.txt
share|improve this answer
    
The result is VAR1"=value1" –  MountainX Feb 22 at 2:33
    
Yep, realized that and made an update. See if it works now. –  Ketan Feb 22 at 2:34
    
Yes, that works! Thanks –  MountainX Feb 22 at 2:42
    
This doesn't handle values containing " or '. –  Gilles Feb 22 at 23:13
    
It also handles comments incorrectly (the leading quote is kept, the trailing quote is deleted together with the #). BTW, it's probably useful to delete the comment together with any preceding whitespace. –  Uwe Feb 22 at 23:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.