10

Is there any way to serialize a shell variable? Suppose I have a variable $VAR, and I want to be able to save it to a file or whatever, and then read it back later to get the same value back?

Is there a portable way of doing this? (I don't think so)

Is there a way to do it in bash or zsh?

  • 2
    Attention: the version of my answer you accepted the other day had a serious issue that will break in some scenarios. I have re-written it to include fixes (and add features) and you should really re-read it from scratch and port your code to use the fixed version. – Caleb Jun 16 '14 at 13:06
  • ^Another^ example of @Caleb's upstanding citizenship. – mikeserv Jun 19 '14 at 19:56
13

Warning: With any of these solutions, you need to be aware that you are trusting the integrity of the data files to be safe as they will get executed as shell code in your script. Securing them is paramount to your script's security!

Simple inline implementation for serializing one or more variables

Yes, in both bash and zsh you can serialize the contents of a variable in a way that is easy to retrieve using the typeset builtin and the -p argument. The output format is such that you can simply source the output to get your stuff back.

 # You have variable(s) $FOO and $BAR already with your stuff
 typeset -p FOO BAR > ./serialized_data.sh

You can get your stuff back like this either later in your script or in another script altogether:

# Load up the serialized data back into the current shell
source serialized_data.sh

This will work for bash, zsh and ksh including passing data between different shells. Bash will translate this to its builtin declare function while zsh implements this with typeset but as bash has an alias for this to work either way for we use typeset here for ksh compatibility.

More complex generalized implementation using functions

The above implementation is really simple, but if you call it frequently you might want to give yourself a utility function to make it easier. Additionally if you ever try to include the above inside custom functions you will run into issues with variable scoping. This version should eliminate those issues.

Note for all of these, in order to maintain bash/zsh cross-compatibility we will be fixing both the cases of typeset and declare so the code should work in either or both shells. This adds some bulk and mess that could be eliminated if you were only doing this for one shell or another.

The main problem with using functions for this (or including the code in other functions) is that the typeset function generates code that, when sourced back into a script from inside a function, defaults to creating a local variable rather than a global one.

This can be fixed with one of several hacks. My initial attempt to to fix this was parse the output of the serialize process through sed to add the -g flag so the created code defines a global variable when sourced back in.

serialize() {
    typeset -p "$1" | sed -E '0,/^(typeset|declare)/{s/ / -g /}' > "./serialized_$1.sh"
}
deserialize() {
    source "./serialized_$1.sh"
}

Note that the funky sed expression is to only match the first occurrence of either 'typeset' or 'declare' and add -g as a first argument. It is necessary to only match the first occurrence because, as Stéphane Chazelas rightly pointed out in comments, otherwise it will also match cases where the serialized string contains literal newlines followed by the word declare or typeset.

In addition to correcting my initial parsing faux pas, Stéphane also suggested a less brittle way to hack this that not only side steps the issues with parsing the strings but could be a useful hook to add additional functionality by using a wrapper function to redefine the actions taken when sourcing the data back in. This assumes you are not playing any other games with the declare or typeset commands, but this technique would be easier to implement in a situation where you were including this functionality as part of another function of your own or you were not in control of the data being written and whether or not it had the -g flag added. Something similar could also be done with aliases, see Gilles's answer for an implementation.

To make the result even more useful, we can iterate over multiple variables passed to our functions by assuming that each word in the argument array is a variable name. The result becomes something like this:

serialize() {
    for var in $@; do
        typeset -p "$var" > "./serialized_$var.sh"
    done
}

deserialize() {
    declare() { builtin declare -g "$@"; }
    typeset() { builtin typeset -g "$@"; }
    for var in $@; do
        source "./serialized_$var.sh"
    done
    unset -f declare typeset
}

With either solution, usage would look like this:

# Load some test data into variables
FOO=(an array or something)
BAR=$(uptime)

# Save it out to our serialized data files
serialize FOO BAR

# For testing purposes unset the variables to we know if it worked
unset FOO BAR

# Load  the data back in from out data files
deserialize FOO BAR

echo "FOO: $FOO\nBAR: $BAR"
  • declare is the bash equivalent of ksh's typeset. bash, zsh also support typeset so in that regard, typeset is more portable. export -p is POSIX, but it doesn't take any argument and its output is shell dependent (though it's well specified for POSIX shells, so for instance when bash or ksh is called as sh). Remember to quote your variables; using the split+glob operator here doesn't make sense. – Stéphane Chazelas Jun 15 '14 at 16:04
  • Note that -E is only found in some BSDs sed. Variable values may contain newline characters, so the sed 's/^.../.../' is not guaranteed to work correctly. – Stéphane Chazelas Jun 15 '14 at 16:07
  • This is exactly what I was looking for! I wanted a convenient way to push variables back and forth between shells. – fwenom Jun 15 '14 at 16:12
  • I meant: a=$'foo\ndeclare bar' bash -c 'declare -p a' for install will output a line that starts with declare. It's probably better to do declare() { builtin declare -g "$@"; } before calling source (and unset afterwards) – Stéphane Chazelas Jun 16 '14 at 10:46
  • 2
    @Gilles, aliases wouldn't work inside functions (need to be defined at the time of the function definition), and with bash that would mean you'd need to do a shopt -s expandalias when not interactive. With functions, you could also enhance the declare wrapper so it only restores the variables you specify. – Stéphane Chazelas Jun 16 '14 at 11:26
3

Use redirection, command substitution, and parameter expansion. Double quotes are needed to preserve whitespace and special characters. The trailing x saves the trailing newlines which would be otherwise removed in the command substitution.

#!/bin/bash
echo "$var"x > file
unset var
var="$(< file)"
var=${var%x}
  • He probably wants to save the variable name also to the file. – user80551 Jun 14 '14 at 11:44
1

Serialize all — POSIX

In any POSIX shell, you can serialize all environment variables with export -p. This doesn't include non-exported shell variables. The output is properly quoted so that you can read it back in the same shell and get exactly the same variable values. The output might not be readable in another shell, for example ksh uses the non-POSIX $'…' syntax.

save_environment () {
  export -p >my_environment
}
restore_environment () {
  . ./my_environment
}

Serialize some or all — ksh, bash, zsh

Ksh (both pdksh/mksh and ATT ksh), bash and zsh provide a better facility with the typeset builtin. typeset -p prints out all defined variables and their values (zsh omits the values of variables that have been hidden with typeset -H). The output contains proper declaration so that environment variables are exported when read back (but if a variable is already exported when read back, it won't be unexported), so that arrays are read back as arrays, etc. Here also, the output is properly quoted but is only guaranteed to be readable in the same shell. You can pass a set of variables to serialize on the command line; if you don't pass any variable then all are serialized.

save_some_variables () {
  typeset -p VAR OTHER_VAR >some_vars
}

In bash and zsh, restoring can't be done from a function because typeset statements inside a function are scoped to that function. You need to run . ./some_vars in the context where you want to use the variables' values, taking care that variables that were global when exported will be redeclared as global. If you want to read back the values within a function and export them, you can declare a temporary alias or function. In zsh:

restore_and_make_all_global () {
  alias typeset='typeset -g'
  . ./some_vars
  unalias typeset
}

In bash (which uses declare rather than typeset):

restore_and_make_all_global () {
  alias declare='declare -g'
  shopt -s expand_aliases
  . ./some_vars
  unalias declare
}

In ksh, typeset declares local variables in functions defined with function function_name { … } and global variables in functions defined with function_name () { … }.

Serialize some — POSIX

If you want more control, you can export the content of a variable manually. To print the content of a variable exactly into a file, use the printf builtin (echo has a few special cases such as echo -n on some shells and adds a newline):

printf %s "$VAR" >VAR.content

You can read this back with $(cat VAR.content), except that the command substitution strips off trailing newlines. To avoid this wrinkle, arrange for the output not to end with a newline ever.

VAR=$(cat VAR.content && echo a)
if [ $? -ne 0 ]; then echo 1>&2 "Error reading back VAR"; exit 2; fi
VAR=${VAR%?}

If you want to print multiple variables, you can quote them with single quotes, and replace all embedded single quotes with '\''. This form of quoting can be read back into any Bourne/POSIX-style shell. The following snippet works in any POSIX shell. It only works for string variables (and numeric variables in shells that have them, though they'll be read back as strings), it doesn't try to deal with array variables in shells that have them.

serialize_variables () {
  for __serialize_variables_x do
    eval "printf $__serialize_variables_x=\\'%s\\'\\\\n \"\$${__serialize_variables_x}\"" |
    sed -e "s/'/'\\\\''/g" -e '1 s/=.../=/' -e '$ s/...$//'
  done
}

Here's another approach which doesn't fork a subprocess but is heavier on string manipulation.

serialize_variables () {
  for __serialize_variables_var do
    eval "__serialize_variables_tail=\${$__serialize_variables_var}"
    while __serialize_variables_quoted="$__serialize_variables_quoted${__serialize_variables_tail%%\'*}"
          [ "${__serialize_variables_tail%%\'*}" != "$__serialize_variables_tail" ]; do
      __serialize_variables_tail="${__serialize_variables_tail#*\'}"
      __serialize_variables_quoted="${__serialize_variables_quoted}'\\''"
    done
    printf "$__serialize_variables_var='%s'\n" "$__serialize_variables_quoted"
  done
}

Note that on shells that allow read-only variables, you'll get an error if you try to read back a variable that's read-only.

  • This brings in variables such as $PWD and $_ - please see your own comments below. – mikeserv Jun 15 '14 at 22:17
  • @Caleb How about making typeset an alias for typeset -g? – Gilles Jun 16 '14 at 11:20
  • @Gilles I thought of that after Stephanie suggested the function method, but I wasn't sure how to portably set the necessary alias expand options across shells. Maybe you could put that in your answer as a viable alternative to the function one I included. – Caleb Jun 16 '14 at 11:28
0

Much thanks to @stéphane-chazelas who pointed out all the problems with my previous attempts, this now seems to work to serialise an array to stdout or into a variable.

This technique does not shell-parse the input (unlike declare -a/declare -p) and so is safe against malicious insertion of metacharacters in the serialised text.

Note: newlines are not escaped, because read deletes the \<newlines> character pair, so -d ... must instead be passed to read, and then unescaped newlines are preserved.

All this is managed in the unserialise function.

Two magic characters are used, the field separator and the record separator (so that multiple arrays can be serialized to the same stream).

These characters can be defined as FS and RS but neither can be defined as newline character because an escaped newline is deleted by read.

The escape character must be \ the backslash, as that is what is used by read to avoid the character being recognized as an IFS character.

serialise will serialise "$@" to stdout, serialise_to will serialise to the varable named in $1

serialise() {
  set -- "${@//\\/\\\\}" # \
  set -- "${@//${FS:-;}/\\${FS:-;}}" # ; - our field separator
  set -- "${@//${RS:-:}/\\${RS:-:}}" # ; - our record separator
  local IFS="${FS:-;}"
  printf ${SERIALIZE_TARGET:+-v"$SERIALIZE_TARGET"} "%s" "$*${RS:-:}"
}
serialise_to() {
  SERIALIZE_TARGET="$1" serialise "${@:2}"
}
unserialise() {
  local IFS="${FS:-;}"
  if test -n "$2"
  then read -d "${RS:-:}" -a "$1" <<<"${*:2}"
  else read -d "${RS:-:}" -a "$1"
  fi
}

and unserialise with:

unserialise data # read from stdin

or

unserialise data "$serialised_data" # from args

e.g.

$ serialise "Now is the time" "For all good men" "To drink \$drink" "At the \`party\`" $'Party\tParty\tParty'
Now is the time;For all good men;To drink $drink;At the `party`;Party   Party   Party:

(without a trailing newline)

read it back:

$ serialise_to s "Now is the time" "For all good men" "To drink \$drink" "At the \`party\`" $'Party\tParty\tParty'
$ unserialise array "$s"
$ echo "${array[@]/#/$'\n'}"

Now is the time 
For all good men 
To drink $drink 
At the `party` 
Party   Party   Party

or

unserialise array # read from stdin

Bash's read respects the escape character \ (unless you pass the -r flag) to remove special meaning of characters such as for input field separation or line delimiting.

If you want to serialise an array instead of a mere argument list then just pass your array as the argument list:

serialise_array "${my_array[@]}"

You can use unserialise in a loop like you would read because it is just a wrapped read - but remember that the stream is not newline separated:

while unserialise array
do ...
done
  • It doesn't work if the elements contain non-printable (in the current locale) or control characters like TAB or newline as then bash and zsh render them as $'\xxx'. Try with bash -c $'printf "%q\n" "\t"' or bash -c $'printf "%q\n" "\u0378"' – Stéphane Chazelas Apr 29 '16 at 9:31
  • darn tootin, you are right! I'll modify my answer to not use printf %q but ${@//../..} iterations to escape white space instead – Sam Liddicott Apr 29 '16 at 10:06
  • That solution depends on $IFS being unmodified and now fails to restore empty array elements properly. In fact, it would make more sense to use a different value of IFS, and use -d '' to avoid having to escape newline. For instance, use : as the field separator and only escape that and the backslash and use IFS=: read -ad '' array to import. – Stéphane Chazelas Apr 29 '16 at 12:35
  • Yup.... I forgot about white-space collapsing special treatment when used as a field separator in read. I'm glad you are on the ball today! You are right about -d "" to avoid escaping \n, but in my case I wanted to read a stream of serialisations - I will adapt the answer though. Thanks! – Sam Liddicott Apr 29 '16 at 12:40
  • Escaping newline doesn't allow it to be preserved, it makes it go away once read. backslash-newline for read is a way to continue a logical line onto another physical line. Edit: ah I see you mention the problem with newline already. – Stéphane Chazelas Apr 29 '16 at 13:27
-2
printf 'VAR=$(cat <<\'$$VAR$$'\n%s\n'$$VAR$$'\n)' "$VAR" >./VAR.file

Another way to do it is to ensure you handle all 'hardquotes like this:

sed '"s/'"'/&"&"&/g;H;1h;$!d;g;'"s/.*/VAR='&'/" <<$$VAR$$ >./VAR.file
$VAR
$$VAR$$

Or with export:

env - "VAR=$VAR" sh -c 'export -p' >./VAR.file 

The first and second options work in any POSIX shell, assuming that the variable's value doesn't contain the string:

"\n${CURRENT_SHELLS_PID}VAR${CURRENT_SHELLS_PID}\n" 

The third option should work for any POSIX shell but may attempt to define other variables such as _ or PWD. The truth is though, that the only variables it might attempt to define are set and maintained by the shell itself - and so even if you do import export's value for any one of them - such $PWD for example - the shell will simply reset them to the correct value immediately anyway - try doing PWD=any_value and see for yourself.

And because - at least with GNU's bash - debug output is automatically safe-quoted for re-input to the shell, this works regardless of the number of 'hard-quotes in "$VAR":

 PS4= VAR=$VAR sh -cx 'VAR=$VAR' 2>./VAR.file

$VAR can later be set to the saved value in any script in which the following path is valid with:

. ./VAR.file
  • I'm not sure what you tried to write in the first command. $$ is the running shell's PID, did you get the quoting wrong and mean \$ or something? The basic approach of using a here document could be made to work, but it's tricky, not one-liner material: whatever you choose as the end marker, you have to pick something that doesn't appear in the string. – Gilles Jun 14 '14 at 23:15
  • The second command doesn't work when $VAR contains %. The third command doesn't always work with values containing multiple lines (even after adding the obviously missing double quotes). – Gilles Jun 14 '14 at 23:17
  • @Gilles - I know its the pid - I used it as a simple source of setting a unique delimiter. What do you mean by "doesn't always" exactly? And I don't understand what double quotes are missing - all of those are variable assignments. Double quotes only confuse the situation in that context. – mikeserv Jun 14 '14 at 23:30
  • @Gilles - I retract the assignment thing - that's an argument to env. I'm still curious what you mean about the multiple lines - sed deletes every line until encountering VAR= until the last - so all lines of $VAR get passed on. Can you please provide an example that breaks it? – mikeserv Jun 14 '14 at 23:52
  • Ah, apologies, the third method does work (with the quoting correction). Well, assuming the variable name (here VAR) isn't changed PWD or _ or perhaps others that some shells define. The second method requires bash; the output format from -v isn't standardized (none of dash, ksh93, mksh and zsh work). – Gilles Jun 15 '14 at 0:21
-2

Almost same but a bit different:

From your script:

#!/usr/bin/ksh 

save_var()
{

    (for ITEM in $*
    do
        LVALUE='${'${ITEM}'}'
        eval RVALUE="$LVALUE"
        echo "$ITEM=\"$RVALUE\""  
    done) >> $cfg_file
}

restore_vars()
{
    . $cfg_file
}

cfg_file=config_file
MY_VAR1="Test value 1"
MY_VAR2="Test 
value 2"

save_var MY_VAR1 MY_VAR2
MY_VAR1=""
MY_VAR2=""

restore_vars 

echo "$MY_VAR1"
echo "$MY_VAR2"

This time above is tested.

  • I can see you didn't test! The core logic works, but that's not the difficult bit. The difficult bit is quoting things properly, and you're not doing any of that. Try variables whose values contain newlines, ', *, etc. – Gilles Jun 14 '14 at 23:13
  • echo "$LVALUE=\"$RVALUE\"" is supposed to keep the newlines as well and the result in the cfg_file should be like : MY_VAR1="Line1\nLine 2" Thus when eval MY_VAR1 it will contain the new lines as well. Of course you might have problems if your stored value contain itself " char. But that could be taken care as well. – vadimbog Jun 15 '14 at 5:34
  • 1
    Btw, why to down vote something which is answering correctly the question asked here? Above works very well for me and using everywhere in my scripts? – vadimbog Jun 15 '14 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.