12

Is there any way to serialize a shell variable? Suppose I have a variable $VAR, and I want to be able to save it to a file or whatever, and then read it back later to get the same value back?

Is there a portable way of doing this? (I don't think so)

Is there a way to do it in bash or zsh?

2
  • 2
    Attention: the version of my answer you accepted the other day had a serious issue that will break in some scenarios. I have re-written it to include fixes (and add features) and you should really re-read it from scratch and port your code to use the fixed version.
    – Caleb
    Jun 16, 2014 at 13:06
  • ^Another^ example of @Caleb's upstanding citizenship.
    – mikeserv
    Jun 19, 2014 at 19:56

6 Answers 6

15

Warning: With any of these solutions, you need to be aware that you are trusting the integrity of the data files to be safe as they will get executed as shell code in your script. Securing them is paramount to your script's security!

Simple inline implementation for serializing one or more variables

Yes, in both bash and zsh you can serialize the contents of a variable in a way that is easy to retrieve using the typeset builtin and the -p argument. The output format is such that you can simply source the output to get your stuff back.

 # You have variable(s) $FOO and $BAR already with your stuff
 typeset -p FOO BAR > ./serialized_data.sh

You can get your stuff back like this either later in your script or in another script altogether:

# Load up the serialized data back into the current shell
source serialized_data.sh

This will work for bash, zsh and ksh including passing data between different shells. Bash will translate this to its builtin declare function while zsh implements this with typeset but as bash has an alias for this to work either way for we use typeset here for ksh compatibility.

More complex generalized implementation using functions

The above implementation is really simple, but if you call it frequently you might want to give yourself a utility function to make it easier. Additionally if you ever try to include the above inside custom functions you will run into issues with variable scoping. This version should eliminate those issues.

Note for all of these, in order to maintain bash/zsh cross-compatibility we will be fixing both the cases of typeset and declare so the code should work in either or both shells. This adds some bulk and mess that could be eliminated if you were only doing this for one shell or another.

The main problem with using functions for this (or including the code in other functions) is that the typeset function generates code that, when sourced back into a script from inside a function, defaults to creating a local variable rather than a global one.

This can be fixed with one of several hacks. My initial attempt to to fix this was parse the output of the serialize process through sed to add the -g flag so the created code defines a global variable when sourced back in.

serialize() {
    typeset -p "$1" | sed -E '0,/^(typeset|declare)/{s/ / -g /}' > "./serialized_$1.sh"
}
deserialize() {
    source "./serialized_$1.sh"
}

Note that the funky sed expression is to only match the first occurrence of either 'typeset' or 'declare' and add -g as a first argument. It is necessary to only match the first occurrence because, as Stéphane Chazelas rightly pointed out in comments, otherwise it will also match cases where the serialized string contains literal newlines followed by the word declare or typeset.

In addition to correcting my initial parsing faux pas, Stéphane also suggested a less brittle way to hack this that not only side steps the issues with parsing the strings but could be a useful hook to add additional functionality by using a wrapper function to redefine the actions taken when sourcing the data back in. This assumes you are not playing any other games with the declare or typeset commands, but this technique would be easier to implement in a situation where you were including this functionality as part of another function of your own or you were not in control of the data being written and whether or not it had the -g flag added. Something similar could also be done with aliases, see Gilles's answer for an implementation.

To make the result even more useful, we can iterate over multiple variables passed to our functions by assuming that each word in the argument array is a variable name. The result becomes something like this:

serialize() {
    for var in $@; do
        typeset -p "$var" > "./serialized_$var.sh"
    done
}

deserialize() {
    declare() { builtin declare -g "$@"; }
    typeset() { builtin typeset -g "$@"; }
    for var in $@; do
        source "./serialized_$var.sh"
    done
    unset -f declare typeset
}

With either solution, usage would look like this:

# Load some test data into variables
FOO=(an array or something)
BAR=$(uptime)

# Save it out to our serialized data files
serialize FOO BAR

# For testing purposes unset the variables to we know if it worked
unset FOO BAR

# Load  the data back in from out data files
deserialize FOO BAR

echo "FOO: $FOO\nBAR: $BAR"
8
  • declare is the bash equivalent of ksh's typeset. bash, zsh also support typeset so in that regard, typeset is more portable. export -p is POSIX, but it doesn't take any argument and its output is shell dependent (though it's well specified for POSIX shells, so for instance when bash or ksh is called as sh). Remember to quote your variables; using the split+glob operator here doesn't make sense. Jun 15, 2014 at 16:04
  • Note that -E is only found in some BSDs sed. Variable values may contain newline characters, so the sed 's/^.../.../' is not guaranteed to work correctly. Jun 15, 2014 at 16:07
  • This is exactly what I was looking for! I wanted a convenient way to push variables back and forth between shells.
    – fwenom
    Jun 15, 2014 at 16:12
  • I meant: a=$'foo\ndeclare bar' bash -c 'declare -p a' for install will output a line that starts with declare. It's probably better to do declare() { builtin declare -g "$@"; } before calling source (and unset afterwards) Jun 16, 2014 at 10:46
  • 2
    @Gilles, aliases wouldn't work inside functions (need to be defined at the time of the function definition), and with bash that would mean you'd need to do a shopt -s expandalias when not interactive. With functions, you could also enhance the declare wrapper so it only restores the variables you specify. Jun 16, 2014 at 11:26
3

Use redirection, command substitution, and parameter expansion. Double quotes are needed to preserve whitespace and special characters. The trailing x saves the trailing newlines which would be otherwise removed in the command substitution.

#!/bin/bash
echo "$var"x > file
unset var
var="$(< file)"
var=${var%x}
1
  • He probably wants to save the variable name also to the file.
    – user80551
    Jun 14, 2014 at 11:44
2

Serialize all — POSIX

In any POSIX shell, you can serialize all environment variables with export -p. This doesn't include non-exported shell variables. The output is properly quoted so that you can read it back in the same shell and get exactly the same variable values. The output might not be readable in another shell, for example ksh uses the non-POSIX $'…' syntax.

save_environment () {
  export -p >my_environment
}
restore_environment () {
  . ./my_environment
}

Serialize some or all — ksh, bash, zsh

Ksh (both pdksh/mksh and ATT ksh), bash and zsh provide a better facility with the typeset builtin. typeset -p prints out all defined variables and their values (zsh omits the values of variables that have been hidden with typeset -H). The output contains proper declaration so that environment variables are exported when read back (but if a variable is already exported when read back, it won't be unexported), so that arrays are read back as arrays, etc. Here also, the output is properly quoted but is only guaranteed to be readable in the same shell. You can pass a set of variables to serialize on the command line; if you don't pass any variable then all are serialized.

save_some_variables () {
  typeset -p VAR OTHER_VAR >some_vars
}

In bash and zsh, restoring can't be done from a function because typeset statements inside a function are scoped to that function. You need to run . ./some_vars in the context where you want to use the variables' values, taking care that variables that were global when exported will be redeclared as global. If you want to read back the values within a function and export them, you can declare a temporary alias or function. In zsh:

restore_and_make_all_global () {
  alias typeset='typeset -g'
  . ./some_vars
  unalias typeset
}

In bash (which uses declare rather than typeset):

restore_and_make_all_global () {
  alias declare='declare -g'
  shopt -s expand_aliases
  . ./some_vars
  unalias declare
}

In ksh, typeset declares local variables in functions defined with function function_name { … } and global variables in functions defined with function_name () { … }.

Serialize some — POSIX

If you want more control, you can export the content of a variable manually. To print the content of a variable exactly into a file, use the printf builtin (echo has a few special cases such as echo -n on some shells and adds a newline):

printf %s "$VAR" >VAR.content

You can read this back with $(cat VAR.content), except that the command substitution strips off trailing newlines. To avoid this wrinkle, arrange for the output not to end with a newline ever.

VAR=$(cat VAR.content && echo a)
if [ $? -ne 0 ]; then echo 1>&2 "Error reading back VAR"; exit 2; fi
VAR=${VAR%?}

If you want to print multiple variables, you can quote them with single quotes, and replace all embedded single quotes with '\''. This form of quoting can be read back into any Bourne/POSIX-style shell. The following snippet works in any POSIX shell. It only works for string variables (and numeric variables in shells that have them, though they'll be read back as strings), it doesn't try to deal with array variables in shells that have them.

serialize_variables () {
  for __serialize_variables_x do
    eval "printf $__serialize_variables_x=\\'%s\\'\\\\n \"\$${__serialize_variables_x}\"" |
    sed -e "s/'/'\\\\''/g" -e '1 s/=.../=/' -e '$ s/...$//'
  done
}

Here's another approach which doesn't fork a subprocess but is heavier on string manipulation.

serialize_variables () {
  for __serialize_variables_var do
    eval "__serialize_variables_tail=\${$__serialize_variables_var}"
    while __serialize_variables_quoted="$__serialize_variables_quoted${__serialize_variables_tail%%\'*}"
          [ "${__serialize_variables_tail%%\'*}" != "$__serialize_variables_tail" ]; do
      __serialize_variables_tail="${__serialize_variables_tail#*\'}"
      __serialize_variables_quoted="${__serialize_variables_quoted}'\\''"
    done
    printf "$__serialize_variables_var='%s'\n" "$__serialize_variables_quoted"
  done
}

Note that on shells that allow read-only variables, you'll get an error if you try to read back a variable that's read-only.

3
  • This brings in variables such as $PWD and $_ - please see your own comments below.
    – mikeserv
    Jun 15, 2014 at 22:17
  • @Caleb How about making typeset an alias for typeset -g? Jun 16, 2014 at 11:20
  • @Gilles I thought of that after Stephanie suggested the function method, but I wasn't sure how to portably set the necessary alias expand options across shells. Maybe you could put that in your answer as a viable alternative to the function one I included.
    – Caleb
    Jun 16, 2014 at 11:28
0

You could use base64:

$ VAR="1/ 
,x"
$ echo "$VAR" | base64 > f
$ VAR=$(cat f | base64 -d)
$ echo "${VAR}X"
1/ 
,xX
-2
printf 'VAR=$(cat <<\'$$VAR$$'\n%s\n'$$VAR$$'\n)' "$VAR" >./VAR.file

Another way to do it is to ensure you handle all 'hardquotes like this:

sed '"s/'"'/&"&"&/g;H;1h;$!d;g;'"s/.*/VAR='&'/" <<$$VAR$$ >./VAR.file
$VAR
$$VAR$$

Or with export:

env - "VAR=$VAR" sh -c 'export -p' >./VAR.file 

The first and second options work in any POSIX shell, assuming that the variable's value doesn't contain the string:

"\n${CURRENT_SHELLS_PID}VAR${CURRENT_SHELLS_PID}\n" 

The third option should work for any POSIX shell but may attempt to define other variables such as _ or PWD. The truth is though, that the only variables it might attempt to define are set and maintained by the shell itself - and so even if you do import export's value for any one of them - such $PWD for example - the shell will simply reset them to the correct value immediately anyway - try doing PWD=any_value and see for yourself.

And because - at least with GNU's bash - debug output is automatically safe-quoted for re-input to the shell, this works regardless of the number of 'hard-quotes in "$VAR":

 PS4= VAR=$VAR sh -cx 'VAR=$VAR' 2>./VAR.file

$VAR can later be set to the saved value in any script in which the following path is valid with:

. ./VAR.file
11
  • I'm not sure what you tried to write in the first command. $$ is the running shell's PID, did you get the quoting wrong and mean \$ or something? The basic approach of using a here document could be made to work, but it's tricky, not one-liner material: whatever you choose as the end marker, you have to pick something that doesn't appear in the string. Jun 14, 2014 at 23:15
  • The second command doesn't work when $VAR contains %. The third command doesn't always work with values containing multiple lines (even after adding the obviously missing double quotes). Jun 14, 2014 at 23:17
  • @Gilles - I know its the pid - I used it as a simple source of setting a unique delimiter. What do you mean by "doesn't always" exactly? And I don't understand what double quotes are missing - all of those are variable assignments. Double quotes only confuse the situation in that context.
    – mikeserv
    Jun 14, 2014 at 23:30
  • @Gilles - I retract the assignment thing - that's an argument to env. I'm still curious what you mean about the multiple lines - sed deletes every line until encountering VAR= until the last - so all lines of $VAR get passed on. Can you please provide an example that breaks it?
    – mikeserv
    Jun 14, 2014 at 23:52
  • Ah, apologies, the third method does work (with the quoting correction). Well, assuming the variable name (here VAR) isn't changed PWD or _ or perhaps others that some shells define. The second method requires bash; the output format from -v isn't standardized (none of dash, ksh93, mksh and zsh work). Jun 15, 2014 at 0:21
-2

Almost same but a bit different:

From your script:

#!/usr/bin/ksh 

save_var()
{

    (for ITEM in $*
    do
        LVALUE='${'${ITEM}'}'
        eval RVALUE="$LVALUE"
        echo "$ITEM=\"$RVALUE\""  
    done) >> $cfg_file
}

restore_vars()
{
    . $cfg_file
}

cfg_file=config_file
MY_VAR1="Test value 1"
MY_VAR2="Test 
value 2"

save_var MY_VAR1 MY_VAR2
MY_VAR1=""
MY_VAR2=""

restore_vars 

echo "$MY_VAR1"
echo "$MY_VAR2"

This time above is tested.

3
  • I can see you didn't test! The core logic works, but that's not the difficult bit. The difficult bit is quoting things properly, and you're not doing any of that. Try variables whose values contain newlines, ', *, etc. Jun 14, 2014 at 23:13
  • echo "$LVALUE=\"$RVALUE\"" is supposed to keep the newlines as well and the result in the cfg_file should be like : MY_VAR1="Line1\nLine 2" Thus when eval MY_VAR1 it will contain the new lines as well. Of course you might have problems if your stored value contain itself " char. But that could be taken care as well.
    – vadimbog
    Jun 15, 2014 at 5:34
  • 1
    Btw, why to down vote something which is answering correctly the question asked here? Above works very well for me and using everywhere in my scripts?
    – vadimbog
    Jun 15, 2014 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.