How to list all non-environment variables in bash?

Question

I'd like to echo all non-environment variables (all self-declared variables), in Bash 3.2.52.

This command to print all variables gave me output I can't understand that seems to me to conflict with the set -x && clear -r mode I already work in.

diff -U 1 <(set -o posix ; set |cut -d= -f1) <(
    exec bash -ic 'set -o posix ; set' | cut -d= -f1)  |
        grep '^[-][^-]'                                |
        cut -d- -f2                                    |
        grep -vE '^(COLUMNS|HISTFILESIZE|HISTSIZE|LINES|PIPESTATUS)$'

I need an echo or printf (or any other "simpler") operation to have such list.
If possible in this version of Bash, how can this be achieved?

Answer 1

GNU Parallel includes env_parallel. Part of env_parallel lists all variables for the supported shells.

For bash this code is:

_names_of_VARIABLES() {
    compgen -A variable
}
_bodies_of_VARIABLES() {
    typeset -p "$@"
}

So given all variables we need to figure out which ones are set by the user. We can do that by hardcoding all variables set by a given version of bash, but that would not be very future proof, because bash may set more variables in future versions (Is $BASH_MYVAR set by the user or by a future version of bash?).

So instead env_parallel asks you to define a clean environment, and run env_parallel --session in that. This will set $PARALLEL_IGNORED_NAMES by listing the names defined before (using the code above).

When later run in an environment where the user has set variables, it is easy to take the difference between the clean environment and the current environment.

env_parallel --session makes it possible to define a clean environment for every session, but if you prefer to have a reference environment that can be used across sessions, simply save the list of variables to a file. So:

# In clean envronment
compgen -A variable > ~/.clean-env-vars

# In environment with user defined vars
compgen -A variable | grep -Fxvf ~/.clean-env-vars

For example:

#!/bin/bash

# In clean envronment
compgen -A variable > ~/.clean-env-vars

myvar=4
yourvar=3

# In environment with user defined vars
compgen -A variable | grep -Fxvf ~/.clean-env-vars

# This prints:
# myvar
# yourvar
# PIPESTATUS (which is set when running a pipe)

Answer 2

Add this to your ~/.bashrc

pre () {
    set -o posix;set > /tmp/tmp-original-envs
}

post () {
    set -o posix;set > /tmp/tmp-new-envs
    diff /tmp/tmp-original-envs /tmp/tmp-new-envs | grep -v BASH_LINENO | grep -v FUNCNAME | grep '>'
}

Then run pre before your modifications and post afterward

Example:

[localhost]$ pre
[localhost]$ test=new
[localhost]$ post
> test=new
[localhost]$ 

Answer 3

Depending on your needs, one simple solution could be:

1. Save current variables before your modifications with set > tmp or set > /tmp/current
2. Do you modifications/run your application
3. Extract what was modified with diff -u <(cat tmp) <(set) or

diff -u <(cat tmp) <(set) | tail -n +3 | grep -v PIPESTATUS | grep -v "_=" | grep -E "^\+"

Where:

tail -n +3: remove the first unneeded line of diff command

grep -v PIPESTATUS | grep -v "_=": remove unneeded variables

grep -E "^\+": display only required/needed changes

Alternative:

diff -u <(set -o posix; set) <(bash -lc 'set -o posix; set') | grep -E "^\+"

Note:

In your script note that exec replace completely the the process it is launched from this may interfere with what you are trying to achieve

Answer 4

I'd like to echo all self-declared variables (all non-environment variables)

This is a task for comm, as to me:

comm -23 <( set | cut -d= -f1 | sort ) <( env | cut -d= -f1 | sort )

-23 means comm should omit printing columns 2 and 3.

3rd is such entries that are found in both lists and 2nd is entries that are only in env's post-processed output.

Or, IOW, show only unique entries that were in 1st list.

P. S. Since some entries can be found in both lists at once (check them with comm -12 …), you could also want to include them into resulting set. It means you shouldn't use -3 in that case and lines from 3rd column would be shown shifted when printed.

Answer 5

Kludgy, bash-only solution:

(setn(){
   set | sed '/^{/,/^}/d;/=/!d;/^BASH[A-Z_]*=\|^PPID=\|^SHLVL=\|^PIPESTATUS=\|^HISTSIZE=\|^HISTFILESIZE=\|^_[a-z_]*=/d' | sort
}
export -f setn
comm -23 <(setn) <("$BASH" -ic setn))

You may have to also filter out other variables via the sed (or make the BASH.. pattern more strict). The /_[a-z_]*=/ is supposed to exclude variables defined by the programmable completion like _backup_glob or _xspecs; that too could be made more strict.

bash will print the values of variables which contain funny characters in the $'...' format, so you'll have to handle that:

foo='
bar
baz
'
...
foo=$'\nbar\nbaz\n'

bash will not create shell variables from envvars whose names contain newlines, so there's no way to break this hack by such a trick.

Answer 6

It seems you had some problems with the solution that uses compgen -A variable. It might be because of set -x you mentioned in the question body, I don't know.

---

Without compgen now. Let's start with this comment. It says:

grep -vxF -f <( declare -x -p ) <( declare -p )

I partially improve it to

grep -vxF -f <( declare -x -p | grep -o '[^ =]*=') <( declare -p | grep -o '[^ =]*=')

There are few problems now:

1. A variable that contains = in its content will make the above code report false positives. Example:

   foo='123
   bar=baz
   =xyz'
   

   Run the code. It will print (among others) foo=, bar= and =. To fix this we should test each line if such variable really exists.

2. Or we can get duplicates:

   foo='123
   aaa foo=456
   bar=baz
   bar=qux'
   

   To fix this we can use sort -u. I prefer LC_COLLATE=POSIX sort -u to get lowercase variables at the end.

3. The above foo, if exported, can "mask" legitimate (not exported) bar. Example:

   export foo
   bar=999
   

   Now the code will not print bar=. To fix this we should not exclude basing on "global" declare -x -p. For each variable we should separately check if it's not exported. This step makes the preliminary grep -vxF unnecessary.

So this:

declare -p | grep -o '[^ =]*=' | LC_COLLATE=POSIX sort -u \
   | while IFS='=' read -r v ; do
      [ -v "$v" ] \
      && bash -c "[ ! -v '$v' ]" \
      && printf '%s\n' "$v"
done

Note I do not strictly check if a variable is exported. I check if it is set in a child bash. There are variables specific to interactive shell that will be reported, you may want to filter them out manually. Or you can run bash -ic …; but multiple times, so it will be even slower.

Since we have a reliable(?) way to get a list of all variables without false positives, we can go back to the original idea of grep -vxF and only run a child bash once. The following snippet works quite well with bash -i. In my case the only variables to manually filter out are COLUMNS and LINES.

( set +x
function _get_vars {
declare -p | grep -o '[^ =]*=' | LC_COLLATE=POSIX sort -u \
   | while IFS='=' read -r v ; do
      [ -v "$v" ] \
      && printf '%s\n' "$v"
   done
}
export -f _get_vars

_get_vars | grep -vxF -f <( bash -ic '_get_vars' ) \
| grep -vE '^(COLUMNS|LINES)$' )

I deliberately use set +x for the subshell just in case; the subshell is also useful to keep the function "local" (it will not be available in your current shell). You may prefer a non-interactive child bash and more filtering. Adjust this to your needs.

---

Maybe, just maybe set +x in the right place will make the compgen approach work for you. How about this?

( set +x
compgen -A variable | grep -vxF -f <( bash -ic 'compgen -A variable' ) \
| grep -vE '^(COLUMNS|LINES)$' )

---

^{Disclaimer: not tested in Bash 3.2.52; tested in 4.4.12.}

Answer 7

I think this solution may get you along the road to listing all non-environment variables. It definitely handles variables defined like ITEM=$'VALUE\nSTRANGE=QUARK' and ALSO=$'VALUE\ndeclare -r LANG=fake', i.e. with an embedded newline.

for item in $(declare -p | awk '$2 !~ /[aAx]/ && $3 ~ /.=/ {print substr($3,1,match($3,"=")-1)}')
do
    if [[ -n "$item" ]] && ! { env -0 | grep -q $'\0'"${item}="; }
    then
        printf ">> %s = %s <<\n" "$item" "${!item}"
        # printf "%s\n" "$item"
    fi
done

You can tweak the printf statement to print just the variable names, add them to an array, or whatever else you want.

Answer 8

my bash version is 3.2.57

I use set +o posix , this change the behavior to display contens for each variables in one line

to filter i use sort| uniq -c , and keep lines with a value of 1 l

bash-3.2$ bar=" foobar= "
bash-3.2$ foo="0
> 1
> 2 "

now i i have 2 variables foo bar

bash-3.2$ A=$(  (  ( set +o posix ; set |cut -d= -f1 ) ; 
                   ( bash -ic 'set +o posix ; set |cut -d= -f1' )  
                ) | sort | uniq -c | sed 's/^ *1 //p;d' | 
               grep -vE '^(COLUMNS|HISTFILESIZE|HISTSIZE|LINES|PIPESTATUS|BASH_EXECUTION_STRING)$' )
bash-3.2$ echo $A
bar foo