9

I'd like to echo all non-environment variables (all self-declared variables), in Bash 3.2.52.

This command to print all variables gave me output I can't understand that seems to me to conflict with the set -x && clear -r mode I already work in.

diff -U 1 <(set -o posix ; set |cut -d= -f1) <(
    exec bash -ic 'set -o posix ; set' | cut -d= -f1)  |
        grep '^[-][^-]'                                |
        cut -d- -f2                                    |
        grep -vE '^(COLUMNS|HISTFILESIZE|HISTSIZE|LINES|PIPESTATUS)$'

I need an echo or printf (or any other "simpler") operation to have such list.
If possible in this version of Bash, how can this be achieved?

1
  • 1
    An unreliable solution (which is why I'm not posting it as an answer): grep -vxF -f <( declare -x -p ) <( declare -p ). It's unreliable whenever there are variables containing newlines (which there almost always are). Also does not work with the stock bash on macOS as the lines in the output of declare -p does not start with declare (while the ones from declare -x -p do).
    – Kusalananda
    Oct 14, 2019 at 7:30

8 Answers 8

5
+100

GNU Parallel includes env_parallel. Part of env_parallel lists all variables for the supported shells.

For bash this code is:

_names_of_VARIABLES() {
    compgen -A variable
}
_bodies_of_VARIABLES() {
    typeset -p "$@"
}

So given all variables we need to figure out which ones are set by the user. We can do that by hardcoding all variables set by a given version of bash, but that would not be very future proof, because bash may set more variables in future versions (Is $BASH_MYVAR set by the user or by a future version of bash?).

So instead env_parallel asks you to define a clean environment, and run env_parallel --session in that. This will set $PARALLEL_IGNORED_NAMES by listing the names defined before (using the code above).

When later run in an environment where the user has set variables, it is easy to take the difference between the clean environment and the current environment.

env_parallel --session makes it possible to define a clean environment for every session, but if you prefer to have a reference environment that can be used across sessions, simply save the list of variables to a file. So:

# In clean envronment
compgen -A variable > ~/.clean-env-vars

# In environment with user defined vars
compgen -A variable | grep -Fxvf ~/.clean-env-vars

For example:

#!/bin/bash

# In clean envronment
compgen -A variable > ~/.clean-env-vars

myvar=4
yourvar=3

# In environment with user defined vars
compgen -A variable | grep -Fxvf ~/.clean-env-vars

# This prints:
# myvar
# yourvar
# PIPESTATUS (which is set when running a pipe)
2
  • I am not sure I have enough knowledge to understand the answer at the moment, but in a test WSL machine, when I ran the last command, I didn't get the values of two variables I declared: x=1 and y=2. Instead, I Got: + compgen -A variable + grep --color=auto -Fxvf /root/.clean-env-vars
    – user149572
    Oct 24, 2019 at 14:58
  • @JohnDoea I tested the example on WSL, and I get the output shown. Not sure why you get a different output. It sounds as if you bash is configured in a special way.
    – Ole Tange
    Oct 25, 2019 at 12:48
2
+250

Add this to your ~/.bashrc

pre () {
    set -o posix;set > /tmp/tmp-original-envs
}

post () {
    set -o posix;set > /tmp/tmp-new-envs
    diff /tmp/tmp-original-envs /tmp/tmp-new-envs | grep -v BASH_LINENO | grep -v FUNCNAME | grep '>'
}

Then run pre before your modifications and post afterward

Example:

[localhost]$ pre
[localhost]$ test=new
[localhost]$ post
> test=new
[localhost]$ 
3
  • Hello. I don't understand what you meant by modifications. Thanks anyway,
    – user149572
    Nov 21, 2019 at 7:28
  • @JohnDoea what i meant is that the pre command is ran on a clean environment to record its state (without non-environment variables), then you can run your application and or declaration of your new variables, afterward the post command find out the new non-environment variable that were declared by comparing the current state to the old one... right now there is no native feature for this purpose, most answer to this question do compare 2 states of bash env variables to keep the solution simple.
    – intika
    Nov 21, 2019 at 16:05
  • Other advanced solutions that i am thinking of to this question, could be patching bash to add the feature, or develop a library that would use LD_PRELOAD to catch the changes while it occur... but those require some development work to be achieved.
    – intika
    Nov 21, 2019 at 16:15
0

Depending on your needs, one simple solution could be:

  1. Save current variables before your modifications with set > tmp or set > /tmp/current
  2. Do you modifications/run your application
  3. Extract what was modified with diff -u <(cat tmp) <(set) or
diff -u <(cat tmp) <(set) | tail -n +3 | grep -v PIPESTATUS | grep -v "_=" | grep -E "^\+"

Where:

tail -n +3: remove the first unneeded line of diff command

grep -v PIPESTATUS | grep -v "_=": remove unneeded variables

grep -E "^\+": display only required/needed changes

Alternative:

diff -u <(set -o posix; set) <(bash -lc 'set -o posix; set') | grep -E "^\+"

Note:

In your script note that exec replace completely the the process it is launched from this may interfere with what you are trying to achieve

1
  • Hello, thank you; I have defined two variables: x=1 and y=2. After I ran the above command and got a huge output, not just a list of variables...
    – user149572
    Oct 24, 2019 at 14:55
0

I'd like to echo all self-declared variables (all non-environment variables)

This is a task for comm, as to me:

comm -23 <( set | cut -d= -f1 | sort ) <( env | cut -d= -f1 | sort )

-23 means comm should omit printing columns 2 and 3.

3rd is such entries that are found in both lists and 2nd is entries that are only in env's post-processed output.

Or, IOW, show only unique entries that were in 1st list.

P. S. Since some entries can be found in both lists at once (check them with comm -12 …), you could also want to include them into resulting set. It means you shouldn't use -3 in that case and lines from 3rd column would be shown shifted when printed.

6
  • Seems to break if there are functions set (I get a lot of lines with { and }).
    – Ole Tange
    Oct 19, 2019 at 8:52
  • It wasn't my intent to handle "functions" case but rather to show how differences between 2 lists can be isolated in a handy way.
    – poige
    Oct 19, 2019 at 9:05
  • 1
    To get just variables invoke set in POSIX mode: set -o posix ; set ; set +o posix. You'll also want to grep for an = before you cut - so something like comm -23 <( set -o posix; set|grep =| cut -d= -f1 | sort ) <( env | cut -d= -f1 | sort ) ... maybe
    – Rich
    Oct 21, 2019 at 19:10
  • Thanks for adding, that might be helpful.
    – poige
    Oct 22, 2019 at 3:54
  • Hello, thank you; I have defined two variables: x=1 and y=2. After I ran the above command and got a huge output, not just a list of variables...
    – user149572
    Oct 24, 2019 at 14:55
0

Kludgy, bash-only solution:

(setn(){
   set | sed '/^{/,/^}/d;/=/!d;/^BASH[A-Z_]*=\|^PPID=\|^SHLVL=\|^PIPESTATUS=\|^HISTSIZE=\|^HISTFILESIZE=\|^_[a-z_]*=/d' | sort
}
export -f setn
comm -23 <(setn) <("$BASH" -ic setn))

You may have to also filter out other variables via the sed (or make the BASH.. pattern more strict). The /_[a-z_]*=/ is supposed to exclude variables defined by the programmable completion like _backup_glob or _xspecs; that too could be made more strict.

bash will print the values of variables which contain funny characters in the $'...' format, so you'll have to handle that:

foo='
bar
baz
'
...
foo=$'\nbar\nbaz\n'

bash will not create shell variables from envvars whose names contain newlines, so there's no way to break this hack by such a trick.

4
  • Seems not to work if user sets BASHMYVAR=1 (which is not a bash variable).
    – Ole Tange
    Oct 19, 2019 at 8:51
  • Already noticed in "you may want to .. make the BASH pattern more strict".
    – user313992
    Oct 19, 2019 at 8:55
  • Hello, thank you; I have defined two variables: x=1 and y=2. After I ran the above command and got a huge output, not just a list of variables...
    – user149572
    Oct 24, 2019 at 14:56
  • I guess you have the programmable completion turned on; that's defining a lot of functions (and some variables), and when in posix mode, set in bash will also print the source of the functions.
    – user313992
    Oct 25, 2019 at 22:45
0

It seems you had some problems with the solution that uses compgen -A variable. It might be because of set -x you mentioned in the question body, I don't know.


Without compgen now. Let's start with this comment. It says:

grep -vxF -f <( declare -x -p ) <( declare -p )

I partially improve it to

grep -vxF -f <( declare -x -p | grep -o '[^ =]*=') <( declare -p | grep -o '[^ =]*=')

There are few problems now:

  1. A variable that contains = in its content will make the above code report false positives. Example:

    foo='123
    bar=baz
    =xyz'
    

    Run the code. It will print (among others) foo=, bar= and =. To fix this we should test each line if such variable really exists.

  2. Or we can get duplicates:

    foo='123
    aaa foo=456
    bar=baz
    bar=qux'
    

    To fix this we can use sort -u. I prefer LC_COLLATE=POSIX sort -u to get lowercase variables at the end.

  3. The above foo, if exported, can "mask" legitimate (not exported) bar. Example:

    export foo
    bar=999
    

    Now the code will not print bar=. To fix this we should not exclude basing on "global" declare -x -p. For each variable we should separately check if it's not exported. This step makes the preliminary grep -vxF unnecessary.

So this:

declare -p | grep -o '[^ =]*=' | LC_COLLATE=POSIX sort -u \
   | while IFS='=' read -r v ; do
      [ -v "$v" ] \
      && bash -c "[ ! -v '$v' ]" \
      && printf '%s\n' "$v"
done

Note I do not strictly check if a variable is exported. I check if it is set in a child bash. There are variables specific to interactive shell that will be reported, you may want to filter them out manually. Or you can run bash -ic …; but multiple times, so it will be even slower.

Since we have a reliable(?) way to get a list of all variables without false positives, we can go back to the original idea of grep -vxF and only run a child bash once. The following snippet works quite well with bash -i. In my case the only variables to manually filter out are COLUMNS and LINES.

( set +x
function _get_vars {
declare -p | grep -o '[^ =]*=' | LC_COLLATE=POSIX sort -u \
   | while IFS='=' read -r v ; do
      [ -v "$v" ] \
      && printf '%s\n' "$v"
   done
}
export -f _get_vars

_get_vars | grep -vxF -f <( bash -ic '_get_vars' ) \
| grep -vE '^(COLUMNS|LINES)$' )

I deliberately use set +x for the subshell just in case; the subshell is also useful to keep the function "local" (it will not be available in your current shell). You may prefer a non-interactive child bash and more filtering. Adjust this to your needs.


Maybe, just maybe set +x in the right place will make the compgen approach work for you. How about this?

( set +x
compgen -A variable | grep -vxF -f <( bash -ic 'compgen -A variable' ) \
| grep -vE '^(COLUMNS|LINES)$' )

Disclaimer: not tested in Bash 3.2.52; tested in 4.4.12.

0

I think this solution may get you along the road to listing all non-environment variables. It definitely handles variables defined like ITEM=$'VALUE\nSTRANGE=QUARK' and ALSO=$'VALUE\ndeclare -r LANG=fake', i.e. with an embedded newline.

for item in $(declare -p | awk '$2 !~ /[aAx]/ && $3 ~ /.=/ {print substr($3,1,match($3,"=")-1)}')
do
    if [[ -n "$item" ]] && ! { env -0 | grep -q $'\0'"${item}="; }
    then
        printf ">> %s = %s <<\n" "$item" "${!item}"
        # printf "%s\n" "$item"
    fi
done

You can tweak the printf statement to print just the variable names, add them to an array, or whatever else you want.

0

my bash version is 3.2.57

I use set +o posix , this change the behavior to display contens for each variables in one line

to filter i use sort| uniq -c , and keep lines with a value of 1 l

bash-3.2$ bar=" foobar= "
bash-3.2$ foo="0
> 1
> 2 "

now i i have 2 variables foo bar

bash-3.2$ A=$(  (  ( set +o posix ; set |cut -d= -f1 ) ; 
                   ( bash -ic 'set +o posix ; set |cut -d= -f1' )  
                ) | sort | uniq -c | sed 's/^ *1 //p;d' | 
               grep -vE '^(COLUMNS|HISTFILESIZE|HISTSIZE|LINES|PIPESTATUS|BASH_EXECUTION_STRING)$' )
bash-3.2$ echo $A
bar foo

You must log in to answer this question.