12

System Info

OS: OS X

bash: GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin16)

Background

I want time machine to exclude a set of directories and files from all my git/nodejs project. My project directories are in ~/code/private/ and ~/code/public/ so I'm trying to use bash looping to do the tmutil.

Issue

Short Version

If I have a calculated string variable k, how do I make it glob in or right before a for-loop:

i='~/code/public/*'
j='*.launch'
k=$i/$j # $k='~/code/public/*/*.launch'

for i in $k # I need $k to glob here
do
    echo $i
done

In the long version below, you will see k=$i/$j. So I cannot hardcode the string in the for loop.

Long Version

#!/bin/bash
exclude='
*.launch
.classpath
.sass-cache
Thumbs.db
bower_components
build
connect.lock
coverage
dist
e2e/*.js
e2e/*.map
libpeerconnection.log
node_modules
npm-debug.log
testem.log
tmp
typings
'

dirs='
~/code/private/*
~/code/public/*
'

for i in $dirs
do
    for j in $exclude
    do
        k=$i/$j # It is correct up to this line

        for l in $k # I need it glob here
        do
            echo $l
        #   Command I want to execute
        #   tmutil addexclusion $l
        done
    done
done

Output

They are not globbed. Not what I want.

~/code/private/*/*.launch                                                                                   
~/code/private/*/.DS_Store                                                                                  
~/code/private/*/.classpath                                                                                 
~/code/private/*/.sass-cache                                                                                
~/code/private/*/.settings                                                                                  
~/code/private/*/Thumbs.db                                                                                  
~/code/private/*/bower_components                                                                           
~/code/private/*/build                                                                                      
~/code/private/*/connect.lock                                                                               
~/code/private/*/coverage                                                                                   
~/code/private/*/dist                                                                                       
~/code/private/*/e2e/*.js                                                                                   
~/code/private/*/e2e/*.map                                                                                  
~/code/private/*/libpeerconnection.log                                                                      
~/code/private/*/node_modules                                                                               
~/code/private/*/npm-debug.log                                                                              
~/code/private/*/testem.log                                                                                 
~/code/private/*/tmp                                                                                        
~/code/private/*/typings                                                                                    
~/code/public/*/*.launch                                                                                    
~/code/public/*/.DS_Store                                                                                   
~/code/public/*/.classpath                                                                                  
~/code/public/*/.sass-cache                                                                                 
~/code/public/*/.settings                                                                                   
~/code/public/*/Thumbs.db                                                                                   
~/code/public/*/bower_components                                                                            
~/code/public/*/build                                                                                       
~/code/public/*/connect.lock                                                                                
~/code/public/*/coverage                                                                                    
~/code/public/*/dist                                                                                        
~/code/public/*/e2e/*.js                                                                                    
~/code/public/*/e2e/*.map                                                                                   
~/code/public/*/libpeerconnection.log                                                                       
~/code/public/*/node_modules                                                                                
~/code/public/*/npm-debug.log                                                                               
~/code/public/*/testem.log                                                                                  
~/code/public/*/tmp                                                                                         
~/code/public/*/typings
  • Single quotes stop shell interpolation in Bash, so you might try double-quoting your variable. – Thomas N Oct 6 '16 at 18:46
  • @ThomasN no, that does not work. k is a calculated string, and I need it stay that way till the loop. Please check my long version. – John Siu Oct 6 '16 at 18:48
  • @ThomasN I updated the short version to make it clearer. – John Siu Oct 6 '16 at 19:00
14

You can force another round of evaluation with eval, but that's not actually necessary. (And eval starts having serious problems the moment your file names contain special characters like $.) The problem isn't with globbing, but with the tilde expansion.

Globbing happens after variable expansion, if the variable is unquoted, as here(*):

$ x="/tm*" ; echo $x
/tmp

So, in the same vein, this is similar to what you did, and works:

$ mkdir -p ~/public/foo/ ; touch ~/public/foo/x.launch
$ i="$HOME/public/*"; j="*.launch"; k="$i/$j"
$ echo $k
/home/foo/public/foo/x.launch

But with the tilde it doesn't:

$ i="~/public/*"; j="*.launch"; k="$i/$j"
$ echo $k
~/public/*/*.launch

This is clearly documented for Bash:

The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, ...

Tilde expansion happens before variable expansion so tildes inside variables are not expanded. The easy workaround is to use $HOME or the full path instead.

(* expanding globs from variables is usually not what you want)


Another thing:

When you loop over the patterns, as here:

exclude="foo *bar"
for j in $exclude ; do
    ...

note that as $exclude is unquoted, it's both split, and also globbed at this point. So if the current directory contains something matching the pattern, it's expanded to that:

$ i="$HOME/public/foo"
$ exclude="*.launch"
$ touch $i/real.launch
$ for j in $exclude ; do           # split and glob, no match
    echo "$i"/$j ; done
/home/foo/public/foo/real.launch

$ touch ./hello.launch
$ for j in $exclude ; do           # split and glob, matches in current dir!
    echo "$i"/$j ; done
/home/foo/public/foo/hello.launch  # not the expected result

To work around this, use an array variable instead of a splitted string:

$ exclude=("*.launch")
$ exclude+=("something else")
$ for j in "${exclude[@]}" ; do echo "$i"/$j ; done
/home/foo/public/foo/real.launch
/home/foo/public/foo/something else

As an added bonus, array entries can also contain whitespace without issues with splitting.


Something similar could be done with find -path, if you don't mind what directory level the targeted files should be. E.g. to find any path ending in /e2e/*.js:

$ dirs="$HOME/public $HOME/private"
$ pattern="*/e2e/*.js"
$ find $dirs -path "$pattern"
/home/foo/public/one/two/three/e2e/asdf.js

We have to use $HOME instead of ~ for the same reason as before, and $dirs needs to be unquoted on the find command line so it gets split, but $pattern should be quoted so it isn't accidentally expanded by the shell.

(I think you could play with -maxdepth on GNU find to limit how deep the search goes, if you care, but that's a bit of a different issue.)

  • Are you the one answer with find? I actually am exploring that route too, as the for-loop is getting complicated. But I am having difficulty with the '-path'. – John Siu Oct 6 '16 at 21:02
  • Credit to you as your info about tilde '~' is more direct to the main issue. I will post the final script and explanation in another answer. But full credit to you :D – John Siu Oct 6 '16 at 21:41
  • @JohnSiu, yeah, using find was what first came to mind. It might be usable too, depending on the exact need. (or better too, for some uses.) – ilkkachu Oct 6 '16 at 21:51
  • @JohnSiu, updated again, to explicitly mention an issue I think Stéphane was pointing out in the comments to another answer – ilkkachu Oct 6 '16 at 23:14
  • 1
    @kevinarpe, I think arrays are basically meant for just that, and yes, "${array[@]}" (with the quotes!) is documented (see here and here) to expand to the elements as distinct words without splitting them further. – ilkkachu Feb 28 '18 at 12:47
3

You can save it as an array instead of a string to use later in many cases and let the globbing happen when you define it. In your case, for example:

k=(~/code/public/*/*.launch)
for i in "${k[@]}"; do

or in the later example, you'll need to eval some of the strings

dirs=(~/code/private/* ~/code/public/*)
for i in "${dirs[@]}"; do
    for j in $exclude; do
        eval "for k in $i/$j; do tmutil addexclusion \"\$k\"; done"
    done
done
  • 1
    Note how $exclude contains wildcards, you'd need to disable globbing before using the split+glob operator on it and restore it for the $i/$j and not use eval but use "$i"/$j – Stéphane Chazelas Oct 6 '16 at 20:59
  • Both you and ilkkachu give good answer. However his answer identified the issue. So credit to him. – John Siu Oct 6 '16 at 21:43
2

@ilkkachu answer solved the main globbing issue. Full credit to him.

V1

However, due to exclude containing entries both with and without wildcard(*), and also they may not exist in all, extra checking is needed after the globbing of $i/$j. I am sharing my findings here.

#!/bin/bash
exclude="
*.launch
.DS_Store
.classpath
.sass-cache
.settings
Thumbs.db
bower_components
build
connect.lock
coverage
dist
e2e/*.js
e2e/*.map
libpeerconnection.log
node_modules
npm-debug.log
testem.log
tmp
typings
"

dirs="
$HOME/code/private/*
$HOME/code/public/*
"

# loop $dirs
for i in $dirs; do
    for j in $exclude ; do
        for k in $i/$j; do
            echo -e "$k"
            if [ -f $k ] || [ -d $k ] ; then
                # Only execute command if dir/file exist
                echo -e "\t^^^ Above file/dir exist! ^^^"
            fi
        done
    done
done

Output Explaination

Following is the partial output to explain the situation.

/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/a.launch
    ^^^ Above file/dir exist! ^^^
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/b.launch
    ^^^ Above file/dir exist! ^^^
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/.DS_Store
    ^^^ Above file/dir exist! ^^^

The above are self explanatory.

/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/.classpath
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/.sass-cache
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/.settings
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/Thumbs.db
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/bower_components
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/build
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/connect.lock
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/coverage
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/dist

The above show up because the exclude entry($j) has no wildcard, $i/$j become a plain string concatenation. However the file/dir does not exist.

/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/e2e/*.js
/Volumes/HD2/JS/code/public/simple-api-example-ng2-express/e2e/*.map

The above show up as exclude entry($j) contain wildcard but has no file/directory match, the globbing of $i/$j just return the original string.

V2

V2 use single quote, eval and shopt -s nullglob to get clean result. No file/dir final checking require.

#!/bin/bash
exclude='
*.launch
.sass-cache
Thumbs.db
bower_components
build
connect.lock
coverage
dist
e2e/*.js
e2e/*.map
libpeerconnection.log
node_modules
npm-debug.log
testem.log
tmp
typings
'

dirs='
$HOME/code/private/*
$HOME/code/public/*
'

for i in $dirs; do
    for j in $exclude ; do
        shopt -s nullglob
        eval "k=$i/$j"
        for l in $k; do
            echo $l
        done
        shopt -u nullglob
    done
done
  • One problem is that in for j in $exclude, the globs in $exclude could get expanded at the time of that $exclude expansion (and calling eval on that is asking for trouble). You'd want globbing enabled for for i in $dir, and for l in $k, but not for for j in $exclude. You'd want a set -f before the latter and a set +f for the other. More generally, you'd want to tune your split+glob operator before using it. In any case, you don't want split+glob for echo $l, so $l should be quoted there. – Stéphane Chazelas Oct 7 '16 at 13:53
  • @StéphaneChazelas are you refering to v1 or v2? For v2, both exclude and dirs are in single quote(), so no globbing till eval`. – John Siu Oct 7 '16 at 14:00
  • Globbing takes place upon unquoted variable expansion in list contexts, that (leaving a variable unquoted) is what we sometimes call the split+glob operator. There's no globbing in assignments to scalar variables. foo=* and foo='*' is the same. But echo $foo and echo "$foo" are not (in shells like bash, it's been fixed in shells like zsh, fish or rc, see also the link above). Here you do want to use that operator, but in some places only the split part, and in others only the glob part. – Stéphane Chazelas Oct 7 '16 at 14:10
  • @StéphaneChazelas Thanks for the info!!! Took me sometime but I understand the concern now. This very valuable!! Thank you!!! – John Siu Oct 7 '16 at 16:13
1

With zsh:

exclude='
*.launch
.classpath
.sass-cache
Thumbs.db
...
'

dirs=(
~/code/private/*
~/code/public/*
)

for f ($^dirs/${^${=~exclude}}(N)) {
  echo $f
}

${^array}string is to expand as $array[1]string $array[2]string.... $=var is to perform word splitting on the variable (something other shells do by default!), $~var does globbing on the variable (something other shells also by default (when you generally don't want them to, you'd have had to quote $f above in other shells)).

(N) is a glob qualifier that turns on nullglob for each of those globs resulting from that $^array1/$^array2 expansion. That makes the globs expand to nothing when they don't match. That also happens to turn a non-glob like ~/code/private/foo/Thumbs.db into one, which means that if that particular doesn't exist, it's not included.

  • This is really nice. I tested and works. However, it seems zsh is more sensitive to newline when using single quote. The way exclude is enclosed is affecting the output. – John Siu Oct 7 '16 at 12:30
  • @JohnSiu, oh yes, you're right. It seems the split+glob and the $^array must be done in two separate steps to make sure the empty elements are discarded (see edit). That looks a bit like a bug in zsh, I'll raise the issue on their mailing list. – Stéphane Chazelas Oct 7 '16 at 12:52
  • I come up with a v2 for bash, which is cleaner, but still not as compact as your zsh script, lol – John Siu Oct 7 '16 at 13:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.