5

I need to write a shell script which can execute all .sh files if they exist in a directory pattern. Something like:

##!/bin/bash
sh /var/scripts/*/my_*.inc.sh

But the above script executes only just one file; the first file which happens (seems chosen by the latest modification date) which is not my point. I need all files matching the rule to be executed.

1
  • No time for a complete answer, but also suggest you look at run-parts.
    – Darren
    Apr 26, 2020 at 17:01

5 Answers 5

9

The reason that your code does not work is because of what happens when a filename globbing pattern expands.

The pattern would expand to a list of matching pathnames. The code in your script would call sh with this list. This means it would execute the first matching name as the script and give the other names as arguments to that script:

sh /var/scripts/dir/my_first.inc.sh /var/scripts/dir/my_second.inc.sh /var/scripts/dir/my_third.inc.sh

Instead, just iterate over the matching names:

#!/bin/sh

for name in /var/scripts/*/my_*.inc.sh; do
    sh "$name"
done

This assumes that each of the matching names is a script that should be executed by sh (not bash). If the individual files are executable and has a proper #!-line, then remove the sh from the invocation of the script in the loop above. If the files are "dot scripts", i.e. script that should be sourced, then replace sh by . instead to have the script execute in the current script's environment.

Note that the script above can be an sh script (#!/bin/sh) as it does not use any bash features. If the other script are "dot scripts", then you may obviously have to change this to #!/bin/bash or whatever other interpreter is needed to source the scripts.

5

A .inc.sh extension suggests those .inc.sh file should be included, or in other words, that their contents should be evaluated as shell code by one shell interpreter, so that that same shell interpreter can execute some other code with the functions, variables, aliases... defined in those files available.

bash's syntax is mostly backward compatible with the sh syntax, especially when its POSIX mode is enabled (bash is actually a sh interpreter when invoked as sh), so even though those functions are presumably written in sh language, you should still be able to have a bash interpreter interpret them.

One notable difference between bash (when not in POSIX mode) and sh is that bash doesn't enable alias expansion in scripts. That can be worked around though by setting the expand_aliases option or the posix option, so you can do:

#! /bin/bash -
set -o posix # increase POSIX sh compatibility, some but not all bash-specific
             # extensions are still available.

for file in /var/scripts/*/my_*.inc.sh; do
  . "$file"
done

# rest of the code that uses the functions/variables, etc defined
# in the files sourced above with the "." special builtin

Note that if there's no matching file, the . command will fail when trying to source a non-existing file called literally /var/scripts/*/my_*.inc.sh and exit the shell.

If instead in that case you wanted to not do anything you could do:

shopt -s nullglob
files=(/var/scripts/*/my_*.inc.sh)
shopt -u nullglob

for file in "${files[@]}"; do
  command . "$file"
done

With nullglob, globs with no match expand to the empty list instead of the unexpanded pattern. With the command prefix, failure of special builtins like . (like when $file is not readable) doesn't cause the shell to exit (and error should still be reported).

To look for .inc.sh files recursively instead of just the subdirs of /var/scripts, you could use the globstar option to enable the ** operator:

shopt -s nullglob globstar
files=(/var/scripts/**/my_*.inc.sh)
shopt -u nullglob globstar

In any case, directory and files whose name starts with . (hidden ones) are omitted. Use the dotglob option if you want to include the,

Obviously, if any of the sourced scripts calls exit or exec or has a syntax error or any of the special builtins called within fail, that will also exit the script.

3

The safest way to iterate over files in shell scripts (including bash), is to use pathname expansion (globbing) in a for loop. The correct code is:

for file in /var/scripts/*/my_*.inc.sh
do
  echo "Executing $file"
  "$file"
done

This will handle paths and filenames with special characters such as spaces, asterisks, newlinews, etc.

If you want to do recursive search, or use options that globbing is not capable of providing, find may be used with the -exec option (bellow assumes recent GNU find):

find a/ -executable -type f -name '*.sh' -exec {} \;

If find is still necessary, but you and to execute multiple commands on each file, read can be used, but you should careful with the options to handle all filenames (bellow assumes bash and GNU find):

#!/bin/bash
find a/ -executable -type f -name '*.sh' -print0 |
  while IFS= read -r -d $'\0' file
  do
    echo "Executing $file"
    "$file"
  done
3
  • The earlier claim (now edited away) that the other solutions won't work with spaces etc. in filenames seems not to apply to the xargs solution. I tested it on Ubuntu 18.04 (bash 4.4.18, findutils (xargs) 4.6.0). I don't know whether it might break on some other systems, though. Apr 26, 2020 at 7:14
  • 1
    @JaripekkaJuhala: It works for spaces but not newline characters (yes they are valid in filenames).
    – user000001
    Apr 26, 2020 at 7:15
  • Sorry, tested again. You're right, newlines won't work :( Apr 26, 2020 at 7:19
2

I don't know whether it is the best solution, but xargs can in general be used for thing like this:

ls /var/scripts/*/my_*.inc.sh | xargs -I {} sh {}

As pointed out by @user000001, this won't work for filenames with a newline character.

4
  • 1
    The problem is not limited to newline characters. single quote (apostrophe), double quote and backslash are also a problem. Also note that the stdin of those scripts is also affected (will be either /dev/null or the pipe depending on that xargs implementation). Apr 26, 2020 at 8:18
  • What if there are spaces in the script file names? Apr 26, 2020 at 16:53
  • @user000001, not really, -L 1 runs one command from each line of input, but those lines are still split into words that make up separate arguments to the command. With -I {}, there's no splitting within the lines, but leading whitespace is stripped (at least with POSIX compliant implementations, not busybox's and with the list of which characters are considered whitespace varying with the implementation). IMO, xargs is only usable with -r0 (non-standard). Apr 27, 2020 at 15:18
  • @StéphaneChazelas: Thanks, I did some tests before commenting, but they didn't include leading whitespace.
    – user000001
    Apr 27, 2020 at 15:22
0

You can use a for loop to iterate (recursively) through all shell files in a path found with find, like so:

for i in $(find /var/scripts/ -type f -name '*.sh')
  do sh "$i"
done
3

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .