How to recursively source shell scripts?

Question

This question is, if it matters, preferably in the context of Bash, but I would appreciate a cross-language solution.

The problem I'm encountering is in the context of "sourcing" a script with another, but doing this recursively. The problem is that it seems to do it infinitely.

The specific case is as follows. I've created a simple bash script to load other bash scripts in the same directory. Since in my project all scripts are in the same directory, this is OK for now.

So the "script loader" is as follows (a file _source_script.sh which defines the following function):

_source_script()
{
    # Define a few colors for the possible error messages below.
    RED=$(tput setaf 1)
    NORMAL=$(tput sgr0)

    EXPECTED_DIR="scripts"

    if [ "$#" -ge  "1" ]
    then
        EXPECTED_DIR=$1
    fi

    # Based on: http://stackoverflow.com/a/1371283/3924118
    CURRENT_DIR=${PWD##*/}

    if [ "$CURRENT_DIR" = "$EXPECTED_DIR" ]
    then
        for (( arg = 2; arg <= $#; arg++ ))
        do
            # Based on:
            # - http://askubuntu.com/questions/306851/how-to-import-a-variable-from-a-script
            # - http://unix.stackexchange.com/questions/114300/whats-the-meaning-of-a-dot-before-a-command-in-shell
            # - http://stackoverflow.com/questions/20094271/bash-using-dot-or-source-calling-another-script-what-is-difference

            printf ". ./${!arg}.sh\n"
            # Try to load script ${!arg}.sh
            . ./${!arg}.sh

            # If it was not loaded successfully, exit with status 1.
            if [ $? -ne 0 ]
            then
                printf "${RED}Script '${!arg}.sh' not loaded successfully. Exiting...${NORMAL}\n"
                exit 1
            fi
        done
    else
        printf "No script loaded: $CURRENT_DIR != $EXPECTED_DIR.\n"
        exit 1
    fi
}

To use this "script loader", other scripts must first "source" it as follows:

. ./_source_script.sh

The problem is when I try to include other scripts by sourcing them through the function _source_script.  For example, when I tried to do (in script some_script.sh):

_source_script scripts colors asserts clean_environment

it kept on running forever.

Inside colors.sh I have:

#!/usr/bin/env bash

# Colors used when printing.
export GREEN=$(tput setaf 2)
export RED=$(tput setaf 1)
export NORMAL=$(tput sgr0)
export YELLOW=$(tput setaf 3)

Inside asserts.sh I have:

...

. ./_source_script.sh
_source_script scripts colors

and inside clean_environment.sh I also have:

. ./_source_script.sh
_source_script scripts colors

From my understanding, this should run recursively until it finds either a script which does not load anything or it finds a cycle, which isn't the case.

So, my solution was to have in some_script.sh the following:

_source_script scripts colors 
_source_script scripts asserts
_source_script scripts clean_environment

that is, run them individually.

So, why can't I "source" multiple scripts in a loop?

Answer 1

Not looking too closely at your code, the general approach would include something akin to C's "header guards":

#ifndef HEADER_H
#define HEADER_H

/* The contents of the header file, with typedefs etc. */

#endif

where HEADER_H is a C preprocessor macro specific to this header, used solely to avoid including the header again if it's been included already. I usually create a macro with the name derived from the header file itself, like MCMC_H for a header file called mcmc.h.

In a shell script, this may be as easy as

if [ -z "$script_source_guard" ]; then
script_source_guard=1

# ... do things (define functions etc.)

fi

The name of the variable script_source_guard needs to be specific to this file. Again, the name may arbitrarily be derived from the source file name; a file whose name is myfuncts.shlib may have a guard variable called MYFUNCTS or _MYFUNCTS or guard_myfuncts or whatever.

Answer 2

It’s a scope problem.  As designed, the _source_script function is, effectively, calling itself recursively, inasmuch as it is doing . ./${!arg}.sh and sourcing files that call _source_script.  But variables in shell scripts are global by default.  And so multiple incarnations of the _source_script function that are active (on the stack) simultaneously, are sharing the same copy of arg.

In detail:

When either asserts.sh or clean_environment.sh calls

_source_script scripts colors

the _source_script function is getting $# = 2.  And so the for (( arg = 2; arg <= $#; arg++ )) loop runs while $arg is <= 2, so it ends up being 3.  When . asserts.sh is called, $arg is already 3, because asserts is $3 in the

_source_script scripts colors asserts clean_environment

command in some_script.sh.  So, if $arg is set to 3 in asserts.sh, that ends up being a no-op — restoring $arg to what it already was.

But, when . clean_environment.sh is called, $arg is 4, because clean_environment is $4 in the

_source_script scripts colors asserts clean_environment

command.  So, if $arg gets set to 3 in clean_environment.sh (when it calls _source_script), the top-level incarnation of _source_script (in some_script.sh) loses its place in the for (( arg = 2; arg <= $#; arg++ )) loop, and goes back and does . clean_environment.sh again.  And again, and again, and again…

Solution

The solution, of course, is simply to put a local arg declaration in the _source_script function.

---

Other notes on your scripts:

• You have this situation:

  • You have some files that define various things (constants, functions, aliases, etc.).
  • You have other files that use those things that are defined, above, and so they . (source) the file(s) that contain the definitions, and

  • Some files are in both of the above groups; e.g., asserts.sh and clean_environment.sh presumably provide some resources to their callers, but they also read colors.sh to get the values defined therein.  Presumably there is a hierarchy:

    • some_script.sh calls colors.sh
    • some_script.sh calls asserts.sh
      • asserts.sh calls colors.sh
    • some_script.sh calls clean_environment.sh
      • clean_environment.sh calls colors.sh

    etc.

• A few users have suggested using “header guards” to prevent shell command files from being processed more than once.  While this suggestion is good and relevant to your situation (as described above), I don’t really see how it’s relevant to your question.

  — except that you should use it in _source_script.sh itself.

  In the above hierarchy, I left out the fact that every file calls _source_script.sh.  This means that the _source_script function is being redefined while it’s running.  That sounds potentially very dangerous.

• You say

  if [ "$#" -ge "1" ]
  then
      EXPECTED_DIR=$1
  fi
   ︙
  

  You might as well just say

  if [ "$#" -le  1  ]
  then
      exit 1                                  # or maybe      return 1
  fi
  EXPECTED_DIR=$1
     ︙
  

  because there’s nothing to do if there are fewer than two arguments.

• It’s safer not to use variables in a printf format string (i.e., the first argument) unless you’re sure you need to.  So

  printf ". ./${!arg}.sh\n"
  printf "${RED}Script '${!arg}.sh' not loaded successfully. Exiting...${NORMAL}\n"
  printf "No script loaded: $CURRENT_DIR != $EXPECTED_DIR.\n"
  

  should be

  printf ". ./%s.sh\n" "${!arg}"
  printf "%sScript '%s.sh' not loaded successfully. Exiting...%s\n" \
                                                      "$RED" "${!arg}" "$NORMAL"
  printf "No script loaded: %s != %s.\n" "$CURRENT_DIR" "$EXPECTED_DIR"
  

• You should use quotes more.

  . ./${!arg}.sh
  

  should be

  . "./${!arg}.sh"
  

  and, strictly speaking, to be really paranoid safe, $? should be "$?".

• But, rather than doing an explicit test on "$?", why not just say

  if ! . "./${!arg}.sh"
  then
      printf "%sScript '%s.sh' not loaded successfully. Exiting...%s\n" \
                                                      "$RED" "${!arg}" "$NORMAL"
      exit 1
  fi
  

  ?

  Note that a . command that succeeds will set $? to the exit status of the last command in the file.  So be careful that your header files (the .sh files that define things for other files to use) return an accurate exit status.

Answer 3

If you want to chain sourcing through one parent file, fine. I would not put the source statements within a function. Simply source the parent file, and it should source all the children, too.

From the bash manual page on the internal command source:

[It] read[s] and executes commands from filename in the current shell environment and return[s] the exit status of the last command executed from filename.

If you cannot guarantee that the current shell environment will persist up execution of a function, you may want to think about your configuration more closely.