Bash pattern to match directories whose names begin with a dot (period), by being "explicit", instead of using "shopt -s dotglob"?

Question

When constructing a pattern that matches a file name such as /home/user/project/.git, how does one match the . character "explicitly" -- that is, without the use of shopt -s dotglob?

The manual at https://www.gnu.org/software/bash/manual/html_node/Filename-Expansion.html states:

When a pattern is used for filename expansion, the character ‘.’ at the start of a filename or immediately following a slash must be matched explicitly, unless the shell option dotglob is set.

What, exactly, does it mean, to "be matched explicitly"?

And again, at http://www.tldp.org/LDP/abs/html/globbingref.html (in the Notes section at the end), the same notion is addressed:

Filename expansion can match dotfiles, but only if the pattern explicitly includes the dot as a literal character.

The note provides the following examples:

~/[.]bashrc    #  Will not expand to ~/.bashrc
~/?bashrc      #  Neither will this.
               #  Wild cards and metacharacters will NOT
               #+ expand to a dot in globbing.

~/.[b]ashrc    #  Will expand to ~/.bashrc
~/.ba?hrc      #  Likewise.
~/.bashr*      #  Likewise.

I fail to understand the inner workings of the last three examples, which will expand to include the "dotfile".

How, specifically, does placing the b in brackets after the . make this an "explicit" match in the example ~/.[b]ashrc? The subsequent examples are even more ambiguous to me. I simply fail to understand how manipulating the pattern in ways that seem completely unrelated to the . character cause the pattern to produce a match.

With regard to why I would like to avoid using shopt -s dotglob, the impetus for this question is rooted in the fact that I am writing these patterns for use in another program's configuration file. I want to exclude paths that contain, for example, "hidden .git directories", and I'm not sure that I have the ability to specify dotglob in any capacity.

In essence: What is the simplest means by which to match the . character by "being explicit"? Placing the next character in brackets "makes it work", but I'd like to know why; I feel like I'm "shooting in the dark" with that approach.

Any explanation as to the underlying behavior in this regard is much appreciated.

EDIT TO ADD:

Initially, it didn't seem relevant, but because people seem to be interested in the specifics of my use-case, I'll explain further.

I'm using a Host-Based Intrusion Detection software called Samhain. Samhain will "alert" whenever the filesystem is modified according to certain user-specified configuration parameters.

I don't want Samhain to alert when files within .git directories (that are located within certain parent directories) are created/modified/deleted. In Samhain, this type of exclusion is performed by defining "ignore rules". The exact specification of these rules is explained at http://www.la-samhna.de/samhain/manual/filedef.html , in 4.2. File/directory specification.

In short:

Wildcard patterns ('*', '?', '[...]') as in shell globbing are supported for paths. The leading '/' is mandatory.

So, I am trying to write an "ignore rule" that will match the .git directories in question, which will, in effect, cause Samhain to exclude them from its monitoring activities.

Initially, I tried this:

[IgnoreAll]
dir = -1/home/user/project/*/*/.git

This didn't work; Samhain still alerted whenever files inside those .git directories changed.

Upon finding the examples cited above, I tried this:

dir = -1/home/user/project/*/*/.[g]it

With this change, Samhain ignores the files, as desired.

In posting this question, I was simply trying to understand why that change has the intended effect.

I will say, I feel less stupid given that the very pattern I was trying to use at first does indeed match the .git directories in question when I use "echo" test:

echo /home/user/project/*/*/.git

So, it wasn't so much that I was misunderstanding something fundamental with regard to pattern-matching, globbing, or file-name expansion in Bash; to the contrary, there seem to be nuances with regard to how Samhain, in particular, implements pattern-matching in this context.

I have no idea why this doesn't work when applied in the context of Samhain's configuration file (obviously). Maybe somebody will be able to explain, given this edit.

Answer 1

When a pattern is used for filename expansion, the character ‘.’ at the start of a filename or immediately following a slash must be matched explicitly, unless the shell option dotglob is set.

This simply means the globs *, ?, and [...] don't match a . at the beginning of a filename. If you want to match a . at the beginning of a filename, you can't use a glob, you have to type the . explicitly. For instance:

$ echo ????
Work
$ echo .???
.gem .pki .ssh .vim

And to answer your other question:

How, specifically, does placing the b in brackets after the . make this an "explicit" match in the example ~/.[b]ashrc?

Just because you're using a glob pattern doesn't mean that the whole pattern is no longer "explicit". In ~/.[b]ashrc, for instance, the characters /.ashrc are all explicitly matched. However, [b] is a glob pattern and also not an explicit match. (Technically, ~ is a tilde expansion and performed earlier than glob expansion, so it is an explicit match as well.) But the other characters, including ., do match explicitly, which is why ~/.[b]ashrc matches ~/.bashrc.

For comparison, ~/?[b]ashrc will not match ~/.bashrc, because the . is no longer matched explicitly.

Answer 2

First of all, I assume that you know what things like [b], ? and * mean in a pathname pattern.  (If you don’t, do more research.)

At the risk of repeating what the others have said, you’re overthinking it.  Patterns that contain the string /. (i.e., a / immediately followed by a .) explicitly include the dot as a literal character.  The point is just that [b], ? and/or * occurring after the . don’t affect whether the pattern can match dotfile(s).  The last three examples are offered as examples of patterns (i.e., not just a plain file/pathname, but something that could potentially match several file/pathnames – or none) that will match ~/.bashrc — as opposed to the first two, which would match ~/.bashrc if . weren’t handled specially.

So, what is your real question?

… I am writing these patterns for use in another program’s configuration file.  I want to exclude paths that contain, for example, “hidden .git directories”, and I’m not sure that I have the ability to specify dotglob in any capacity.

I guess you want to do something (like chown or cp) to all files/directories except those beginning with dot.  But your code is going to be used in somebody else’s script (via the . or source command), and you’re afraid to do your_command * because the script may have set dotglob, so * would expand to all files, including “hidden” ones.  And you don’t want to turn off dotglob because you don’t want to break the functionality of the existing script.

1. Use a smarter wildcard (pathname expansion pattern).

   I hope that you understand wildcards (a.k.a. globs) like [abc] — they match any of the characters a, b or c.  For example, the string c[aou]t matches cat, cot and cut;  d[iou]g matches dig, dog and dug.  (They can be, and commonly are, used with ranges; e.g., [a-z] and [0-9].)  Well, a special case of this is [!abc] — it matches any character except a, b or c.  So you can use [!.]* (or directory_name/[!.]*) to match names that begin with a character other than dot.  Paradoxically, [.] (at the beginning of a filename) won’t match a dot if dotglob is not set, but [!.] will exclude a dot regardless of the setting of dotglob.

   This will give the same result whether dotglob is set or not.

2. Use dotglob (in a subshell).

   Shell options (shopts) are local to a process, and process attributes never flow backwards (uphill) from child to parent.  So

   (shopt -u dotglob; your_command *)

   will run your_command on non-hidden files only, without affecting the settings and behavior of the rest of the script.

3. Use dotglob (without using a subshell).

   Some people prefer to avoid subshells because they use extra resources.  But the cost is minuscule (unless you do it in a loop that executes many times), so this is not a very good reason.  A better reason to avoid a subshell is if you need to do something that affects the environment of the shell, like cd or umask.

   If this is your situation, you can temporarily turn off dotglob, and later restore the previous setting.

   If you type shopt dotglob (without -s or -u), it reports (displays) the current setting of the dotglob option.  (shopt with no parameters lists the current settings of all the options.)  It also sets the exit status accordingly.  The -q flag suppresses the display, so you can do

   shopt -q dotglob
   dotglob_setting=$?
   shopt -u dotglob
   your_command *
   if [ "$dotglob_setting" = 0 ]
   then
       shopt -s dotglob
   fi

---

But wait … you said “another program’s configuration file”.  What are you talking about?  If you’re talking about writing or modifying a file that says something like ignore=*.o, then this whole question doesn’t make sense, because that file will be processed (and interpreted) by whatever program processes it, and that program will decide how to interpret * — the shell has nothing to do with it.

---

OK, now that we have a better idea of what the question is:

The short answer is that the behavior that you’re seeing doesn’t make sense.  If a .git directory exists, then specifying it exactly (literally) as .git and specifying it with a wildcard / glob pattern of .[g]it should behave identically.

The longer answer: I stand by the last paragraph of the first version of my answer.  Samhain is reading and parsing its policy configuration file.  It might use the shell to interpret wildcards in the config file, but I guess that it’s doing it internally.

And, if it is “using the shell”, which shell is it using?  On many systems, /bin/sh is not bash.  Their baseline behavior with regards to pathname expansion patterns (i.e., wildcards) should be the same, but once you step off the porch, you’re in a swamp.  The POSIX specification for the shell doesn’t even have the shopt command, and (AFAIK) doesn’t have any way to make * expand to all files (and not just non-hidden ones).

If you feel like wasting spending some more time on this, you might experiment with putting  /home/user/project/* into the Samhain config file and seeing whether it interprets it as all files or just non-hidden ones.  If it interprets it as all files, we can conclude

1. Samhain isn’t using /bin/sh to expand wildcards.
2. It isn’t using standard, default rules for wildcards (the ones you discussed at such length in your question).
3. The documentation is wrong (or, at best, incomplete and misleading) inasmuch as it says, “Wildcard patterns (‘*’, ‘?’, ‘[...]’) as in shell globbing are supported for paths.” without saying that (unlike in the shell’s default behavior) * means all files.
4. It might be using bash in dotglob mode to expand wildcards.  But this doesn’t make sense; as I said, the handling of .git and .[g]it doesn’t correspond to the normal behavior of any shell that I know of.  It’s almost certainly got its own code for wildcards.

But in any case I believe that we can say with some confidence that your conclusion is correct: Samhain has a bug with regard to the handling of wildcards in IgnoreAll specifications.  You might want to file a bug report with the vendor.  Or, since you’ve found a workaround, you could just forget about it.