11

Is there a historical reason why Bash "globbing" and regular expressions are not identical? For example, I believe that in Bash [1-2]* matches anything that starts with a 1 or a 2 followed by anything else, while as a regular expression [1-2]* would match only a sequence of 1s and 2s. My Bash scripting and REGEX foo are both pretty weak and I regularly run into problems associated with these differences which made me curious was to why they are different.

  • 3
    Would you consider doing rm -- ^[^.].*\.txt$ instead of rm -- *.txt? – Stéphane Chazelas Jun 10 '14 at 7:59
  • 1
    Much of your Q's are touched on in this thread from lwn: lwn.net/Articles/96687 – slm Jun 10 '14 at 8:04
  • There are commands that operate on filenames and take regexp. For example find, find . -regex ".*\.txt$" | xargs rm --, or rename for renaming files (it is sed for filenames), beware some systems have a different rename. – ctrl-alt-delor Jun 10 '14 at 11:29
  • @richard, my ^[^.].*\.txt$ was to take into account the ignoring of dot files. Note that the -regex is a GNU extensions, some shells like ksh93 or zsh can incorporate regexps in their globs (try for instance: ksh93 -c 'echo ~(E:^[^.].*\.txt$)') – Stéphane Chazelas Jun 10 '14 at 11:41
  • 2
    That bash follows existing practice so carefully while avoiding irreconcilably incompatible changes and extensions is one of its greatest strengths. – ormaaj Jun 18 '14 at 16:36
12

bash was initially designed in the late 80s as a partial clone of ksh with some interactive features from csh/tcsh.

The origins of globbing have to be found in those earlier shells which it builds upon.

ksh itself is an extension of the Bourne shell. The Bourne shell itself (first released in 1979 in Unix V7) was a clean implementation from scratch, but it did not depart completely from the Thompson shell (the shell of V1 -> V6) and incorporated features from the Mashey shell.

In particular, command arguments were still separated by blanks, | was now the new pipe operator but ^ was still supported as an alternative (and also explains why you do [!a-z] and not [^a-z]), $1 was still the first argument to a script and backslash was still the escape character. So many of the regexp operators (^\|$) have a special meaning of their own in the shell.

The Thompson shell relied on an external utility for globbing. When sh found unquoted *, [ or ?s in the command, it would run the command through glob.

rm *.txt

would end up running glob as:

["glob", "rm", "*.txt"]

and glob would end up running rm with the list of files matching that pattern.

grep a.\*b *.txt

would run glob as:

["glob", "grep", "a.\252b", "*.txt"]

The * above has been quoted by setting the 8th bit on that character, preventing glob from treating it as a wildcard. glob would then remove that bit before calling grep.

To do the equivalent with regexps, that would have been:

regexp rm '\.txt$'

Or:

regexp rm '^[^.].*\.txt$'

to exclude dot-files.

The need to escape the operators as they double as shell special characters, the fact that ., common in filenames is a regexp operator makes it not very appropriate to match filenames and complicated for a beginner. In most cases, all you need is wildcards that can replace either one (?) or any number (*) of characters.

Now, different shells added different globbing operators. Nowadays, the ksh and zsh globs (and to some extent bash -O extglob which implements a subset of ksh globs) are functionally equivalent to regexps with a syntax that is less cumbersome to use with filenames and the current shell syntax. For instance, in zsh (with extendedglob extension), you can do:

echo a#.txt

if you want (unlikely) to match filenames that consist of sequences of a followed by .txt. Easier than echo (^a*\.txt$) (here using braces as a way to isolate the regex operators from the shell operators which could have been one way shells could deal with it).

echo (foo|bar|<1-20>).(#i)mpg

For mpg files (case insensitive) whose basename is foo, bar or a decimal number from 1 to 20...

ksh93 now can also incorporate regexps (basic, extended, perl-like or "augmented") in its globs (though it's quite buggy) and even provides a tool to convert between glob and regexp (printf %R, printf %P):

echo ~(Ei:.*\.txt)

to match (non-hidden) txt files with Extended regular expressions, case-insensitively.

  • Cool write-up! You actually can't use ~(opt:pat) for any of the capitalized options. Maybe print -r -- ~(Ei).*\.txt$. Putting the pattern inside seems to only be useful to avoid having to toggle an option on then off for part of a pattern. Oddly you can mix-and-match multiple pattern languages within the same glob though. ~(Ki)*.~(E)txt$ is equivalent. (In the end everything just gets converted to regex and passed to libast's regex engine internally). – ormaaj Jun 18 '14 at 16:24
  • @ormaaj, ~(Ei:.*\.txt) works for me even with 15 year old versions like ksh93 o+. – Stéphane Chazelas Jun 5 '18 at 13:04
  • Works with one of my saved test binaries too (2014-12-24), but I recall running into issues with that. Things were always randomly broken and fixed again between each version back when ksh was still commercially developed. I remember the pattern matching code being one of the fragile areas. – ormaaj Jun 6 '18 at 15:22
  • @ormaaj, one different between ~(E)x and ~(E:x) is that the latter is anchored (matches on x only while the former matches on anything containing x), which may be the kind of issue you ran into (use ~(-lr)~(E:x) to remove the anchoring, ~(E-lr:x) won't do). In any case, I agree it's pretty buggy, even in the latest version. – Stéphane Chazelas Jun 6 '18 at 15:32
9

Regular languages were introduced by Kleene in 1956. The seminal paper didn't have the full modern notation for regular expressions, but it did introduce the “Kleen star”: A* meaning “any number of repetitions of A”. In the next decade, some more or less standard notations emerged, in particular . for an arbitrary character and ? to mean that the previous character is optional.

Bash's globbing notation stems from the glob command introduced all the way back in Unix v1 in 1971. At the time, globbing was performed by a separate program; it was later moved into the shell. The early glob command has ? to mean “any one character” and * to mean “any sequence of characters”. I don't know why the characters were chosen; ? is pretty intuitive, and * may have been inspired from the one in regular expressions.

Globbing wasn't intended to be as general as regular expressions, and regular expressions were not very widespread at the time, so there was no call to unify the concepts. From the start, there were syntactic incompatibilities, with ?, . and * meaning different things in file name patterns and in regular expressions.

Modern shells such as bash expand on glob patterns, but it was gradual evolution maintaining backward compatibility. Ksh88 (the 1988 version of the Korn shell) introduced an extended syntax for shell patterns, which could not be the same syntax as usual regular expressions but was strongly inspired by it: *(PATTERN) to mean any number of repetitions of PATTERN, @(PATTERN1|PATTERN2) to mean “PATTERN1 or PATTERN2”, etc.

Modern versions of bash (since 2.02) support ksh88's extended patterns, if you issue shopt -s extglob first.

  • Has Bash ever not supported extglobs? As far as I'm aware Bash, zsh, and {pd,m}ksh have supported the exact same globs as documented in the ksh88 manual since early days. Ksh to this day doesn't even have an option to disable "extended" glob quantifiers, and ksh93 is the only one of the bunch to have any extensions beyond what ksh88 had. – ormaaj Jun 18 '14 at 15:32
  • 2
    @ormaaj Ksh88 extended globs and the extglob option were introduced in bash 2.02 somewhere around 1998. Zsh acquired ksh_glob in the 3.1 series somewhere around the same time. Zsh has many globbing extensions of its own (some requiring the extended_glob option). – Gilles 'SO- stop being evil' Jun 18 '14 at 15:42
  • I see. So it actually was late enough to justify the need for an option. (I think the default being off is rather pointless these days but, interesting.) – ormaaj Jun 18 '14 at 16:02
  • 1
    @ormaaj, Note that in bash, contrary to ksh, extglob makes bash non-POSIX compliant because it's not disabled in variables. In ksh, var='@(*)'; echo $var expands to all the file names in the current dir that start with @( and end in ) as POSIX requires while in bash -O extglob it expands to all the files. (still, one may consider bash behaviour makes more sense here (and the ksh behaviour is quite a pain when you do want to have patterns in variables)). That glob syntax is so awkward because of that (POSIX/Bourne compatibility). Compare with zsh extended globs. – Stéphane Chazelas Jun 18 '14 at 18:09
  • @StéphaneChazelas That's all true, and I like how ksh is somewhat smart about it. It rarely comes in to play though unless actually constrained to POSIX. With almost every use for wordsplitting replaced by better features, and storing patterns in variables being an extreme nuisance anyway since you have to empty IFS, disable brace expansion everywhere but bash. I think it's still impossible to be completely safe with stored patterns. This old escape issue was never really resolved for instance. – ormaaj Jun 18 '14 at 19:50
1

Historical reason: YES. Reference:
http://en.wikipedia.org/wiki/Glob_(programming)#Origin

Just to showcase the divergence, here is a good and easy example: a*

  • shell globbing: meaning is, first character is a and then whatever (a, ab, abca...)
  • regex: meaning is, zero or more repetitions of character a (a, aa, aaa...)

I'd readily agree that this discrepancy in meaning is very confusing for new users.

Globbing is perhaps easier to grasp for newcomers, but it also is less powerful construct.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.