Assume that a directory has 100 files starting with the letter 'a'.

If I do a grep <some string> a* from the terminal, how will the shell handle this?

Will it expand the regular expression, get a list of all files starting with a and grep on each one of those sequentially? Or is there some other way?

Assume that I have an array of the above filenames that begin with 'a'. Will it take more/less time if I write a for loop and do the iteration myself in a shell script or a c program?

  • 7
    BTW, it's a glob not a regular expression. Big difference. – Aaron D. Marasco Aug 4 '11 at 2:35

First, a nitpick: a string like a* in normal shell syntax is a glob, which works differently from regular expressions.

On a high-level overview, the shell interpreter (i.e. bash) expands the string a* to a list of every file name matching the pattern a*. These then become part of the command line parameters to a single instance of grep (for the programmers, all the expanded words go as separate strings into the argv argument of main). That single grep command then parses the arguments in whatever way it chooses, and it is up to grep to interpret those arguments as file names, options, option arguments, regular expressions, etc., and take the appropriate actions. Everything occurs sequentially (AFAIK no grep implementation uses multiple threads).

If you implement a loop in a shell script to do that same thing, it is almost guaranteed to be slower than the above process, for the following reasons. If you spawn a new grep process for each file, it will most certainly be slower due to the overhead of process creation being multiplied unnecessarily. If you constructed the argument list yourself in the shell script and used a single instance of grep, anything you do in shell will still be slower because shell commands have to interpreted (by bash), which adds an extra layer of code, and you'll just be re-implementing what bash was already doing faster internally in compiled code.

As for writing it yourself in C, you can probably easily get comparable performance to the process described in the first paragraph but it's unlikely that you'll be able to achieve enough of a performance gain over the current grep/bash implementations to justify the time spent without delving into machine-specific performance optimizations or sacrificing portability. Maybe you could try to come up with an arbitrarily parallelizable version of grep, but even that may not help as you are more likely to be I/O bound than CPU bound. Glob expansion and grep are already "fast enough" for most "normal" purposes.

  • Thanks for the very detailed answer. Actually, I need to grep gzipped files (few GB each). I have a list of those files. I now have a choice of either building a regex (complicated) to match those files or iterate over the known list and run grep on each one of those (easy). Hence the worry about performance. – harithski Aug 3 '11 at 18:53
  • try zcat and zgrep; no need to decompress them one by one – jw013 Aug 4 '11 at 1:31
  • Yes, of course. I am using zgrep. – harithski Aug 4 '11 at 4:40

Yes, it will expand to a list of files and feeds the resulting list to the grep program. At least that is what man bash says in subsection Pathname Expansion.

There is another way of using the expansion in simple cases as you mention: write grep <some_string> a and before pressing *, press ESC. This will expand the list of matched files right in the command line, so you can verify the list is OK before pressing Enter.

As for the second part of your question, it depends. If you mean to write a for-loop that runs grep on each of the files in turn, then it would definitely be slower, because the grep program will be run not once, but once per file. However, what is important to keep in mind is that there is a certain limit on the expanded length of command-line arguments you can use, though it is typically quite high. To see that, you can try grep adasdsadf /usr/*/*/* >/dev/null.

  • 2
    ESC+* is not exactly the same as letting bash expand * because ESC+* will insert dotfiles (names that start with a .) whereas the expansion of * depends on the dotglob shopt setting. The key sequence to expand and insert globs is C-x * by default and maps to the readline command glob-expand-word. – jw013 Aug 3 '11 at 8:48
  • 1
    @jw013 Thanks for the information! It doesn't seem to change the case of a* expansion, but is certainly important in broader scope. – rozcietrzewiacz Aug 3 '11 at 8:55
  • 2
    zsh note: just hitting the tab key on expandable parameters (glob patterns, brace-expansion, command-substitution, …) will expand them. – Stéphane Gimenez Aug 3 '11 at 8:59
  • @jw013 Actually, I just tested the C-x shortcut and it does not expand the list of files on my system (using bash). – rozcietrzewiacz Aug 3 '11 at 9:25
  • 1
    @roz Right - I hardly ever use it anyways, just wanted to point out the (rather nitpicky) difference :). C-x * only does globs which only do filenames, but Esc * actually does a lot more since it is insert-completions, as in all possible completions. This means using Esc * on an empty command line will insert the name of every single executable file in your $PATH, for instance. – jw013 Aug 3 '11 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.