11

I have again and again had this problem: I have a glob, that matches exactly the correct files, but causes Command line too long. Every time I have converted it to some combination of find and grep that works for the particular situation, but which is not 100% equivalent.

For example:

./foo*bar/quux[A-Z]{.bak,}/pic[0-9][0-9][0-9][0-9]?.jpg

Is there a tool for converting globs into find expressions that I am not aware of? Or is there an option for find to match the glob without matching a the same glob in a subdir (e.g. foo/*.jpg is not allowed to match bar/foo/*.jpg)?

  • Expand the brace and you should be able to use the resultant expressions with -path or -ipath. find . -path './foo*bar/quux[A-Z]/pic[0-9][0-9][0-9][0-9]?.jpg' should work - except that it will match /fooz/blah/bar/quuxA/pic1234d.jpg. Will that be a problem? – muru Aug 18 '17 at 8:44
  • Yes, that will be a problem. It has to be 100% equivalent. – Ole Tange Aug 18 '17 at 8:44
  • The problem is that we have no idea, what is exactly the difference. Your pattern is pretty okay. – user259412 Aug 18 '17 at 8:45
  • I added your extension post as an answer to the question. I hope it is not so bad. – user259412 Aug 18 '17 at 13:46
  • Can't you do echo <glob> | cat, assuming my knowledge of bash, echo is build-in, and thus doesn't have the max command limit – Ferrybig Aug 18 '17 at 14:06
15

If the problem is that you get an argument-list-is-too-long error, use a loop, or a shell built-in. While command glob-that-matches-too-much can error out, for f in glob-that-matches-too-much does not, so you can just do:

for f in foo*bar/quux[A-Z]{.bak,}/pic[0-9][0-9][0-9][0-9]?.jpg
do
    something "$f"
done

The loop might be excruciatingly slow, but it should work.

Or:

printf "%s\0" foo*bar/quux[A-Z]{.bak,}/pic[0-9][0-9][0-9][0-9]?.jpg |
  xargs -r0 something

(printf being builtin in most shells, the above works around the limitation of the execve() system call)

$ cat /usr/share/**/* > /dev/null
zsh: argument list too long: cat
$ printf "%s\n" /usr/share/**/* | wc -l
165606

Also works with bash. I'm not sure exactly where this is documented though.


Both Vim's glob2regpat() and Python's fnmatch.translate() can convert globs to regexes, but both also use .* for *, matching across /.

  • If that is true, then replacing something with echo ought to do it. – Ole Tange Aug 18 '17 at 9:45
  • 1
    @OleTange That's why I suggested printf - it will be faster than calling echo thousands of times, and offers more flexibility. – muru Aug 18 '17 at 9:46
  • 4
    There’s a limit on the arguments that can be passed through exec, which applies to external commands such as cat; but that limit doesn’t apply to shell builtin commands such as printf. – Stephen Kitt Aug 18 '17 at 9:50
  • 1
    @OleTange The line is not too long because printf is a builtin, and the shells presumably use the same method for supplying arguments to it that they use for enumerating arguments for for. cat is not a builtin. – muru Aug 18 '17 at 9:50
  • 1
    Technically there are shells like mksh where printf is not builtin and shells like ksh93 where cat is (or can be) builtin. See also zargs in zsh to work around it without having to resort to xargs. – Stéphane Chazelas Aug 18 '17 at 10:03
9

find (for the -name/-path standard predicates) uses wildcard patterns just like globs (note that {a,b} is not a glob operator; after expansion, you get two globs). The main difference is the handling of slashes (and dot files and dirs not being treated specially in find). * in globs won't span several directories. */*/* will cause up to 2 levels of directories to be listed. Adding a -path './*/*/*' will match any files that are at least 3 levels deep and won't stop find from listing the contents of any directory at any depth.

For that particular

./foo*bar/quux[A-Z]{.bak,}/pic[0-9][0-9][0-9][0-9]?.jpg

couple of globs, it's easy to translate, you're wanting directories at depth 3, so you can use:

find . -mindepth 3 -maxdepth 3 \
       \( -path './foo*bar/quux[A-Z].bak/pic[0-9][0-9][0-9][0-9]?.jpg' -o \
          -path './foo*bar/quux[A-Z]/pic[0-9][0-9][0-9][0-9]?.jpg' \) \
       -exec cmd {} +

(or -depth 3 with some find implementations). Or POSIXly:

find . -path './*/*/*' -prune \
       \( -path './foo*bar/quux[A-Z].bak/pic[0-9][0-9][0-9][0-9]?.jpg' -o \
          -path './foo*bar/quux[A-Z]/pic[0-9][0-9][0-9][0-9]?.jpg' \) \
       -exec cmd {} +

Which would guarantee that those * and ? could not match / characters.

(find, contrary to globs would read the content of directories other than foo*bar ones in the current directory¹, and not sort the list of files. But if we leave aside the problem that what is matched by [A-Z] or the behaviour of */? with regards to invalid characters is unspecified, you'd get the same list of files).

But in any case, as @muru has shown, there's no need to resort to find if it's just for splitting the list of files into several runs to work around the limit of the execve() system call. Some shells like zsh (with zargs) or ksh93 (with command -x) even have builtin support for that.

With zsh (whose globs also have the equivalent of -type f and most other find predicates), for instance:

autoload zargs # if not already in ~/.zshrc
zargs ./foo*bar/quux[A-Z](|.bak)/pic[0-9][0-9][0-9][0-9]?.jpg(.) -- cmd

((|.bak) is a glob operator contrary to {,.bak}, the (.) glob qualifier is the equivalent of find's -type f, add oN in there to skip the sorting like with find, D to include dot-files (doesn't apply to this glob))


¹ For find to crawl the directory tree like globs would, you'd need something like:

find . ! -name . \( \
  \( -path './*/*' -o -name 'foo*bar' -o -prune \) \
  -path './*/*/*' -prune -name 'pic[0-9][0-9][0-9][0-9]?.jpg' -exec cmd {} + -o \
  \( ! -path './*/*' -o -name 'quux[A-Z]' -o -name 'quux[A-Z].bak' -o -prune \) \)

That is prune all directories at level 1 except the foo*bar ones, and all at level 2 except the quux[A-Z] or quux[A-Z].bak ones, and then select the pic... ones at level 3 (and prune all directories at that level).

3

You could write a regex for find matching your requirements:

find . -regextype egrep -regex './foo[^/]*bar/quux[A-Z](\.bak)?/pic[0-9][0-9][0-9][0-9][^/]?\.jpg'
  • Is there a tool that does this conversion to avoid human errors? – Ole Tange Aug 18 '17 at 9:22
  • No, but the only changes I made were to escape ., add the optional match for .bak and change * to [^/]* to not match paths like /foo/foo/bar etc. – sebasth Aug 18 '17 at 9:23
  • But even your conversion is wrong. ? is not changed into [^/]. This is exactly the kind of human error I want to avoid. – Ole Tange Aug 18 '17 at 9:24
  • 1
    I think with egrep, you can shorten [0-9][0-9][0-9][0-9]? to [0-9]{3,4} – wjandrea Aug 18 '17 at 21:09
  • 1
    @OleTange See Create regex from glob expression – wjandrea Aug 18 '17 at 21:11
0

Generalising on the note on my other answer, as a more direct answer to your question, you could use this POSIX sh script to convert the glob to a find expression:

#! /bin/sh -
glob=${1#./}
shift
n=$#
p='./*'

while true; do
  case $glob in
    (*/*)
      set -- "$@" \( ! -path "$p" -o -path "$p/*" -o -name "${glob%%/*}" -o -prune \)
      glob=${glob#*/} p=$p/*;;
    (*)
      set -- "$@" -path "$p" -prune -name "$glob"
      while [ "$n" -gt 0 ]; do
        set -- "$@" "$1"
        shift
        n=$((n - 1))
      done
      break;;
  esac
done
find . "$@"

To be used with one standard sh glob (so not the two globs of your example which uses brace expansion):

glob2find './foo*bar/quux[A-Z].bak/pic[0-9][0-9][0-9][0-9]?.jpg' \
  -type f -exec cmd {} +

(that doesn't ignore dot-files or dot-dirs except . and .. and doesn't sort the list of files).

That one only works with globs relative to the current directory, with no . or .. components. With some effort, you could extend it to any glob, more than a glob... That could also be optimised so that glob2find 'dir/*' doesn't look for dir the same was as it would for a pattern.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.