31

In a file system where filenames are in UTF-8, I have a file with a faulty name; it is displayed as: D�sinstaller, actual name according to zsh: D$'\351'sinstaller, Latin1 for Désinstaller, itself a French barbarism for "uninstall." Zsh would not match it with [[ $file =~ '^.*$' ]] but would match it with a globbing *—this is the behavior I expect.

Now I still expect to find it when running find . -name '*'—as a matter of fact, I would never expect a filename to fail this test. However, with LANG=en_US.utf8, the file does not show up, and I have to set LANG=C (or en_US, or '') for it to work.

Question: What is the implementation behind, and how could I have predicted that outcome?

Infos: Arch Linux 3.14.37-1-lts, find (GNU findutils) 4.4.2

  • 1
    have you considered convmv to convert file names to utf-8? – ctrl-alt-delor Apr 9 '15 at 18:53
  • @richard: In fact, I use to rely on [[ $file =~ '^.*$' ]] failing to use recode on the file name, but I'll now look into convmv if need be. Thanks. – Michaël Apr 9 '15 at 22:22
25

That's a really nice catch. From a quick look at the source code for GNU find, I would say this boils down to how fnmatch behaves on invalid byte sequences (pred_name_common in pred.c):

b = fnmatch (str, base, flags) == 0;
(...)
return b;

This code tests the return value of fnmatch for equality with 0, but does not check for errors; this results in any errors being reported as "doesn't match".

It has been suggested, many years ago, to change the behavior of this libc function to always return true on the * pattern, even on broken file names, but from what I can tell the idea must have been rejected (see the thread starting at https://sourceware.org/ml/libc-hacker/2002-11/msg00071.html):

When fnmatch detects an invalid multibyte character it should fall back to single byte matching, so that "*" has a chance to match such a string.

And why is this better or more correct? Is there existing practice?

As mentioned by Stéphane Chazelas in a comment, and also in the same 2002 thread, this is inconsistent with the glob expansion performed by shells, which do not choke on invalid characters. Perhaps even more puzzling is the fact that reversing the test will match only those files that have broken names (create files in bash with touch $'D\351marrer' $'Touch\303\251' $'\346\227\245\346\234\254\350\252\236'):

$ find -name '*'
.
./Touché
./日本語

$ find -not -name '*'
./D?marrer

So, to answer your question, you could have predicted this by knowing the behavior of your fnmatch in this case, and knowing how find handles this function's return value; you probably could not have found out solely by reading the documentation.

  • My guess for why there is no fix for * is that then it would be inconsistent with D*staller. – ctrl-alt-delor Apr 9 '15 at 18:50
  • 7
    @richard, the idea would be that D*staller would match on $'D\351sinstaller' as well like it does in the glob of all the shells I've tested. Given that GNU fnmatch behaviour is not consistent with that of the GNU shell, I'd say it's a bug. – Stéphane Chazelas Apr 9 '15 at 19:35
  • 1
    Great indepth answer, dhag; much appreciated. Would you mind pointing out the standard spec to which fnmatch complies? I can find the usual POSIX regexp spec specifying that . should only match valid characters in the encoding—hence my expectation that .* does not match invalid strings—but I cannot find a matching specification for the globbing star. – Michaël Apr 9 '15 at 22:27
  • 1
    The closest specification I can find online is on this OpenGroup page. It states Matching shall be based on the bit pattern used for encoding the character, not on the graphic representation of the character. and The <asterisk> is a pattern that shall match any string, including the null string. This can arguably be interpreted as @StéphaneChazelas's suggestion. 13 years later, it may be time to ping upstream again :-) – Michaël Apr 10 '15 at 14:04
  • @Michaël, I couldn't find anything better either. Perhaps, as a point of comparison, GNU find on Mac OS behaves in a way consistent with the shell's globbing (i.e., -name '*' matches all files, broken names included), so presumably BSD's version of fnmatch, which doesn't claim POSIX.2 cnoformance, unlike the GNU version, has a different, and arguably saner, interpretation of what should be done on invalid characters. – dhag Apr 10 '15 at 14:13
13

find -name option uses shell pattern matching notation to perform matching filename. * is a pattern matching multiple characters, shall match a string of zero or more characters.

find uses fnmatch to check pattern matching, so you can use ltrace to check the result:

$ touch $'\U1212'aa
$ touch D$'\351'sinstaller
$ LC_ALL=en_US.utf8 ltrace -e fnmatch find -name '*'          
find->fnmatch("foo", "foo", 0)                   = 0
find->fnmatch("Foo", "foo", 0)                   = 1
find->fnmatch("Foo", "foo", 16)                  = 0
find->fnmatch("*", ".", 0)                       = 0
.
find->fnmatch("*", "D\351sinstaller", 0)         = -1
find->fnmatch("*", "\341\210\222aa", 0)          = 0
./ሒaa
+++ exited (status 0) +++

With D\351sinstaller, fnmatch return -1, indicated that it failed to match. A valid character like ሒaa will be matched.

In your case, with UTF-8 locale, \351 is an invalid character, causing pattern matching fail.

  • 3
    At the very least, +1 for the use of ltrace. I did know about strace, but ltrace is new to me. Lovely! – Michaël Apr 9 '15 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.