Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

In a directory size 80GB with approx 700,000 files, there are some file names with non-English characters in the file name. Other than trawling through the file list laboriously is there:

  • An easy way to list or otherwise identify these file names?
  • A way to generate printable non-English language characters - those characters that are not listed in the printable range of man ascii (so I can test that these files are being identified)?
share|improve this question
add comment

3 Answers

up vote 10 down vote accepted

Assuming that "foreign" means "not an ASCII character", then you can use find with a pattern to find all files not having printable ASCII characters in their names:

LC_ALL=C find . -name '*[! -~]*'

(The space is the first printable character listed on http://www.asciitable.com/, ~ is the last.)

The hint for LC_ALL=C is required (actually, LC_CTYPE=C and LC_COLLATE=C), otherwise the character range is interpreted incorrectly. See also the manual page glob(7). Since LC_ALL=C causes find to interpret strings as ASCII, it will print multi-byte characters (such as π) as question marks. To fix this, pipe to some program (e.g. cat) or redirect to file.

Instead of specifying character ranges, [:print:] can also be used to select "printable characters". Be sure to set the C locale or you get quite (seemingly) arbitrary behavior.

Example:

$ touch $(printf '\u03c0') "$(printf 'x\ty')"
$ ls -F
dir/  foo  foo.c  xrestop-0.4/  xrestop-0.4.tar.gz  π
$ find -name '*[! -~]*'       # this is broken (LC_COLLATE=en_US.UTF-8)
./x?y
./dir
./π
... (a lot more)
./foo.c
$ LC_ALL=C find . -name '*[! -~]*'
./x?y
./??
$ LC_ALL=C find . -name '*[! -~]*' | cat
./x y
./π
$ LC_ALL=C find . -name '*[![:print:]]*' | cat
./x y
./π
share|improve this answer
    
Be aware that you have file names that are using foreign character sets that are incompatible with UTF-8 or ASCII. In those cases, you may see question marks instead of characters. –  Lekensteyn Jan 17 at 11:23
1  
+1, but I would use LC_ALL=C instead of LC_COLLATE=C as it's doesn't make much sense to set LC_COLLATE to C without setting LC_CTYPE and to make sure it still works even when the LC_ALL variable is in the environment. –  Stephane Chazelas Jan 17 at 11:47
    
If SPC is printable, then what about TAB and LF which are also typically found in text files? –  Stephane Chazelas Jan 17 at 11:56
    
Thanks - this found six files, which had long hyphen, short hyphen and a variant of single quote. These had all originated from MS Word. No difference in the files listed between LC_ALL and LC_COLLATE. LC_COLLATE displayed the non-ASCII chars correctly whereas LC_ALL displayed ??? instead. Excellent answer! –  suspectus Jan 17 at 12:35
1  
@suspectus I updated by answer based on suggestions from Stephane. For LC_COLLATE and LC_CTYPE, see also the find(1) manpage. –  Lekensteyn Jan 17 at 12:44
add comment

If you translate each file name using tr -d '[\200-\377]' and compare it with the original name, then any file names that have special characters will not be the same.

(The above assuming that you mean non-ASCII with foreign)

share|improve this answer
1  
That also removes [ and ] in most tr implementations. –  Stephane Chazelas Jan 17 at 11:21
    
Yes - it did remove [ and ] on my system. –  suspectus Jan 17 at 11:57
    
+1 - the solution did find all the (six) file names with non ASCII symbols (in addition to the [ and ]s). thanks. –  suspectus Jan 17 at 12:26
add comment

You can use tr to delete any foreign character from a filename and compare the result with the original filename to see if it contained foreign characters.

find . -type f > filenames
while read filename; do
      stripped="$(printf '%s\n' "$filename" | tr -d -C '[[:alnum:]][[:space:]][[:punct:]]')"
      test "$filename" = "$stripped" || printf '%s\n' "$filename"; 
done < filenames
share|improve this answer
2  
that is a nice extension to my answer, but it is too simple, file names can have newlines in them and then your script will not work –  Timo Jan 17 at 11:16
1  
If you want to post-process find output, use NUL-terminated output/input as shown in this answer. –  Lekensteyn Jan 17 at 11:20
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.