3

I have a huge Folder with thousands of Files. Some Files have some Characters in it, which are not allowed. (UTF-8 signs) So I have a white list of allowed characters and a beginning of a bash-script to get a list of files with the path to it, which have some characters not on that white list.

#!/bin/bash
regex="^[a-zA-Z0-9._- ]+$"


while IFS=  read -r -d $'\0'; do
    filename=`echo "$REPLY" | rev  | cut -d/ -f1| rev`
    filepath=`echo "$REPLY" | rev  | cut -d/ -f2- | rev`

    if ! [[ "$filename" =~  "$regex" ]]
    then
            echo "$filepath $filename"
    fi
done < <(find /path/to/folder -type f -print0)

This is another beginning of a script

find /path/to/folder -type f -regextype posix-extended ! -iregex "\/([A-Z0-9\-\_\.\ \/]*)"

And here are some Files in that storage

/symlnks/data/DATEN_EINGANG/DATENLIEFERUNG/Aestetico_19-11-2015/Probenbox_Probenkästen.pdf
/symlnks/data/DATEN_EINGANG/DATENLIEFERUNG/Aestetico_19-11-2015/Probenbox_final.pdf
/symlnks/data/DATEN_EINGANG/DATENLIEFERUNG/Aestetico_19-11-2015/._Probenbox_final.pdf
1

A possible solution is. to use grep with perl-regex. The flag for this is -P

For example it should look like this:

#!/bin/bash

regex="[^-_0-9A-Za-z\. ]+"

while IFS=  read -r -d $'\0'; do
    filepath=${REPLY%/*}
    filename=${REPLY##*/}

    #use grep with perl-regex -P and 
    #-q for quiet to prevent output to stdin

    echo "$filename" | grep -qP "$regex" 
    #now we compare the return code from grep
    if  [[ "$?" -eq 0 ]]
    then
        echo "match: $filename"
    else
        echo "nomatch: $filename"

    fi


done < <(find /symlnks -type f -print0)
1

If you're looking to distinguish ascii from utf, the command file is probably you're best bet. man file for details.

Here's a way to find all the files in the current directory whose names are in ascii or non-ascii:

$ cat foo.sh
#!/bin/bash
echo "$1" > /tmp/name.txt
file /tmp/name.txt | grep -q $2
exit $?

$ find . -maxdepth 1 -type f -name \*txt -exec ./foo.sh {} UTF \; -a -print
./へ.txt
./robenbox_Probenkästen.txt
$ find . -maxdepth 1 -type f -name \*txt -exec ./foo.sh {} ASCII \; -a -print
./foo.txt
./log.txt
./utf8.txt

Here's my first answer...

Thus:

$ find . -maxdepth 1 -type f -name \*txt -exec ./foo.sh {} ASCII \; -a -print
./ascii.txt
$ find . -maxdepth 1 -type f -name \*txt -exec ./foo.sh {} utf \; -a -print
./utf8.txt
$ cat foo.sh
#!/bin/bash
file $1 | grep -q $2
exit $?

Because:

$ cat ascii.txt 
English is a West Germanic language that was first spoken in early medieval England and is now a global lingua franca.
$ cat utf8.txt 
Texts written with Man'yōgana use two different kanji for each of the syllables now pronounced き ki, ひ hi, み mi, け ke, へ he, め me, こ ko, そ so, と to, の no, も mo, よ yo and ろ ro.
$ file ascii.txt 
ascii.txt: ASCII text
$ file utf8.txt 
utf8.txt: UTF-8 Unicode text
ken@ken-x230: ~$ od -c utf8.txt 
0000000   T   e   x   t   s       w   r   i   t   t   e   n       w   i
0000020   t   h       M   a   n   '   y 305 215   g   a   n   a       u
0000040   s   e       t   w   o       d   i   f   f   e   r   e   n   t
0000060       k   a   n   j   i       f   o   r       e   a   c   h    
0000100   o   f       t   h   e       s   y   l   l   a   b   l   e   s
0000120       n   o   w       p   r   o   n   o   u   n   c   e   d    
0000140 343 201 215       k   i   ,     343 201 262       h   i   ,    
0000160 343 201 277       m   i   ,     343 201 221       k   e   ,    
0000200 343 201 270       h   e   ,     343 202 201       m   e   ,    
0000220 343 201 223       k   o   ,     343 201 235       s   o   ,    
0000240 343 201 250       t   o   ,     343 201 256       n   o   ,    
0000260 343 202 202       m   o   ,     343 202 210       y   o       a
0000300   n   d     343 202 215       r   o   .  \n
0000313
ken@ken-x230: ~$ od -c ascii.txt 
0000000   E   n   g   l   i   s   h       i   s       a       W   e   s
0000020   t       G   e   r   m   a   n   i   c       l   a   n   g   u
0000040   a   g   e       t   h   a   t       w   a   s       f   i   r
0000060   s   t       s   p   o   k   e   n       i   n       e   a   r
0000100   l   y       m   e   d   i   e   v   a   l       E   n   g   l
0000120   a   n   d       a   n   d       i   s       n   o   w       a
0000140       g   l   o   b   a   l       l   i   n   g   u   a       f
0000160   r   a   n   c   a   .  \n
0000167
  • 1
    Welcome to stackexchange. It looks like the OP is trying to find special characters in file names, not the file contents. – Wildcard Dec 4 '15 at 16:23
  • They specifically say "UTF-8 signs". I think they're trying to use regex="^[a-zA-Z0-9._- ]+$" as a proxy for the character set. – kenberland Dec 4 '15 at 17:12
  • Yes, UTF-8, but in the file name not the contents. PDFs aren't ASCII-only no matter what; they're binary. See the filename examples the OP gave. – Wildcard Dec 4 '15 at 18:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.