Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a bunch of directories and subdirectories that contain files with special characters, like this file:

robbie@phil:~$ ls test�sktest.txt 
test?sktest.txt

Find reveals an escape sequence:

robbie@phil:~$ find test�sktest.txt -ls 
424512 4000 -rwxr--r-x   1 robbie   robbie    4091743 Jan 26 00:34 test\323sktest.txt

The only reason I can even type their names on the console is because of tab completion. This also means I can rename them manually (and strip the special character).

I've set LC_ALL to UTF-8, which does not seem to help (also not on a new shell):

robbie@phil:~$ echo $LC_ALL
en_US.UTF-8

I'm connecting to the machine using ssh from my mac. It's an Ubuntu install:

robbie@phil:~$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=7.10
DISTRIB_CODENAME=gutsy
DISTRIB_DESCRIPTION="Ubuntu 7.10"

Shell is Bash, TERM is set to xterm-color.

These files have been there for quite a while, and they have not been created using that install of Ubuntu. So I don't know what the system encoding settings used to be.

I've tried things along the lines of:

find . -type f -ls | sed 's/[^a-zA-Z0-9]//g'

But I can't find a solution that does everything I want:

  1. Identify all files that have undisplayable characters (the above ignores way too much)
  2. For all those files in a directory tree (recursively), execute mv oldname newname
  3. Optionally, the ability to transliterate special characters such as ä to a (not required, but would be awesome)

OR

  1. Correctly display all these files (and no errors in applications when trying to open them)

I have bits and pieces, like iterating over all files and moving them, but identifying the files and formatting them correctly for the mv command seems to be the hard part.

Any extra information as to why they do not display correctly, or how to "guess" the correct encoding are also welcome. (I've tried convmv but it doesn't seem to do exactly what I want: http://j3e.de/linux/convmv/)

share|improve this question
    
The single answer below follows the first way (find them and rename to your new encoding), but the second way would also be interesting: now, when you know the encoding used for the remote filenames, how to ssh to the remote host in such a way that the filenames are displayed correctly (and can be managed by typing their names with your keyboard)? –  imz -- Ivan Zakharyaschev Nov 25 '11 at 17:52
add comment

3 Answers

up vote 14 down vote accepted

I guess you see this invalid character because the name contains a byte sequence that isn't valid UTF-8. File names on typical unix filesystems (including yours) are byte strings, and it's up to applications to decide on what encoding to use. Nowadays, there is a trend to use UTF-8, but it's not universal, especially in locales that could never live with plain ASCII and have been using other encodings since before UTF-8 even existed.

Try LC_CTYPE=en_US.iso88591 ls to see if the file name makes sense in ISO-8859-1 (latin-1). If it doesn't, try other locales. Note that only the LC_CTYPE locale setting matters here.

In a UTF-8 locale, the following command will show you all files whose name is not valid UTF-8:

grep-invalid-utf8 () {
  perl -l -ne '/^([\000-\177]|[\300-\337][\200-\277]|[\340-\357][\200-\277]{2}|[\360-\367][\200-\277]{3}|[\370-\373][\200-\277]{4}|[\374-\375][\200-\277]{5})*$/ or print'
}
find | grep-invalid-utf8

You can check if they make more sense in another locale with recode or iconv:

find | grep-invalid-utf8 | recode latin1..utf8
find | grep-invalid-utf8 | iconv -f latin1 -t utf8

Once you've determined that a bunch of file names are in a certain encoding (e.g. latin1), one way to rename them is

find | grep-invalid-utf8 |
rename 'BEGIN {binmode STDIN, ":encoding(latin1)"; use Encode;}
        $_=encode("utf8", $_)'

This uses the perl rename command available on Debian and Ubuntu. You can pass it -n to show what it would be doing without actually renaming the files.

share|improve this answer
    
Thanks I will try some of these things later today! Looks like this will be the accepted answer :) –  RobbieV Jan 26 '11 at 10:17
    
The find | grep '[[:print:]]' command seems to simply return all files. Shouldn't UTF-8 be compatible with many other encodings with "normal" characters? –  RobbieV Jan 26 '11 at 21:23
    
@RobbieV: I typoed and meant grep [^[:print:]] to search for unprintable characters. But I've just tested with GNU grep and invalid UTF-8 sequences aren't caught by [^[:print:]] (which makes sense as they're not unprintable characters, they aren't characters at all). I've edited my post with a longer way of grepping lines with invalid utf8 sequences. Note that I've also fixed the direction of the recode and iconv examples. –  Gilles Jan 26 '11 at 22:14
    
That worked perfectly. Tried all the commands except the iconv one, and they all work as expected. Pure magic! –  RobbieV Jan 26 '11 at 22:55
    
Even the suggested latin1 encoding was the correct one :) –  RobbieV Jan 26 '11 at 22:55
add comment

Now, when you know which encoding is used for the filenames on the remote end ("latin1" -- according to the comments to the first answer), you could also follow the second way -- run a local termninal and ssh in such a way that the remote filenames are displayed correctly (rather than the first way: rename them).

Like me, you could start a terminal locally that would work in that special encoding, perhaps, like this:

LC_ALL=en_US.latin1 xvt &

xvt stands for your terminal program.

Perhaps, the existing locale is called en_US.iso88591, and not en_US.latin1, as I assumed.

share|improve this answer
add comment

This doesn't meet the bulk requirements, but I have just had a similar problem where I had multiple versions of a file with similar names which differed only by a single weird character. Unfortunately this meant that I could not rename the offenders using the wildcard trick I usually use.

In the end I used Filezilla to connect as an SFTP client, browsed to the files and renamed them using the GUI. Filezilla handled the dodgy chars quite well.

Hope this helps someone else :-)

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.