I have a bunch of directories and subdirectories that contain files with special characters, like this file:

robbie@phil:~$ ls test�sktest.txt 

Find reveals an escape sequence:

robbie@phil:~$ find test�sktest.txt -ls 
424512 4000 -rwxr--r-x   1 robbie   robbie    4091743 Jan 26 00:34 test\323sktest.txt

The only reason I can even type their names on the console is because of tab completion. This also means I can rename them manually (and strip the special character).

I've set LC_ALL to UTF-8, which does not seem to help (also not on a new shell):

robbie@phil:~$ echo $LC_ALL

I'm connecting to the machine using ssh from my mac. It's an Ubuntu install:

robbie@phil:~$ cat /etc/lsb-release 

Shell is Bash, TERM is set to xterm-color.

These files have been there for quite a while, and they have not been created using that install of Ubuntu. So I don't know what the system encoding settings used to be.

I've tried things along the lines of:

find . -type f -ls | sed 's/[^a-zA-Z0-9]//g'

But I can't find a solution that does everything I want:

  1. Identify all files that have undisplayable characters (the above ignores way too much)
  2. For all those files in a directory tree (recursively), execute mv oldname newname
  3. Optionally, the ability to transliterate special characters such as ä to a (not required, but would be awesome)


  1. Correctly display all these files (and no errors in applications when trying to open them)

I have bits and pieces, like iterating over all files and moving them, but identifying the files and formatting them correctly for the mv command seems to be the hard part.

Any extra information as to why they do not display correctly, or how to "guess" the correct encoding are also welcome. (I've tried convmv but it doesn't seem to do exactly what I want: http://j3e.de/linux/convmv/)

up vote 19 down vote accepted

I guess you see this invalid character because the name contains a byte sequence that isn't valid UTF-8. File names on typical unix filesystems (including yours) are byte strings, and it's up to applications to decide on what encoding to use. Nowadays, there is a trend to use UTF-8, but it's not universal, especially in locales that could never live with plain ASCII and have been using other encodings since before UTF-8 even existed.

Try LC_CTYPE=en_US.iso88591 ls to see if the file name makes sense in ISO-8859-1 (latin-1). If it doesn't, try other locales. Note that only the LC_CTYPE locale setting matters here.

In a UTF-8 locale, the following command will show you all files whose name is not valid UTF-8:

grep-invalid-utf8 () {
  perl -l -ne '/^([\000-\177]|[\300-\337][\200-\277]|[\340-\357][\200-\277]{2}|[\360-\367][\200-\277]{3}|[\370-\373][\200-\277]{4}|[\374-\375][\200-\277]{5})*$/ or print'
find | grep-invalid-utf8

You can check if they make more sense in another locale with recode or iconv:

find | grep-invalid-utf8 | recode latin1..utf8
find | grep-invalid-utf8 | iconv -f latin1 -t utf8

Once you've determined that a bunch of file names are in a certain encoding (e.g. latin1), one way to rename them is

find | grep-invalid-utf8 |
rename 'BEGIN {binmode STDIN, ":encoding(latin1)"; use Encode;}
        $_=encode("utf8", $_)'

This uses the perl rename command available on Debian and Ubuntu. You can pass it -n to show what it would be doing without actually renaming the files.

  • Thanks I will try some of these things later today! Looks like this will be the accepted answer :) – RobbieV Jan 26 '11 at 10:17
  • The find | grep '[[:print:]]' command seems to simply return all files. Shouldn't UTF-8 be compatible with many other encodings with "normal" characters? – RobbieV Jan 26 '11 at 21:23
  • @RobbieV: I typoed and meant grep [^[:print:]] to search for unprintable characters. But I've just tested with GNU grep and invalid UTF-8 sequences aren't caught by [^[:print:]] (which makes sense as they're not unprintable characters, they aren't characters at all). I've edited my post with a longer way of grepping lines with invalid utf8 sequences. Note that I've also fixed the direction of the recode and iconv examples. – Gilles Jan 26 '11 at 22:14
  • That worked perfectly. Tried all the commands except the iconv one, and they all work as expected. Pure magic! – RobbieV Jan 26 '11 at 22:55
  • Even the suggested latin1 encoding was the correct one :) – RobbieV Jan 26 '11 at 22:55

I know this is an old question but i have been searching all night for a similar solution. I found a few helpful tips but they did not do exactly what i needed, so I had to mix and match a few to get the correct outcome I was looking for

to simply remove special characters and replace them with a (.) dot

for f in *.txt; do mv "$f" `echo $f | sed "s/[^a-zA-Z0-9.]/./g"`; done

to use in a cronjob I did the following to run every minute

*/1 * * * * cd /path/to/files/ && for f in *.txt; do mv "$f" `echo $f | sed "s/[^a-zA-Z0-9.]/./g"`; done >/dev/null 2>&1

I hope someone finds this helpful as it has made my day :)

  • (1) For clarity, you might want to change `…` to $(…) — see this, this, and this.  (2) You should always quote your shell variable references (e.g., "$f") unless you have a good reason not to, and you’re sure you know what you’re doing.  This applies even to echo "$f" | sed ….  It also applies to the entire $(…) (or `…`) expression; i.e., mv "$f" "$(echo "$f" | sed "…")".  … (Cont’d) – Scott May 27 '16 at 7:29
  • (Cont’d) …  (3) You should say mv -- "$f" …, to protect against filenames beginning with -. (4) If you have files named “foo♥bar.txt” and “foo♠bar.txt”, this will (try to) rename both of them to “foo.bar.txt”, possibly causing all but one of the files to be destroyed. (5) Why on earth would you want to do this once every minute? – Scott May 27 '16 at 7:29
  • I have a torrent script that auto downloads files. and sometimes some files have characters in them that throws the uploader off. so by simply renaming files with special characters my cron fixed all my problems and the uploader does its job smoothly. – Topps70 Jun 1 '16 at 0:09
  • so (this fi'le tha,t was - down_loaded.ext) turns into (this.fi.le.tha.t.was.down.loaded.ext) – Topps70 Jun 1 '16 at 0:12

Now, when you know which encoding is used for the filenames on the remote end ("latin1" -- according to the comments to the first answer), you could also follow the second way -- run a local termninal and ssh in such a way that the remote filenames are displayed correctly (rather than the first way: rename them).

Like me, you could start a terminal locally that would work in that special encoding, perhaps, like this:

LC_ALL=en_US.latin1 xvt &

xvt stands for your terminal program.

Perhaps, the existing locale is called en_US.iso88591, and not en_US.latin1, as I assumed.

This doesn't meet the bulk requirements, but I have just had a similar problem where I had multiple versions of a file with similar names which differed only by a single weird character. Unfortunately this meant that I could not rename the offenders using the wildcard trick I usually use.

In the end I used Filezilla to connect as an SFTP client, browsed to the files and renamed them using the GUI. Filezilla handled the dodgy chars quite well.

Hope this helps someone else :-)

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.