I have a larger directory with files whose filenames contain special characters such as line breaks. The origin of these characters is seemingly that I copied and pasted text from within pdf files (titles and author names) to the 'save as' dialog of pdf-readers, ignoring that they contained these invalid characters. With ls or a file manager I do not see the special characters but their presence hinders me to copy or to rename them. So how can I recursively rename the files removing all invalid characters? Note that I do not want to remove regular utf8 characters such as umlauts, spaces etc.

  • There's really no compelling reason modern tools can't handle files with whitespace (even newlines) in their names. If you're having trouble, you're either using readline tools or not quoting correctly. Shell globs and find with -exec will not have a problem. – kojiro Feb 5 '14 at 4:07
  • I didn't say that whitespaces cause problems. In fact I would want to keep them. With newlines I am not sure if what some programs (text editors) display as such are indeed only newlines. Generally I would like to remove them however just like other characters which might cause trouble (are not correctly displayed on the command line or difficult to enter with utf8 encoding enabled). Furthermore they should be compatible with common file systems including vfat. – highsciguy Feb 5 '14 at 10:25

The perl-way:


opendir(DIR,".") or die "$@:$!";
while ($in = readdir(DIR)) {
  next unless -f $in;
  ($out = $in) =~ s/[^a-zA-Z0-9._-]//g;
  warn "$@:$!" unless rename $in, $out;

The regex filters only a-zA-Z ... (could also be [:print:] for printable chars) as valid chars. There is no checking for empty target names.

  • I could probably improve it by replacing your [^a-zA-Z0-9._-] by [\W._\-]+. I do not know the ($out = $in) construct. What is it doing? The [:print:] character class would probably be too large if that includes all perl printable characters (newlines as well). – highsciguy Feb 5 '14 at 10:31
  • Yes probably, it always depends what you want, the regexp is set by discrete expressions, so it's advantage is to see which characters will be passed. ($out = $in) is an assigment, like $out = $in; $out =~ ... – user55518 Feb 5 '14 at 13:02

With zsh:

autoload zmv
zmv -n '(**/)(*)' '$1${2//[^[:print:]]}'

(remove the -n when happy). If you want to include hidden files (and look into hidden dirs):

zmv -Qn '(**/)(*)(D)' '$1${2//[^[:print:]]}'

That removes the characters that are non-printable in your locale.

  • +1 for speed; it works for me but a bash solution would still be very useful – highsciguy Feb 4 '14 at 16:11
  • A character which seems to cause trouble but does not seem to be addressed is . Another one appears as \#37. – highsciguy Feb 4 '14 at 16:29
  • @highsciguy, is U+2019 (right single quotation mark), that's not special to any tool that I know. In which way is it causing you problem? Can you maybe edit your question with more specifics as to what characters are a problem. Are they those that you cannot enter with your keyboard? What type of keyboard do you have? – Stéphane Chazelas Feb 4 '14 at 16:55
  • No, I can enter it, but I see that some files containing this character cannot be copied by rsync. I just checked however that I can create and move files with this character manually, so I am not sure anymore what's causing the problem for these. – highsciguy Feb 4 '14 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.