How to deal with characters like ":" or "?" that make invalid filenames?

Question

I just tried to move a directory containing music files with thunar 4.10 It complained that a file name was invalid.

It turned out that one file name (song title) contained a question mark. I suspected that this was a problem, removed the question mark and could indeed copy the file. Adding the "?" back in was not possible. I also tried it with rename on the command line but that didn't work either. (not sure what thunar uses under the hood, so this test might be moot)

Now if a question mark makes the file name invalid, how could this file be created in the first place? I created the files with SoundJuicer from a newly obtained CD. I was able to play the file (with "?" in the name) in various players.

What's going on here? Can I have the "?" in the name or not? Why is the file manager unable to handle such files while other applications seem to be ok with it?

Update: Next song has a ":" in it. Same problem as with the "?".

These are not invalid characters to Unix; typically only the NUL character and the / character are invalid filenames (the / being the directory separator).

This was what my intuition told me as well, because I never had any issues with file names in Linux and could throw pretty much everything sensible at it and it worked ok. This is what motivated the question here. I never encountered invalid file names before.

Were you trying to move the files to a USB stick? If so, is that stick formatted as FAT32 or as a native Linux filesystem?

The target is indeed a USB stick that I bought today. I opened gparted and it is formatted as FAT32.

I'm not exactly sure but that's a Windows thing right? And Windows has a bunch of characters that it doesn't support, apaprently including ? and :. Am I right?

Answer 1

These characters ? and : are not valid on a FAT32 filesystem, so if that is where you need to copy your files you will need to rename them.

From the command line you can use command-line tools such as rename (sometimes known as prename) to replace these characters with _ or even to remove them:

rename 's/[?<>\\:*|\"]/_/g'    # Change invalid characters to _
rename 's/[?<>\\:*|\"]//g'     # Remove invalid characters

I am not familiar with thunar so I do not know if there is a way to perform this substitution/replacement operation directly.

I have just found Linux copy to fat32 filesystem: invalid argument which suggests adding this into the pax command (another tool to copy files), so that you can keep your full filenames on your local disk but convert the filenames during the copy to your USB device:

pax -rw -s '/[?<>\\:*|\"]/_/gp' *.mp3 /media/usb_device

If the complete filenames are really important to you, I would suggest that you reformat the USB stick to use a Linux-native filesystem such as ext4. (There are Windows drivers available for the extN family of filesystems if that's necessary.)

Answer 2

It is definitely allowed to have ? in a filename.

From the POSIX standard:

Filename

A sequence of bytes consisting of 1 to {NAME_MAX} bytes used to name a file. The bytes composing the name shall not contain the <NUL> or <slash> characters. In the context of a pathname, each filename shall be followed by a <slash> or a <NUL> character; elsewhere, a filename followed by a <NUL> character forms a string (but not necessarily a character string). The filenames dot and dot-dot have special meaning. A filename is sometimes referred to as a "pathname component". See also Pathname.

So the only characters a filename may not contain is \0 (signifying an end-of-string) and / (which is a path separator). Also, the filenames . and .. are special and may not be used by files that you create.

Some characters are special to the shell and needs to be escaped or properly quoted for the shell to not treat them as globbing patterns. The question mark is one of those characters, as it usually stands for "any one character" in shell globbing patterns.

My guess is that Thunar is choosing not to handle these filenames as the characters : and ? (and <, >, ", |, \, and *) are not allowed on Windows, and it wants to force you to use filenames that are portable between Windows and Unix. I don't know the reasoning behind this, but it could have something to do with making Thunar work on filesystems shared with Windows machines.

EDIT: After the question was updated with further information, it is clear that the files will need to conform to Windows naming rules as the filesystem is indeed a Windows filesystem.

@roaima's answer shows how to rename files using the Linux rename utility. This is a another solution that will work on systems without rename, a bash function, fatmv that renames files containing illegal Windows characters by changing them to underscores (_):

function fatmv {
    name="$( basename "$1" )"
    destdir="$2"

    fat_name="$( tr ':?<>"|\\*' '_' <<<"$name" )"

    if [[ -n "$destdir" ]]; then
      fat_name="$destdir"/"$fat_name"
    fi

    echo mv -f -- "$1" "$fat_name"
}

If a second argument is used, this is treated as a destination directory the which the renamed file should be moved:

$ fatmv '<illegal>:filename?' /mount/fat_disk/
mv -f -- <illegal>:filename? /mount/fat_disk//_illegal__filename_

Test this as is, and then remove the echo to make it actually do something.

Answer 3

The ? is a special character that has significance in bash, You can work with it by escaping it with \

ie:

touch test\?

will make the file:

test?

and you can move it with

mv test\? /new/loction/test\?

Edit: The same for any special character, such as .

You also need it for file names that contain spaces

this is a file

would have to be moved with:

mv this\ is\ a\ file <location>

Answer 4

To apply @roaimas answer to all files under a given directory, you can use the command

find /path/to/dir -exec rename 's/[?<>\\:*|\"/_/g' {} \;

which applies rename to all files.

Answer 5

To apply @serv-inc and @roaimas to not only replace ? and : but also all other characters (e. g. unicode characters which are not FAT32 valid), that are not in [A-Za-z0-9] with _ of the names of all files in a directory, you can use the following:

for file in *.mp3 ; do var=$( echo $file | sed -e 's/[^A-Za-z0-9]/_/g'); var=${var:0:60} ; var=${var%_mp3}.mp3 ; mv "$file" $var ; done

I also had problems with too long filenames of MP3 files on FAT32, so it cuts the basename off at 60 characters (I don't know the max. length, but 60 characters works for me). Be careful this overwrites the original file names by moving the files.