I just tried to move a directory containing music files with thunar 4.10 It complained that a file name was invalid.

It turned out that one file name (song title) contained a question mark. I suspected that this was a problem, removed the question mark and could indeed copy the file. Adding the "?" back in was not possible. I also tried it with rename on the command line but that didn't work either. (not sure what thunar uses under the hood, so this test might be moot)

Now if a question mark makes the file name invalid, how could this file be created in the first place? I created the files with SoundJuicer from a newly obtained CD. I was able to play the file (with "?" in the name) in various players.

What's going on here? Can I have the "?" in the name or not? Why is the file manager unable to handle such files while other applications seem to be ok with it?

Update: Next song has a ":" in it. Same problem as with the "?".

These are not invalid characters to Unix; typically only the NUL character and the / character are invalid filenames (the / being the directory separator).

This was what my intuition told me as well, because I never had any issues with file names in Linux and could throw pretty much everything sensible at it and it worked ok. This is what motivated the question here. I never encountered invalid file names before.

Were you trying to move the files to a USB stick? If so, is that stick formatted as FAT32 or as a native Linux filesystem?

The target is indeed a USB stick that I bought today. I opened gparted and it is formatted as FAT32.

I'm not exactly sure but that's a Windows thing right? And Windows has a bunch of characters that it doesn't support, apaprently including ? and :. Am I right?

  • 3
    These are not invalid characters to Unix; typically only the NUL character and the / character are invalid filenames (the / being the directory separator). Now in the command line you may need to quote the ? (and ` ` and * and similar) - e.g. touch 'test?' - to ensure the shell doesn't try to interpret them, but that's a different problem. – Stephen Harris Aug 1 '16 at 17:34
  • 4
    Were you trying to move the files to a USB stick? If so, is that stick formatted as FAT32 or as a native Linux filesystem? – roaima Aug 1 '16 at 17:44
  • @roaima sorry, for the missing information, I edited my answer to include it. – null Aug 1 '16 at 17:53
  • 1
    Just tried it on my machine and Thunar 1.6.10 ( xfce 4.12 ) and did not have a problem creating and moving files with ? or : in the name. I also assume, as roaima already pointed out, that rather the filesystem does not support the special characters. – Thomas Aug 1 '16 at 17:53
up vote 6 down vote accepted

These characters ? and : are not valid on a FAT32 filesystem, so if that is where you need to copy your files you will need to rename them.

From the command line you can use command-line tools such as rename (sometimes known as prename) to replace these characters with _ or even to remove them:

rename 's/[?<>\\:*|\"]/_/g'    # Change invalid characters to _
rename 's/[?<>\\:*|\"]//g'     # Remove invalid characters

I am not familiar with thunar so I do not know if there is a way to perform this substitution/replacement operation directly.

I have just found Linux copy to fat32 filesystem: invalid argument which suggests adding this into the pax command (another tool to copy files), so that you can keep your full filenames on your local disk but convert the filenames during the copy to your USB device:

pax -rw -s '/[?<>\\:*|\"]/_/gp' *.mp3 /media/usb_device

If the complete filenames are really important to you, I would suggest that you reformat the USB stick to use a Linux-native filesystem such as ext4. (There are Windows drivers available for the extN family of filesystems if that's necessary.)

  • The newly bought USB stick came with some encryption software on it that I totally ignored. Had I paid some attention to it, I would have noticed that the executables are .app for Mac and .exe for Win, which could have been a hint on checking the file system. – null Aug 1 '16 at 18:06

It is definitely allowed to have ? in a filename.

From the POSIX standard:

Filename

A sequence of bytes consisting of 1 to {NAME_MAX} bytes used to name a file. The bytes composing the name shall not contain the <NUL> or <slash> characters. In the context of a pathname, each filename shall be followed by a <slash> or a <NUL> character; elsewhere, a filename followed by a <NUL> character forms a string (but not necessarily a character string). The filenames dot and dot-dot have special meaning. A filename is sometimes referred to as a "pathname component". See also Pathname.

So the only characters a filename may not contain is \0 (signifying an end-of-string) and / (which is a path separator). Also, the filenames . and .. are special and may not be used by files that you create.

Some characters are special to the shell and needs to be escaped or properly quoted for the shell to not treat them as globbing patterns. The question mark is one of those characters, as it usually stands for "any one character" in shell globbing patterns.

My guess is that Thunar is choosing not to handle these filenames as the characters : and ? (and <, >, ", |, \, and *) are not allowed on Windows, and it wants to force you to use filenames that are portable between Windows and Unix. I don't know the reasoning behind this, but it could have something to do with making Thunar work on filesystems shared with Windows machines.

EDIT: After the question was updated with further information, it is clear that the files will need to conform to Windows naming rules as the filesystem is indeed a Windows filesystem.

@roaima's answer shows how to rename files using the Linux rename utility. This is a another solution that will work on systems without rename, a bash function, fatmv that renames files containing illegal Windows characters by changing them to underscores (_):

function fatmv {
    name="$( basename "$1" )"
    destdir="$2"

    fat_name="$( tr ':?<>"|\\*' '_' <<<"$name" )"

    if [[ -n "$destdir" ]]; then
      fat_name="$destdir"/"$fat_name"
    fi

    echo mv -f -- "$1" "$fat_name"
}

If a second argument is used, this is treated as a destination directory the which the renamed file should be moved:

$ fatmv '<illegal>:filename?' /mount/fat_disk/
mv -f -- <illegal>:filename? /mount/fat_disk//_illegal__filename_

Test this as is, and then remove the echo to make it actually do something.

  • So how can the OP handle these files in their GUI filemanager? – roaima Aug 1 '16 at 17:39

The ? is a special character that has significance in bash, You can work with it by escaping it with \

ie:

touch test\?

will make the file:

test?

and you can move it with

mv test\? /new/loction/test\?

Edit: The same for any special character, such as .

You also need it for file names that contain spaces

this is a file

would have to be moved with:

mv this\ is\ a\ file <location>
  • But how do you escape a ? in a name when you're using a GUI filemanager? – roaima Aug 1 '16 at 17:38
  • Thank you for the info. I'm sure the makers of the thunar file manager were aware of that? Taking care of the "problematic" files individually seems tedious. Can I configure thunar somehow to be aware of the problem and do the escaping? Or is the command line my only option? – null Aug 1 '16 at 17:38
  • My Apologies...Honestly It's been so long since I've done a GUI file operation that even reading that you were using Thunar, it didn't click that this was a GUI issue. I'll look for more information for you. – Gravy Aug 1 '16 at 17:42

To apply @roaimas answer to all files under a given directory, you can use the command

find /path/to/dir -exec rename 's/[?<>\\:*|\"/_/g' {} \;

which applies rename to all files.

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.