Does it depend on what file system I use? For example, ext2/ext3/ext4 but also what happens when I insert one of those "joliet" CD-ROMs with ISO 9660? I've heard that POSIX contains some sort of spec for the charset encoding of filenames?

Essentially, what I wonder is if I got a UTF-8 encoded filename, what processing/coversion do I need to do before I pass it to a file I/O API in Linux?

  • The answers below say that the OS and filesystem don't care about encodings. Some filesystems, such as HFS+, do care a great deal. HFS+, I believe, requires UTF-8, which it converts internally to a restricted dialect of UTF-16. NTFS also has a similar issue but I'm not clear on the details. – zmccord Apr 22 '15 at 15:21
  • HFS+ also requires that names be decomposed which doesn't play nice with linux's tendency to use precomposed. web.archive.org/web/20080518105836/http://developer.apple.com/… – user12439 Aug 27 '17 at 10:27

As noted by others, there isn't really an answer to this: filenames and paths do not have an encoding; the OS only deals with sequence of bytes. Individual applications may choose to interpret them as being encoded in some way, but this varies.

Specifically, Glib (used by Gtk+ apps) assumes that all file names are UTF-8 encoded, regardless of the user's locale. This may be overridden with the environment variables G_FILENAME_ENCODING and G_BROKEN_FILENAMES.

On the other hand, Qt defaults to assuming that all file names are encoded in the current user's locale. An individual application may choose to override this assumption, though I do not know of any that do, and there is no external override switch.

Modern Linux distributions are set up such that all users are using UTF-8 locales and paths on foreign filesystem mounts are translated to UTF-8, so this difference in strategies generally has no effect. However, if you really want to be safe, you cannot assume any structure about filenames beyond "NUL-terminated, '/'-delimited sequence of bytes".

(Also note: locale may vary by process. Two different processes run by the same user may be in different locales simply by having different environment variables set.)

  • 2
    "NUL-terminated, '/'-delimited sequence of bytes" But without an encoding, how do you know what byte represents '/'? – Jack Jun 5 '19 at 4:13
  • 1
    @Jack Always '\x2F' regardless of what looks like /. Notably different in SJIS. – ephemient Jun 5 '19 at 5:26
  • 1
    Ah, okay. Would you consider updating the answer with that info? Maybe it's just because I recently worked on a charset conversion library, but the phrase "'/'-delimited sequence of bytes" makes no sense to me. – Jack Jun 5 '19 at 7:47
  • So how to see bytes of filename in SSH session in HEX? – Dims Jun 25 '19 at 7:13

The unix/posix layer of linux doesn't care which encoding you use. It stores the byte sequence of your current encoding as-is.

I think those mount options are there to help you convert specific filesystems that define a charset to your system charset. (CDROMs, NTFS and the FAT variants use some unicode variants).

I wish unix defined a system global encoding, but it is actually a per user setting. So if you define a different encoding then your collegue, your filenames will show up differently.

  • Ok so then I should probably check what locale the user is currently using and convert to that for new files so that he will see the filename correctly in Nautilus etc. How can I tell what the current filename charset is for the current user? – martin Sep 15 '10 at 18:28
  • 1
    @martin It's not even that simple... Different processes can use different encodings, depending on env variables and the language it was written in. – Basic Mar 5 '16 at 20:42

It depends on how you mount the file system, just take a look at mount options for different file systems in man mount. For example iso9660, vfat and fat have iocharset and utf8 options.

  • So if I mount it using utf8 should I also pass utf8 to the open() syscall? – martin Sep 15 '10 at 17:11
  • Also I found this ( library.gnome.org/devel/glib/unstable/… ) which seems to indicate that the charset encoding of filenames is dependent on what locale is set? – martin Sep 15 '10 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.