58

Does it depend on what file system I use? For example, ext2/ext3/ext4 but also what happens when I insert one of those "joliet" CD-ROMs with ISO 9660? I've heard that POSIX contains some sort of spec for the charset encoding of filenames?

Essentially, what I wonder is if I got a UTF-8 encoded filename, what processing/coversion do I need to do before I pass it to a file I/O API in Linux?

2
  • 1
    The answers below say that the OS and filesystem don't care about encodings. Some filesystems, such as HFS+, do care a great deal. HFS+, I believe, requires UTF-8, which it converts internally to a restricted dialect of UTF-16. NTFS also has a similar issue but I'm not clear on the details.
    – zmccord
    Commented Apr 22, 2015 at 15:21
  • HFS+ also requires that names be decomposed which doesn't play nice with linux's tendency to use precomposed. web.archive.org/web/20080518105836/http://developer.apple.com/…
    – user12439
    Commented Aug 27, 2017 at 10:27

3 Answers 3

58

As noted by others, there isn't really an answer to this: filenames and paths do not have an encoding; the OS only deals with sequence of bytes. Individual applications may choose to interpret them as being encoded in some way, but this varies.

Specifically, Glib (used by Gtk+ apps) assumes that all file names are UTF-8 encoded, regardless of the user's locale. This may be overridden with the environment variables G_FILENAME_ENCODING and G_BROKEN_FILENAMES.

On the other hand, Qt defaults to assuming that all file names are encoded in the current user's locale. An individual application may choose to override this assumption, though I do not know of any that do, and there is no external override switch.

Modern Linux distributions are set up such that all users are using UTF-8 locales and paths on foreign filesystem mounts are translated to UTF-8, so this difference in strategies generally has no effect. However, if you really want to be safe, you cannot assume any structure about filenames beyond "NUL-terminated, '/'-delimited sequence of bytes".

(Also note: locale may vary by process. Two different processes run by the same user may be in different locales simply by having different environment variables set.)

7
  • 3
    "NUL-terminated, '/'-delimited sequence of bytes" But without an encoding, how do you know what byte represents '/'?
    – Jack
    Commented Jun 5, 2019 at 4:13
  • 1
    @Jack Always '\x2F' regardless of what looks like /. Notably different in SJIS.
    – ephemient
    Commented Jun 5, 2019 at 5:26
  • 1
    Ah, okay. Would you consider updating the answer with that info? Maybe it's just because I recently worked on a charset conversion library, but the phrase "'/'-delimited sequence of bytes" makes no sense to me.
    – Jack
    Commented Jun 5, 2019 at 7:47
  • So how to see bytes of filename in SSH session in HEX?
    – Dims
    Commented Jun 25, 2019 at 7:13
  • Please refer to the encoding in filesystems as noted by @adam byrtek below. The filesystem determines the meaning of the bytes. Everything else tries to help you to guess the bytes (you have to pass to open() ) if you know the meaning, but not the mount options. Commented Nov 21, 2021 at 7:52
11

The unix/posix layer of linux doesn't care which encoding you use. It stores the byte sequence of your current encoding as-is.

I think those mount options are there to help you convert specific filesystems that define a charset to your system charset. (CDROMs, NTFS and the FAT variants use some unicode variants).

I wish unix defined a system global encoding, but it is actually a per user setting. So if you define a different encoding then your collegue, your filenames will show up differently.

2
  • Ok so then I should probably check what locale the user is currently using and convert to that for new files so that he will see the filename correctly in Nautilus etc. How can I tell what the current filename charset is for the current user?
    – martin
    Commented Sep 15, 2010 at 18:28
  • 1
    @martin It's not even that simple... Different processes can use different encodings, depending on env variables and the language it was written in.
    – Basic
    Commented Mar 5, 2016 at 20:42
6

It depends on how you mount the file system, just take a look at mount options for different file systems in man mount. For example iso9660, vfat and fat have iocharset and utf8 options.

2
  • So if I mount it using utf8 should I also pass utf8 to the open() syscall?
    – martin
    Commented Sep 15, 2010 at 17:11
  • Also I found this ( library.gnome.org/devel/glib/unstable/… ) which seems to indicate that the charset encoding of filenames is dependent on what locale is set?
    – martin
    Commented Sep 15, 2010 at 17:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .