Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a hard time understanding how the file name encoding works. On unix.SE I find contradicting explanations.

File names are stored as characters

To quote another answer: Several questions about file-system character encoding on linux

[…] as you mention in your question, a UNIX file name is just a sequence of characters; the kernel knows nothing about the encoding, which entirely a user-space (i.e., application-level) concept.

If file names are stored as characters, there has to be some kind of encoding involved, since finally the file name has to end up as a bit or byte sequence on the disk. If the user can choose any encoding to map the characters to a byte sequence that is fed to the kernel, it is possible to create any byte sequence for a valid file name.

Assume the following: A user uses a random encoding X, which translates the file foo into the byte sequence α and saves it to disk. Another user uses encoding Y. In this encoding α translates to /, which is not allowed as a file name. However, for the first user the file is valid.

I assume that this scenario cannot happen.

File names are stored as binary blobs

To quote another answer: What charset encoding is used for filenames and paths on Linux?

As noted by others, there isn't really an answer to this: filenames and paths do not have an encoding; the OS only deals with sequence of bytes. Individual applications may choose to interpret them as being encoded in some way, but this varies.

If the system does not deal with characters, how can particular characters (e.g. / or NULL) be forbidden in file names? There no notion of a / without an encoding.

An explanation would be that file system can store file names containing any character and it's only the user programs that take an encoding into account that would choke on file names containing invalid characters. That, in turn, means that file systems and the kernel can, without any difficulty, handle file names containing a /.

I also assume that this is wrong.

Where does the encoding take place and where is the restriction posed of not allowing particular characters?

share|improve this question
    
Null is the same (0) in all encodings. –  Kevin May 22 '12 at 19:40
    
@Kevin Not quite: not in, say, UTF-16, or UCS-4 (= UTF-32), or most other multibyte encodings that aren't extensions of ASCII. –  Gilles May 22 '12 at 22:51
    
Actually, Riccardo Murri's answer should have mentioned bytes and not characters there. Most filesystems do store bytes. –  Gilles May 22 '12 at 22:54
add comment

3 Answers 3

up vote 11 down vote accepted

Short answer: restrictions imposed in Unix/Linux/BSD kernel, namei() function. Encoding takes place in user level programs like xterm, firefox or ls.

I think you're starting from incorrect premises. A file name in Unix is a string of bytes with arbitrary values. A few values, 0x0 (ASCII Nul) and 0x2f (ASCII '/') are just not allowed, not as part of a multi-byte character encoding, not as anything. A "byte" can contain a number representing a character (in ASCII and some other encodings) but a "character" can require more than 1 byte (for example, code points about 0x7f in UTF-8 representation of Unicode).

These restrictions arise from file name printing conventions and the ASCII character set. The original Unixes used ASCII '/' (numerically 0x2f) valued bytes to separate pieces of a partially- or fully-qualified path (like '/usr/bin/cat' has pieces "usr", "bin" and "cat"). The original Unixes used ASCII Nul to terminate strings. Other than those two values, bytes in file names may assume any other value. You can see an echo of this in the UTF-8 encoding for Unicode. Printable ASCII characters, including '/', take only one byte in UTF-8. UTF-8 for code points above does not include any Zero-valued bytes, except for the Nul control character. UTF-8 was invented for Plan-9, The Pretender to the Throne of Unix.

Older Unixes (and it looks like Linux) had a namei() function that just looks at paths a byte at a time, and breaks the paths into pieces at 0x2F valued bytes, stopping at a zero-valued byte. namei() is part of the Unix/Linux/BSD kernel, so that's where the exceptional byte values get enforced.

Notice that so far, I've talked about byte values, not characters. namei() does not enforce any character semantics on the bytes. That's up to the user-level programs, like ls, which might sort file names based on byte values, or character values. xterm decides what pixels to light up for file names based on the character encoding. If you don't tell xterm you've got UTF-8 encoded filenames, you'll see a lot of gibberish when you invoke it. If vim isn't compiled to detect UTF-8 (or whatever, UTF-16, UTF-32) encodings, you'll see a lot of gibberish when you open a "text file" containing UTF-8 encoded characters.

share|improve this answer
add comment

The separation of bytes vs. characters came much after Unix was designed. When it was designed the use of the words only conveyed something about how 8 (or 6, or 9) bits was interpreted but the word encodings weren't mentioned.

Filenames are sequences of bytes. Any byte except 0x2f "/" is allowed. A byte containing 0x00 can't even get through to the kernel due to its use as a string terminator. An application can interpret the sequence of bytes according to an encoding it chooses. If that sounds messy I suppose it is.

There's more information at http://www.gtk.org/api/2.6/glib/glib-Character-Set-Conversion.html you may find useful.

share|improve this answer
add comment

The thing is, the kernel doesn't care one bit how the applications interpret the data it is given as a filename.

Let's imagine I have a C application that deals with exclusively UTF-16 strings. And I enter, via a properly configured input method, the ∯ symbol (Unicode 0x222F) into the "Save As" prompt/dialog.

If the application doesn't do any form of translation and sends that, in a plain old C string (char*) to, say, fopen in write mode, the kernel will not see ∯, or even try to imagine that. It will see two chars, one after the other, with values 0x22 0x2F (assuming 8bit chars and no funnies in the C library).
That is, from the kernel's point of view, a valid char (") followed by / (ASCII 0x2F). fopen will return EISDIR (i.e. "that looks like a directory and you requested write mode!").
If I had entered ∮ (Unicode 0x222E), the kernel would have seen two fine chars, and created a file which, as seen through an ASCII-speaking application, would be named "..

If I had entered a in the application as a file name, and the application passed it along in UTF-16 to the kernel, the kernel would read 0x00 0x61, and actually not even consider that 0x61, because the 0x00 already terminates the string, as far as it is concerned. Error message would be the same as for an empty filename (ENOENT I believe).

So the kernel does indeed take the data as a blob. It's a stream of chars. The invalid "characters" in your user-space encoding of your choice are those that generate 0x00 or 0x2F ("null" and /) in their blob (binary representation that gets passed to the kernel).

share|improve this answer
    
If I get you right, then there is no such thing as invalid characters. There are just invalid byte sequences. And the values 0x00 and 0x2F are hard coded in the kernel. That in turn means, that directories are not separated by a /, but to whatever character maps to 0x2F in the encoding in use. –  Marco May 22 '12 at 20:28
    
Yes, that's the idea if you want to see it that way. (But that might be incorrect. A kernel might have a "native encoding" where / isn't 0x2F - might not use 8-bit chars, in fact.) The "traditional" dir separator is /. That is 0x27 on 8bit byte ASCII (not EBCDIC for instance) systems. –  Mat May 22 '12 at 20:41
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.