13

Due to a lot of file exchange works between Windows (GBK encoding) and Linux (UTF-8 encoding), it will encounter character encoding issues easily, such as:

  • zip/tar files whose name contains chinese characters on Windows system, unzip/untar it in Linux system.
  • run migrated legacy java web application (designed on Windows system, using GBK encoding in JSP) which write GBK-encoding-named files to disk.
  • ftp get/put GBK-encoding-named files between Windows FTP server and Linux client.
  • switch LANG environment in Linux.

The common issue of the previous mentioned are file locating/naming. After googled, I got an article Using Unicode in Linux https://www.linux.com/news/using-unicode-linux/, it said:

the operating system and many utilities do not realize what characters the bytes in file names represent.

So, it's possible to have two files with same name (SAME when their names are decoded by correct character set, but DIFFERENT in bytes), such as 中文.txt, but in different encoding:

[root@fedora test]# ls
????  中文
[root@fedora test]# ls | iconv -f GBK
中文
涓iconv: illegal input sequence at position 7
[root@fedora test]# ls 中文 && ls $'\xd6\xd0\xce\xc4' | iconv -f gbk
中文
中文

Questions:

  1. Is it possible to config linux filesystem use fixed character encoding (like NTFS use UTF-16 internally) to store file names regardless of LANG/LC_ALL environment?
  2. Or, what I actually want ask is: Is it possible to let file name 中文.txt ($'\xe4\xb8\xad\xe6\x96\x87.txt') in zh_CN.UTF-8 environment and file name 中文.txt ($'\xd6\xd0\xce\xc4.txt') in zh_CN.GBK environment refer to same file?
  3. If it's not configurable, then is it possible to patch kernel to translate character encoding between file-system and current environment (just a question, not request implementation)? and how much performance con effect if it's possible?
2
  • You could tackle the problem from the Windows side by using Cygwin 1.7, which does automatically translate between the filesystem's UTF-16 encoding and whatever encoding has been specified in the locale settings. It defaults to UTF-8, so for example Cygwin tar would encode filenames as UTF-8.
    – ak2
    Jun 24, 2011 at 8:13
  • @ak2 Thanks, Cygwin is really good, I've been use it for years. The tar/zip case is just an example, in real environment, the zip/tar files may be created by others (such as download a file from internet).
    – LiuYan 刘研
    Jun 24, 2011 at 8:34

4 Answers 4

Reset to default
8

I have reformulated your questions a bit, for reasons that should appear evident when you read them in sequence.

1. Is it possible to config linux filesystem use fixed character encoding to store file names regardless of LANG/LC_ALL environment?

No, this is not possible: as you mention in your question, a UNIX file name is just a sequence of bytes; the kernel knows nothing about the encoding, which entirely a user-space (i.e., application-level) concept.

In other words, the kernel knows nothing about LANG/LC_*, so it cannot translate.

2. Is it possible to let different file names refer to same file?

You can have multiple directory entries referring to the same file; you can make that through hard links or symbolic links.

Be aware, however, that the file names that are not valid in the current encoding (e.g., your GBK character string when you're working in a UTF-8 locale) will display badly, if at all.

3. Is it possible to patch the kernel to translate character encoding between file-system and current environment?

You cannot patch the kernel to do this (see 1.), but you could -in theory- patch the C library (e.g., glibc) to perform this translation, and always convert file names to UTF-8 when it calls the kernel, and convert them back to the current encoding when it reads a file name from the kernel.

A simpler approach could be to write an overlay filesystem with FUSE, that just redirects any filesystem request to another location after converting the file name to/from UTF-8. Ideally you could mount this filesystem in ~/trans, and when an access is made to ~/trans/a/GBK/encoded/path then the FUSE filesystem really accesses /a/UTF-8/encoded/path.

However, the problem with these approaches is: what do you do with files that already exist on your filesystem and are not UTF-8 encoded? You cannot just simply pass them untranslated, because then you don't know how to convert them; you cannot mangle them by translating invalid character sequences to ? because that could create conflicts...

1
1

What you can do is limit the amount of supported locales to only UTF-8 locales.

http://www.fifi.org/cgi-bin/man2html/usr/share/man/man5/locale.gen.5

1
  • 3
    Personally, I wish there's only 1 charset encoding (UTF-8) in the world, but there're legacy application still running, and interoperability between Windows and Linux must be achieved, most people must face this nightmare.
    – LiuYan 刘研
    Jun 22, 2011 at 18:38
1

OEM code page selection is broken in both vanilla unzip and vanilla p7zip. I made patches fixing this issue and there is ppa for Ubuntu with p7zip with this patch applied.

0
0

This issue with zips has been fixed in the most recent far2l file and archive manager. For zip legacy charset detection by far2l to work properly, your system language setting should match the one set on the system where the archive was created (Windows' internal "zip folders" tool uses just the same logic). Also you can do

LANG=zh_CN.UTF-8 far2l
2
  • Are you personally associated with that project?
    – Kusalananda ♦
    Oct 12, 2020 at 8:05
  • I did some research for it, and wrote few patches, but most of the work was done by it's author, elfmz
    – unxed
    Oct 12, 2020 at 8:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .