2

I'm trying to access from a MAC (Using Finder, or the commands 'find' or 'ls') some folders whose names contain accented chars. Some of those folders are fully visible on the MAC, some not. Here is what I did and what I found:

  • I use the "lftp mirror" command to get folders from a server to my Synology NAS.
  • Lftp is configured to process file names in UTF-8
  • When I try to access some of those folders that contain accented chars from the MAC (using the Finder or even the commands 'ls' or 'find'), the folder is not reachable.
  • Using the GUI from the NAS, I noticed that the folders in question have names with accented chars made of two different chars (When you delete an 'é' for example, you first delete the accent and then the 'e').
  • Folders whose names contain simple accented chars are OK.
  • When I copy those files directly to my MAC (using rsync with format conversion from UTF-8 to UTF-8 MAC), everything is OK. I even suspect I do not need to use the conversion command and simply copy the files (TBC).

So, question is 'simple': do I need to systematically convert the files before using them, or is there a better way.

Additionally, I do not want to convert the files in the directory where I am mirroring my remote server. I will need to copy the renamed files to another location to avoid issues with the mirror command.

2
  • To me the big question would be how those special characters came to pass and made it onto your NAS. Did the files originate on a system that e.g. uses a European Windows-Character set rather than UTF-8?
    – tink
    May 23, 2023 at 21:10
  • Thanks Tink. This, I need to check, together with the kind of OS running on that server.
    – Eric31
    May 25, 2023 at 7:58

1 Answer 1

2

I think you've probably hit a Unicode normalization issue, but I'm not quite sure how, exactly.

My guess is that rsync is decomposing composed characters for macOS, but lftp isn't. You can confirm this by doing a hex dump of an affected filename on the NAS and on your Mac. Here's an example on my Linux server:

> ls -l
total 1
-rw-rw-r-- 1 sam sam 0 May 24 11:16 NFC Pokémon
-rw-rw-r-- 1 sam sam 0 May 24 11:16 NFD Pokémon
> find . -name 'NFC*' | hexdump -C
00000000  2e 2f 4e 46 43 20 50 6f  6b c3 a9 6d 6f 6e 0a     |./NFC Pok..mon.|
0000000f
> find . -name 'NFD*' | hexdump -C
00000000  2e 2f 4e 46 44 20 50 6f  6b 65 cc 81 6d 6f 6e 0a  |./NFD Poke..mon.|
00000010

In the NFC filename “é” is represented by the byte sequence c3 a9, which is the UTF-8 encoding of the character U+00E9 LATIN SMALL LETTER E WITH ACUTE. In the NFD filename, “é” is represented by the byte sequence 65 cc 81, which is two UTF-8 characters. 65 decodes to U+0065 LATIN SMALL LETTER E, and cc 81 decodes to U+0301 COMBINING ACUTE ACCENT.

(I used find to print the filename here because I have a superstitious distrust of how shells handle filename encoding and globbing.)

Anyway, you can use this method to figure out how exactly the filenames are encoded on your NAS, as well as what exactly rsync and lftp are doing (or not doing) to the filenames when saving them to your Mac.

Regardless of what exactly is going on, I think most of the time you can probably fix things by composing all your filenames. I do this on my server on a semi-regular basis:

convmv -r -f utf-8 -t utf-8 --nfc /data/AUDIO/Library

N.B.: If you do this to files that are currently being seeded in a torrent client, it might break things. I think it depends on the client. Other software that depends on the exact byte sequences in filenames could also lose track of files you renormalize.

3
  • Many thanks Shammy. I tried what you suggest and found out that the "é" char seems to be encoded as 65 cc 81 on MAC and on Syno. What is strange is that with same encoding, the behaviour is different: on one case MAC cannot acces it, on the other case it can. On Syno, I did not find the hexdump command, and uses the "od -A n -t x1" command instead. Not sure it produces the same result.
    – Eric31
    May 25, 2023 at 12:24
  • Yeah, that od command gives the same result for me. Have you compared the encoding on you Mac when you did the transfer with rsync vs lftp?
    – smammy
    May 26, 2023 at 12:38
  • Yes, and this is where I'm a bit puzzled: Encoding on the Syno (With two separated chars forming the "é") and on the Mac (The version I transfered with rsync, and not the original version that is not reachable on the MAC) are identical (65 cc 81, corresponding in UTF8 as lowercase e + "combining acute accent").
    – Eric31
    May 28, 2023 at 8:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .