10

While working with a colleague I found a strange issue that seems related to encoding. We're working with some images that have simple enough file names such as city.gif or wine.gif, but as one might expect things get more complicated when using special characters such as é, ë, à. We're also working with Dutch data that has these characters, e.g.café (pub). (We do not have control over the origin of the files.) Here's where the issues start to arise. The following file names are just an example. The issue also occurs for other characters with diacritics.

café-2.png
cafetaria.png
café.png

The first and last item should have an accented e in there (accent aigu, é). That's how it's shown in Linux (CentOS 6 & 7) in a terminal when running ls. But here comes Windows! (Using Windows 10, 64 bit.) When connected on Windows through SSL with our server and then calling ls, the list above looks like this:

café-2.png
cafetaria.png
caf▒.png

As you can hopefully see, the first line still has the accented e é, but the third one doesn't. Instead, I see this character - which is medium shade in unicode (9618 decimal). This is strange in itself. However, when I connect through SFTP with Filezilla (still on Windows) I get to see this:

café-2.png
cafetaria.png
café.png

So now things have turned around: in the first one, é has changed into the sequence and in the third one everything's fine. I found here that this is most likely due to a Latin-1 <-> UTF-8 conversion that went wrong, if I got it right. But that can't be all that's going on, right?

Linux shows everything as we'd expect, Windows shows seemingly inconsistent behaviour depending on the way we view the filename (SSH (putty), or SFTP (filezilla)). Is there a way to 'normalise' these filenames - i.e. edit them -, and make sure that they are all the same on every OS; or at least consistent, and if so, how? UTF-8 is our encoding of choice.

Even though this may same merely an aesthetic issue, it isn't. When trying to download things through SFTP in Windows from our Linux server, I cannot download the files that have the issue mentioned above. Filezilla will throw an error such as Can't download file café-2.png: café-2.png does not exist on the server. Which seems to me that Filezilla reads the directory and the filename, interprets it in some encoding, sends a GET request to the server with its interpretation, but that interpretation differs from the Linux file name so consequently the file is not found.

Ultimately it would be nice if there is a solution available, even though I am also interested in why this happens. Does it occur because the image files were possibly created on different Operating Systems? Does it occur because the Linux server interprets them wrong, or is Windows messing up? Hopefully there is a solution where we can just contact our sysadmin and ask them to turn on a switch in the server config, but I'm afraid it's not as easy as that.

  • 1
    It's a matter of the client (PuTTY, etc), and their configuration, and not relevant to Windows. For PuTTY, that's done in the translation section. – Thomas Dickey Mar 6 '17 at 22:50
  • 2
    It looks kind of like that the é in "café-2.png" is UTF-8 encoded, but the é in "café.png" is ISO-8859-1 encoded. Can you run python -c "import sys; print(repr(sys.argv[1]))" café-2.png and python -c "import sys; print(repr(sys.argv[1]))" café.png? – Oskar Skog Mar 6 '17 at 23:04
  • @OskarSkog I will try that in the morning. But I always thought that file names didn't 'have' an encoding, in other words: it's as the OS wants it. So would that mean the different files were created on different OS's? (We have no control over the origin of the files.) – Bram Vanroy Mar 6 '17 at 23:06
  • On unix like operating systems, the filename is just a string of bytes. The concept of characters come on a higher level. – Oskar Skog Mar 6 '17 at 23:07
  • 1
    Not even close to an answer, or a solution, merely a thought on the avenue to pursue. From the OP it seems that the files may have assorted origins, with no control over the name generated by the source, and it's too late to apply filters to correct incoming file name snafus. The solution is likely to involve running a script on the server that can detect and correct file name errors, possibly even standardizing the character set/code page used for names. Then the OP can use the same code page in Filezilla, or other client, and things will work. Beyond my skills, but a lead to follow, maybe. – Gypsy Spellweaver Mar 12 '17 at 6:46
11

But here comes Windows!

Windows has nothing whatsoever to do with this. You could reproduce this same exact behaviour with a local instance of (say) GNOME Terminal, with appropriately selected terminal encoding and appropriately configured locale for ls, without any Windows being in the picture at all.

The only thing that Windows does is clearly show what is going on here. Your Windows FTP program is taking the bytes in the filenames and displaying them as the relevant code points in code page 1252. This, a single-byte encoding with almost everything above 0x1F a printable glyph, tells us exactly what the bytes in your filenames are.

Your second filename is largely uninformative, but the first and third are telling.

  • The first filename is the byte sequence 63 61 66 c3 a9 2d 32 2e 70 6e 67 — In code page 1252 this is café-2.png. It is also the UTF-8 encoding of café-2.png.
  • The third filename is the byte sequence 63 61 66 e9 2e 70 6e 67 — In code page 1252 this is café.png. It is, however, not a valid UTF-8 encoding. e9 begins an incomplete character encoding sequence.

So what is happening is that the things are are not using code page 1252 but that are using UTF-8, namely your SSH session and your local terminal emulator, are handling the valid UTF-8 in the same way as one another but are handling the invalid UTF-8 in two different ways:

  • The one that is displaying the block graphic is very probably simply using that block graphic as the general replacement output character for invalid UTF-8 sequences.
  • The one that is displaying the letter é is falling back to Code Page 1252 when it encounters an invalid encoding.

Your underlying problem is a system that is somehow generating some filenames encoded as UTF-8 and other filenames encoded in Code Page 1252.

  • I don't agree that Windows has nothing to do with this. It would likely not happen on other Linux. The problem is default encoding, and afaik Windows has(or at least has had) used their CP and not UTF, resulting in this problem happening even on the same OS across countries. You can reproduce this on Linux, but Linux's more consistent in choosing Unicode – MatthewRock Mar 7 '17 at 7:56
  • Hi there! Thanks for the elaborate answer. You focus on what is happening, which is nice: I always like to understand what is happening. But could you perhaps shed a light on why this is happening, and how we can counter the issues that follow from this inconsistency? I have added two paragraphs to clarify what I mean. – Bram Vanroy Mar 7 '17 at 9:11
  • I wonder why both "café"s are shown as the same when they aren't. Does GNU's ls(1) have a ridiculous encoding error handling? – Oskar Skog Mar 7 '17 at 13:33
  • @MatthewRock In this case I think Windows really has nothing to do with it. I'm not happy with most of what M$ does, and willingly admit many of its evils, yet I can't see blame given where none is due. As the answer makes plain, the problem is with the byte values of the names themselves. In this instance Windows exposed the symptom, but isn't the problem. No more than the thermometer is the problem when it show your fever is 104°. The problem originates with whatever processes created the names on the server that has the files the OP is trying to access. – Gypsy Spellweaver Mar 12 '17 at 6:39
  • Could you provide more information and possible solutions? Otherwise I have spent my bounty for nothing. – Bram Vanroy Mar 15 '17 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.