While working with a colleague I found a strange issue that seems related to encoding. We're working with some images that have simple enough file names such as
wine.gif, but as one might expect things get more complicated when using special characters such as
à. We're also working with Dutch data that has these characters, e.g.
café (pub). (We do not have control over the origin of the files.) Here's where the issues start to arise. The following file names are just an example. The issue also occurs for other characters with diacritics.
café-2.png cafetaria.png café.png
The first and last item should have an accented e in there (accent aigu,
é). That's how it's shown in Linux (CentOS 6 & 7) in a terminal when running
ls. But here comes Windows! (Using Windows 10, 64 bit.) When connected on Windows through SSL with our server and then calling
ls, the list above looks like this:
café-2.png cafetaria.png caf▒.png
As you can hopefully see, the first line still has the accented e
é, but the third one doesn't. Instead, I see
▒ this character - which is
medium shade in unicode (9618 decimal). This is strange in itself. However, when I connect through SFTP with Filezilla (still on Windows) I get to see this:
cafÃ©-2.png cafetaria.png café.png
So now things have turned around: in the first one,
é has changed into the sequence and in the third one everything's fine. I found here that this is most likely due to a Latin-1 <-> UTF-8 conversion that went wrong, if I got it right. But that can't be all that's going on, right?
Linux shows everything as we'd expect, Windows shows seemingly inconsistent behaviour depending on the way we view the filename (SSH (putty), or SFTP (filezilla)). Is there a way to 'normalise' these filenames - i.e. edit them -, and make sure that they are all the same on every OS; or at least consistent, and if so, how?
UTF-8 is our encoding of choice.
Even though this may same merely an aesthetic issue, it isn't. When trying to download things through SFTP in Windows from our Linux server, I cannot download the files that have the issue mentioned above. Filezilla will throw an error such as
Can't download file cafÃ©-2.png: cafÃ©-2.png does not exist on the server. Which seems to me that Filezilla reads the directory and the filename, interprets it in some encoding, sends a GET request to the server with its interpretation, but that interpretation differs from the Linux file name so consequently the file is not found.
Ultimately it would be nice if there is a solution available, even though I am also interested in why this happens. Does it occur because the image files were possibly created on different Operating Systems? Does it occur because the Linux server interprets them wrong, or is Windows messing up? Hopefully there is a solution where we can just contact our sysadmin and ask them to turn on a switch in the server config, but I'm afraid it's not as easy as that.