I have a bash script to compare filenames in different locations. They have special characters in the filenames that are in two different encodings, so my script doesn't match those particular files.

Is there anything I can do to make bash match them?

They are encoded as utf8 and as TIS-620 Example filenames in utf8 and TIS-620 respectively Löffler and Löffler

example of script:

for i in /dir1/*; do
if [ ! -h "$i" ]; then
[ -d "/dir2/${i##*/}" ]

using unum I can this information:

utf8 version of ö

Octal  Decimal      Hex        HTML    Character   Unicode
0366      246     0xF6      ö    "ö"         LATIN SMALL LETTER O WITH DIAERESIS

TIS-620 version of ö

Octal  Decimal      Hex        HTML    Character   Unicode
0157      111     0x6F      o    "o"         LATIN SMALL LETTER O
01410      776    0x308      ̈    "̈"         COMBINING DIAERESIS


I figured out what causing the mismatch in the first place, a tool using UTF normalisation. I would still like to know how to match the same character in different encodings. Another way of putting it would be to say, how can I use UTF normalisation inline for BASH scripts?

  • 4
    Your question is very unclear. Are the special characters in the file names or their contents? You should edit the question to include examples with these characters. Jul 10 '17 at 9:14
  • i made the question more clear about the specific characters and that the question is specifically in reference to filenames with different encodings. can you check it?
    – jakethedog
    Jul 10 '17 at 9:30
  • That clarifies the question, alright. I pretty much just use UTF-8 for everything (when not using Windows) so I can't help answer the question but the following questions seem to have relevant information: unix.stackexchange.com/questions/39175/… unix.stackexchange.com/questions/2089/… Jul 10 '17 at 9:37

I take that you need to compare some "text" encoded in the codepage of TIS-620 (Thai) with the utf8 encoding (universal) equivalent.

Well, as the most universal encoding (which will encode as many characters as UTF-32) is utf8, we should convert the most local encoding TIS-620 to it.

The usual encoding conversion tool is iconv. Whith that tool, you can do:

$ printf '\xC1' | iconv -f TIS620 -t utf8

And see (if your terminal accepts utf8) the character ม. The character has a value of C1 looking at the table in wikipedia's TIS-620.

Or, to "see" the bytes that make that character (in utf8):

$ printf '\xC1' | iconv -f TIS620 -t utf8 | od -vAn -tx1
e0 b8 a1

Which are the 3 bytes that result when encoding the character with the Unicode code point number U0E21 from fileformat or, also at www.utf8-chartable.de:

U+0E21  ม   e0 b8 a1    THAI CHARACTER MO MA

The list of encodings available for TIS620 in iconv are:

$ iconv -l | grep 620

Choose one that match the encoding of your filenames.

However, I fail to find an umlaut ö in thai.
The thai TIS620 page Or even the (very old) translation of thai to ISO/IEC 10646-1:1993 do not show the existence of an o with umlaut in thai.

Could you please re-edit your question?

About umlaut

Let's assume that the console/terminal is configured to understand utf8. And, Let's create three filenames in a directory with diferent umlauts.

  1. Latin ö (as one unicode code point) (represented as 0xC3 0xB6 in utf8).
    Latin ö

    $ printf 'L\xC3\xB6ffler'; echo
  2. Latin ö (as a letter o followed by a diaresis) (is 0x6F 0xCC 0x88 in utf8).

    $ printf 'Lo\xCC\x88ffler'; echo
  3. And: Cyrillic o with diaeresis (is 0xD3 0xA7 in utf-8)
    Cyrillic O with diaeresis

    $ printf 'L\xD3\xA7ffler'; echo

To create the three files with those filenames, you may use:

$ touch $(printf 'L\xC3\xB6ffler Lo\xCC\x88ffler L\xD3\xA7ffler')

A way to list such files is to use a Glob that match (only those files).
In this case, the trailing ffler appears on all files.

$ echo *ffler
Löffler Löffler Lӧffler

What results from that echo could be viewed in detail with:

$ echo *ffler | od -vAn -tx1c
  4c  6f  cc  88  66  66  6c  65  72  20  4c  c3  b6  66  66  6c
   L   o 314 210   f   f   l   e   r       L 303 266   f   f   l
  65  72  20  4c  d3  a7  66  66  6c  65  72  0a
   e   r       L 323 247   f   f   l   e   r  \n

Which just refflects the fact that each is different.

If they are asigned to the positional arguments of the shell:

$ set -- $(echo *ffler)

We can compare each one:

[ "$1" == "$2" ] && echo "Diferent" || echo "Equal"

However, it is reasonable to expect that the first and the second to be equivalent.
But they are different in the way the "composition" is done.
The 'L\xC3\xB6ffler' use the NFC (composed) form.
The 'Lo\xCC\x88ffler' use the NFD (de-composed) form.

You can use uconv (from icu-devtools package)to convert beetwen those two forms.
In decomposed form:

$ echo *ffler | uconv -x any-nfd | od -vAn -tx1c
  4c  6f  cc  88  66  66  6c  65  72  20  4c  6f  cc  88  66  66
   L   o 314 210   f   f   l   e   r       L   o 314 210   f   f
  6c  65  72  20  4c  d0  be  cc  88  66  66  6c  65  72  0a
   l   e   r       L 320 276 314 210   f   f   l   e   r  \n

In pre-composed form:

$  echo *ffler | uconv -x any-nfc | od -vAn -tx1c
  4c  c3  b6  66  66  6c  65  72  20  4c  c3  b6  66  66  6c  65
   L 303 266   f   f   l   e   r       L 303 266   f   f   l   e
  72  20  4c  d3  a7  66  66  6c  65  72  0a
   r       L 323 247   f   f   l   e   r  \n

Now, if we set those values as Positional parameters and compare them:

$ set -- $( echo *ffler | uconv -x any-nfc | od -vAn -tx1c )
$ [ "$1" == "$2" ] && echo "Diferent" || echo "Equal"

The cyrylic character is not equivalent to any of this composition forms.
If you need to convert it, so you can compare that name to the others, you need a tool that understand multi-byte characters.

$ echo *ffler | sed 's/\xd3\xa7/\xc3\xb6/g' | od -vAn -tx1c
  4c  6f  cc  88  66  66  6c  65  72  20  4c  c3  b6  66  66  6c
   L   o 314 210   f   f   l   e   r       L 303 266   f   f   l
  65  72  20  4c  c3  b6  66  66  6c  65  72  0a
   e   r       L 303 266   f   f   l   e   r  \n

And working in only the NFC form:

$ echo *ffler | uconv -x any-nfc | sed 's/\xd3\xa7/\xc3\xb6/g' | od -vAn -tx1c
  4c  c3  b6  66  66  6c  65  72  20  4c  c3  b6  66  66  6c  65
   L 303 266   f   f   l   e   r       L 303 266   f   f   l   e
  72  20  4c  c3  b6  66  66  6c  65  72  0a
   r       L 303 266   f   f   l   e   r  \n

Now the three names are exactly the same.

Is the above something that clarify your real concern?
Is it even close?

  • Thanks for providing a detailed response. Can you help me with my problem though? It would not surprise me if the encoding is falsely reported as TIS620. It is kind of beside the point. The question is not how to identify the encoding, or change the encoding. It is how to match two filenames that appear to be the same, but because of different encoding, do not.
    – jakethedog
    Jul 13 '17 at 7:17
  • @jakethedog Could you provide an example of those two filenames? It is quite hard to solve a problem without a clear example of it. I'll provide a way to convert the umlaut beetwen its two representations in UNICODE. Maybe that is your real problem.
    – ImHere
    Jul 13 '17 at 17:27
  • @jakethedog I added a full section to deal with the normalization forms. Maybe it is closer to what you ask. Please give some comment as feedback.
    – ImHere
    Jul 13 '17 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.