force bash script to match filenames with special characters in different encodings

Question

I have a bash script to compare filenames in different locations. They have special characters in the filenames that are in two different encodings, so my script doesn't match those particular files.

Is there anything I can do to make bash match them?

They are encoded as utf8 and as TIS-620 Example filenames in utf8 and TIS-620 respectively Löffler and Löffler

example of script:

for i in /dir1/*; do
if [ ! -h "$i" ]; then
[ -d "/dir2/${i##*/}" ]
fi

using unum I can this information:

utf8 version of ö

Octal  Decimal      Hex        HTML    Character   Unicode
0366      246     0xF6      ö    "ö"         LATIN SMALL LETTER O WITH DIAERESIS

TIS-620 version of ö

Octal  Decimal      Hex        HTML    Character   Unicode
0157      111     0x6F      o    "o"         LATIN SMALL LETTER O
01410      776    0x308      ̈    "̈"         COMBINING DIAERESIS

EDIT:

I figured out what causing the mismatch in the first place, a tool using UTF normalisation. I would still like to know how to match the same character in different encodings. Another way of putting it would be to say, how can I use UTF normalisation inline for BASH scripts?

Answer 1

I take that you need to compare some "text" encoded in the codepage of TIS-620 (Thai) with the utf8 encoding (universal) equivalent.

Well, as the most universal encoding (which will encode as many characters as UTF-32) is utf8, we should convert the most local encoding TIS-620 to it.

The usual encoding conversion tool is iconv. Whith that tool, you can do:

$ printf '\xC1' | iconv -f TIS620 -t utf8
ม

And see (if your terminal accepts utf8) the character ม. The character ม has a value of C1 looking at the table in wikipedia's TIS-620.

Or, to "see" the bytes that make that character (in utf8):

$ printf '\xC1' | iconv -f TIS620 -t utf8 | od -vAn -tx1
e0 b8 a1

Which are the 3 bytes that result when encoding the character with the Unicode code point number U0E21 from fileformat or, also at www.utf8-chartable.de:

U+0E21  ม   e0 b8 a1    THAI CHARACTER MO MA

The list of encodings available for TIS620 in iconv are:

$ iconv -l | grep 620
TIS-620//
TIS620-0//
TIS620.2529-1//
TIS620.2533-0//
TIS620//

Choose one that match the encoding of your filenames.

---

However, I fail to find an umlaut ö in thai.
The thai TIS620 page Or even the (very old) translation of thai to ISO/IEC 10646-1:1993 do not show the existence of an o with umlaut in thai.

Could you please re-edit your question?

---

About umlaut

Let's assume that the console/terminal is configured to understand utf8. And, Let's create three filenames in a directory with diferent umlauts.

1. Latin ö (as one unicode code point) (represented as 0xC3 0xB6 in utf8).
   LATIN SMALL LETTER O WITH DIAERESIS (U+00F6)
   Latin ö

   $ printf 'L\xC3\xB6ffler'; echo
   Löffler
   

2. Latin ö (as a letter o followed by a diaresis) (is 0x6F 0xCC 0x88 in utf8).
   COMBINING DIAERESIS (U+0308)
   Diaeresis

   $ printf 'Lo\xCC\x88ffler'; echo
   Löffler
   

3. And: Cyrillic o with diaeresis (is 0xD3 0xA7 in utf-8)
   CYRILLIC SMALL LETTER O WITH DIAERESIS (U+04E7)
   Cyrillic O with diaeresis

   $ printf 'L\xD3\xA7ffler'; echo
   Lӧffler
   

To create the three files with those filenames, you may use:

$ touch $(printf 'L\xC3\xB6ffler Lo\xCC\x88ffler L\xD3\xA7ffler')

A way to list such files is to use a Glob that match (only those files).
In this case, the trailing ffler appears on all files.

$ echo *ffler
Löffler Löffler Lӧffler

What results from that echo could be viewed in detail with:

$ echo *ffler | od -vAn -tx1c
  4c  6f  cc  88  66  66  6c  65  72  20  4c  c3  b6  66  66  6c
   L   o 314 210   f   f   l   e   r       L 303 266   f   f   l
  65  72  20  4c  d3  a7  66  66  6c  65  72  0a
   e   r       L 323 247   f   f   l   e   r  \n

Which just refflects the fact that each is different.

If they are asigned to the positional arguments of the shell:

$ set -- $(echo *ffler)

We can compare each one:

[ "$1" == "$2" ] && echo "Diferent" || echo "Equal"

However, it is reasonable to expect that the first and the second to be equivalent.
But they are different in the way the "composition" is done.
The 'L\xC3\xB6ffler' use the NFC (composed) form.
The 'Lo\xCC\x88ffler' use the NFD (de-composed) form.

You can use uconv (from icu-devtools package)to convert beetwen those two forms.
In decomposed form:

$ echo *ffler | uconv -x any-nfd | od -vAn -tx1c
  4c  6f  cc  88  66  66  6c  65  72  20  4c  6f  cc  88  66  66
   L   o 314 210   f   f   l   e   r       L   o 314 210   f   f
  6c  65  72  20  4c  d0  be  cc  88  66  66  6c  65  72  0a
   l   e   r       L 320 276 314 210   f   f   l   e   r  \n

In pre-composed form:

$  echo *ffler | uconv -x any-nfc | od -vAn -tx1c
  4c  c3  b6  66  66  6c  65  72  20  4c  c3  b6  66  66  6c  65
   L 303 266   f   f   l   e   r       L 303 266   f   f   l   e
  72  20  4c  d3  a7  66  66  6c  65  72  0a
   r       L 323 247   f   f   l   e   r  \n

Now, if we set those values as Positional parameters and compare them:

$ set -- $( echo *ffler | uconv -x any-nfc | od -vAn -tx1c )
$ [ "$1" == "$2" ] && echo "Diferent" || echo "Equal"

The cyrylic character is not equivalent to any of this composition forms.
If you need to convert it, so you can compare that name to the others, you need a tool that understand multi-byte characters.

$ echo *ffler | sed 's/\xd3\xa7/\xc3\xb6/g' | od -vAn -tx1c
  4c  6f  cc  88  66  66  6c  65  72  20  4c  c3  b6  66  66  6c
   L   o 314 210   f   f   l   e   r       L 303 266   f   f   l
  65  72  20  4c  c3  b6  66  66  6c  65  72  0a
   e   r       L 303 266   f   f   l   e   r  \n

And working in only the NFC form:

$ echo *ffler | uconv -x any-nfc | sed 's/\xd3\xa7/\xc3\xb6/g' | od -vAn -tx1c
  4c  c3  b6  66  66  6c  65  72  20  4c  c3  b6  66  66  6c  65
   L 303 266   f   f   l   e   r       L 303 266   f   f   l   e
  72  20  4c  c3  b6  66  66  6c  65  72  0a
   r       L 303 266   f   f   l   e   r  \n

Now the three names are exactly the same.

Is the above something that clarify your real concern?
Is it even close?