I'm using a script that has LC_ALL=C set throughout it to optimize data processing, and am trying to use idn2 over a Perl solution to gain increased speeds. Below are data and script samples:
domains.txt
дольщикиспб.рф
шляхтен.рф
สารสกัดจากสมุนไพร.com
google.com
puny.bash
#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C
cat domains.txt | LC_ALL= idn2
unset LC_ALL
Output
xn--90afmajeumr0f6a.xn--p1ai
xn--e1alhsoq4c.xn--p1ai
xn--12cau1c1a4atlh5dbe1gkg3hzj.com
google.com
This works perfectly. However, when running against many idna domains, one creates this error:
Charset: ANSI_X3.4-1968
input[0] = 0x62
input[1] = 0x69
input[2] = 0x72
input[3] = 0x65
input[4] = 0x79
input[5] = 0x73
input[6] = 0x65
input[7] = 0x6c
input[8] = 0x2d
input[9] = 0x7a
input[10] = 0xc3
input[11] = 0xae
input[12] = 0x72
input[13] = 0x61
input[14] = 0x61
input[15] = 0x74
input[16] = 0x2e
input[17] = 0x63
input[18] = 0x6f
input[19] = 0x6d
idn2: toAscii: could not convert string to UTF-8
And I'm wondering how idn2 can be used to convert any idna domain to ASCII since it's much faster than Perl. Specifically, how would the LC_ALL= idn2 statement in puny.bash be changed to work against any idna domain?
