0

I'm using a script that has LC_ALL=C set throughout it to optimize data processing, and am trying to use idn2 over a Perl solution to gain increased speeds. Below are data and script samples:

domains.txt

дольщикиспб.рф
шляхтен.рф
สารสกัดจากสมุนไพร.com
google.com

puny.bash

#!/usr/bin/env bash
set -euo pipefail

export LC_ALL=C
cat domains.txt | LC_ALL= idn2
unset LC_ALL

Output

xn--90afmajeumr0f6a.xn--p1ai
xn--e1alhsoq4c.xn--p1ai
xn--12cau1c1a4atlh5dbe1gkg3hzj.com
google.com

This works perfectly. However, when running against many idna domains, one creates this error:

Charset: ANSI_X3.4-1968
input[0] = 0x62
input[1] = 0x69
input[2] = 0x72
input[3] = 0x65
input[4] = 0x79
input[5] = 0x73
input[6] = 0x65
input[7] = 0x6c
input[8] = 0x2d
input[9] = 0x7a
input[10] = 0xc3
input[11] = 0xae
input[12] = 0x72
input[13] = 0x61
input[14] = 0x61
input[15] = 0x74
input[16] = 0x2e
input[17] = 0x63
input[18] = 0x6f
input[19] = 0x6d
idn2: toAscii: could not convert string to UTF-8

And I'm wondering how idn2 can be used to convert any idna domain to ASCII since it's much faster than Perl. Specifically, how would the LC_ALL= idn2 statement in puny.bash be changed to work against any idna domain?

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.