3

I can use the iconv command to "translit" a utf-8 string to an ASCII-only string with characters being replaced with their closest ASCII character. However, my problem is that I need the resulting string to contain exactly the same number of characters (code points) as the source string. For example the British pound (£) character is being replaced with three characters (GBP) or the character æ with the two characters ae. Instead, I want these characters to be replaced with a single character (for example, L or ? or a or e). Is there a method / option for this?

Overview

I am trying to develop an "ASCII-based" find routine for Latin-based text. For example, find 'uco' in '£Húçôz' should return position 3 (I am counting from 1 for the first position) and not 5 (because£ is expanded to 3 characters and not one). Please, note that, what I am trying to develop is much more complex; I have tried to simplify my question as much as possible here.

Also note that, this find routine need not be perfect in matching non-ASCII Latin-based text strings, but the (first) position of the match (if there is one) should point to the correct character (code point) position.

Improve this question
5
  • It seams more like grep than find. Some sort of close-ish match.
    – ctrl-alt-delor
    Jun 24, 2019 at 9:55
  • I think the bit of terminology you're seeking is "(extended) grapheme cluster".
    – Michael Homer
    Jun 24, 2019 at 10:00
  • Searching like this is honestly very hard and you'll need to make a lot of decisions about what you consider "equivalent", but normalization form NFKC is probably also helpful. I don't think there's going to be a useful generic flattening.
    – Michael Homer
    Jun 24, 2019 at 10:02
  • I was also thinking about the idea of combining characters. Unicode knows that ä can be made by combining a combiling-" and an a. So it could know that an ä is related to an a. However doing this for the whole unicode will be a MASSIVE project. If you are doing it for just European languages, then it is probably do-able.
    – ctrl-alt-delor
    Jun 24, 2019 at 10:10
  • That is the reason I am resorting to the iconv routine. It does a good job with the TRANSLIT option. The only problem is in the number of ASCII characters it returns. Returning a single ? will be acceptable for any problematic (for example non-"Latin-like") character.
    – FedKad
    Jun 24, 2019 at 10:24

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .