I like to find international text, like "müller", or even chinese text, in files, no matter which unicode representation and case is used.

I.e, it should not matter whether the "ü" is stored in NFC or NFD format.

I'd think that grep could do this, but at least the version available on macOS 10.13.6 finds only one normalization form.

I'd expect that, with algorithms like Aho–Corasick that can search for alternatives of letters in words, it should not be so difficult to build a tree that includes every alternative form of a UTF8 character and then search for all combinations. Yet, at least the grep I tried doesn't do that.

Are there implementations of grep or other text find tools that can do this properly?

  • Is a pipeline through uconv reasonable? – Fox Apr 28 at 14:22
  • 1
    You mean to preprocess the files I want to search? Maybe. The files are binary, and maybe lots of MB in size. Will that be fairly fast compared to the following find operation? Don't want this to go down by factor 10, but factor 2 would be okay. – Thomas Tempelmann Apr 28 at 15:12
  • I think you'll gonna need to match every form of the matched text, in case it's short, it wouldn't be too hard to find all its possible form. Trying to NFC all the data wouldn't work because NFC algorithm is not additive, NFC a large text segment is super inefficient – 炸鱼薯条德里克 Apr 28 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.