I have a number of XML files containing sanskrit texts to be converted to tex. Latex has a maximum of 63 characters per word for its hyphenation to work, everything longer than that will not be hyphenated. Now I would like to grep my files for these words, only that grep doesn't appear to be the right tool here. Some of the words use IAST encoding, others Devanāgarī. I suppose a perl one-liner could do that?

  • 3
    perl -lne 'print for /\w{66,}/g' (untested) – Stéphane Chazelas Mar 15 '16 at 12:35
  • @StéphaneChazelas That's going in the right direction, but it seems for \w to work I would need a sanskrit locale? Like this it doesn't catch letters with diacritics, such as ī, and it doesn't handle Devanāgarī script at all. – muk.li Mar 15 '16 at 12:51
  • 3
    How about \S{63,} -- that will include punctuation, but should catch the long words too. – glenn jackman Mar 15 '16 at 12:53
  • 1
    Just add -CD to read the input as utf8. – choroba Mar 15 '16 at 12:55
  • 2
    @muk.li: Ah, so use -CSD to turn stdout to utf8, too. – choroba Mar 15 '16 at 13:05

In an attempt to give this Q a proper answer, based - on - the - comments (heeding Sobrique's note that parsing XML should really be done with an XML parser):

perl -CSD -lne 'print for /\w{63,}/g' input-file-here
| improve this answer | |
  • Actually I had that time already found the solution. The files being XML files didn't really matter. And \w (after switching input and output to utf-8) is better than \S, as otherwise punctuation and parts of the markup are matched, too. – muk.li Jun 6 '16 at 17:31
  • I've updated it to use \w instead of \S -- I'm always a fan of what works! – Jeff Schaller Jun 6 '16 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.