I have a text file that contains a large number of records, each on a single line. Some of the records have special characters that have been corrupted, and I'm attempting to find those by looking for multiple sequences of characters higher than x80

Here is a single line sample with the incorrect characters highlighted:

enter image description here

The hex string of interest is:

49 CC 80 C2 B9 6E

When I use GNU Grep, grep --color='auto' -P -n "[\x80-\xFF]" record.txt it matches only part of the line, it matches the superscript 1 (¹) but not the Ì:

enter image description here

Grep doesn't seem to be able to break the combined character+diacritic apart...

What I'd like to do is keep only the lines that have two or more consecutive x80 characters - and to be able to match on actual characters that show up in the hex code - i.e. 49 CC 80 C2 B9 6E seems like it should match something like "[\x80-\xFF]{2,10}" - but this matching does not work.

So, to clarify, when I use this, the line matches:

grep --color='auto' -P -n "[\x80-\xFF]" record.txt

But when I use this, it does not:

grep --color='auto' -P -n "[\x80-\xFF]{2,10}" record.txt

Shouldn't the second one match, too, since the sequence of bytes is CC 80 C2 B9 which is a string of 4 consecutive bytes with the values of x80-xFF?

  • 2
    That regex should match lines containing any high bit characters, though I do not know what "part of the line" means, as grep probably isn't doing anything fancy with encoding. perl -C0 -nle 'print if m/[\x80-\xFF]{2,10}/' appears to match as well, though I only reproduced a subset of your hard-to-copy screenshot for testing. – thrig Feb 12 '16 at 23:04
  • Thanks @thrig. I agree - the screenshot was not ideal - but wasn't sure the best way to share the sample lines without having the characters further scrambled - thoughts? – cwd Feb 16 '16 at 19:41
  • The raw data could be shared as the output of e.g. xxd or hexdump, which in the case of xxd can then have xxd -r run on it to reverse it back to binary. – thrig Feb 16 '16 at 22:10
  • "binary search" has a meaning in computer science. :) – Kaz Oct 7 '16 at 15:44

This could be locale related. If so, using the C (aka POSIX) locale, where characters are bytes, may work:

LC_ALL=C grep --color='auto' -P -n "[\x80-\xFF]{2,10}" record.txt

Grep can be wonky with odd characters.. try:

grep --color='auto' -P -n "[\x80-\xFF]" record.txt |  iconv -f utf-16 -t utf-16

It might get your letters back.. but your colors will be lost. Might be worth mucking with the utf-16 and utf-8.

And make sure your console is able to handle uft-8, and isn't assigned to some ansi setting.

  • How can I add in the {2,10} piece to specify that I want to match only when there are 2-10 characters in the x80-xFF range? I updated my question to clarify. – cwd Feb 16 '16 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.