How can I do a binary search of a text file for a specific character string?

Question

I have a text file that contains a large number of records, each on a single line. Some of the records have special characters that have been corrupted, and I'm attempting to find those by looking for multiple sequences of characters higher than x80

Here is a single line sample with the incorrect characters highlighted:

The hex string of interest is:

49 CC 80 C2 B9 6E

When I use GNU Grep, grep --color='auto' -P -n "[\x80-\xFF]" record.txt it matches only part of the line, it matches the superscript 1 (¹) but not the Ì:

Grep doesn't seem to be able to break the combined character+diacritic apart...

What I'd like to do is keep only the lines that have two or more consecutive x80 characters - and to be able to match on actual characters that show up in the hex code - i.e. 49 CC 80 C2 B9 6E seems like it should match something like "[\x80-\xFF]{2,10}" - but this matching does not work.

So, to clarify, when I use this, the line matches:

grep --color='auto' -P -n "[\x80-\xFF]" record.txt

But when I use this, it does not:

grep --color='auto' -P -n "[\x80-\xFF]{2,10}" record.txt

Shouldn't the second one match, too, since the sequence of bytes is CC 80 C2 B9 which is a string of 4 consecutive bytes with the values of x80-xFF?

Answer 1

This could be locale related. If so, using the C (aka POSIX) locale, where characters are bytes, may work:

LC_ALL=C grep --color='auto' -P -n "[\x80-\xFF]{2,10}" record.txt

Answer 2

Grep can be wonky with odd characters.. try:

grep --color='auto' -P -n "[\x80-\xFF]" record.txt |  iconv -f utf-16 -t utf-16

It might get your letters back.. but your colors will be lost. Might be worth mucking with the utf-16 and utf-8.

And make sure your console is able to handle uft-8, and isn't assigned to some ansi setting.