I have a text file that contains a large number of records, each on a single line. Some of the records have special characters that have been corrupted, and I'm attempting to find those by looking for multiple sequences of characters higher than x80
Here is a single line sample with the incorrect characters highlighted:
The hex string of interest is:
49 CC 80 C2 B9 6E
When I use GNU Grep, grep --color='auto' -P -n "[\x80-\xFF]" record.txt
it matches only part of the line, it matches the superscript 1 (¹
) but not the Ì
:
Grep doesn't seem to be able to break the combined character+diacritic apart...
What I'd like to do is keep only the lines that have two or more consecutive x80
characters - and to be able to match on actual characters that show up in the hex code - i.e. 49 CC 80 C2 B9 6E
seems like it should match something like "[\x80-\xFF]{2,10}"
- but this matching does not work.
So, to clarify, when I use this, the line matches:
grep --color='auto' -P -n "[\x80-\xFF]" record.txt
But when I use this, it does not:
grep --color='auto' -P -n "[\x80-\xFF]{2,10}" record.txt
Shouldn't the second one match, too, since the sequence of bytes is CC 80 C2 B9
which is a string of 4 consecutive bytes with the values of x80-xFF
?
grep
probably isn't doing anything fancy with encoding.perl -C0 -nle 'print if m/[\x80-\xFF]{2,10}/'
appears to match as well, though I only reproduced a subset of your hard-to-copy screenshot for testing.xxd
orhexdump
, which in the case ofxxd
can then havexxd -r
run on it to reverse it back to binary.