How to grep for hex pattern w/ wildcards?

Question

I'm trying to find the offset of a hex pattern in a file. This works for one specific value:

$ grep -obUaP -m1 "\x00\x50\x53\x46\x01\x01\x00\x00\x34\x01\x00\x00" file.bin
3088:PSF4

However, this pattern includes a few bytes that will change, so I need to include wildcards in my grep. I can't figure out how to do that. Here's everything I've tried so far:

• \x.., \x., .., and every similar form I can think of does not match
• \x[0-9][0-9] does not match
• \x.* does not match
• just .* (ie., \x00.*\x01) does match, but it's greedy and matches more than the pattern

Probably overlooking something silly, but I'm running into a wall here.

How do you specify a wildcard in hex, or at least when using grep with perl-regex to search for hex?

Answer 1

grep -P '\xAB' doesn't look for a hex character. There is no such thing as a hex character. \xAB is PCRE syntax to match a character whose codepoint value expressed in hexadecimal is 0xAB (171 in decimal).

codepoint here would be the Unicode codepoint in locales that use UTF-8 and byte value in locales that use a single byte charset (GNU grep -P doesn't support multibyte charsets other than UTF-8).

So \xAB would match on the U+00AB character («) in a UTF-8 locale (where that character is encoded on 2 bytes: 0xc2 and 0xab) and the 0xAB byte in single-bytes locales (for instance, which represents the Ћ in a locale using the iso8859-5 charset).

If you want to match on byte value, you should make sure the locale uses a single-byte charset, the C locale is probably your best bet.

LC_ALL=C grep -P '\xAB'

matches on the 0xAB (171) byte, regardless of what character if any it represents in any charset.

To match on any single byte, again, you can use . (assuming the C locale or any local with single byte per character charset).

To match on a byte value within a range, as @Angle115 already said: [\x01-\x45] (here for byte values 1 to 0x45 / 69)

But bear in mind that grep matches on the contents of text lines¹, so it will never find the newline character which is the line delimiter and, regardless of the locale always has value 0x0A² (10 in decimal).

So LC_ALL=C grep -P '\x23.\xab' would match on a sequence of 3 bytes, the first one with value 0x23, the second with any value except 0xA and the third one with value 0xAB.

To be able to search for bytes with arbitrary values including 0xA, you'd need to treat the whole input as a whole, not one line or nul-delimited record at a time like grep does.

For that, you could use pcregrep with its -M (multiline) option along with the (?s) flag (for newline not to be treated specially by .) or use perl with its slurp-mode:

LC_ALL=C pcregrep --file-offsets -Ma '(?s)\x23.\xab' < file

(pcregrep doesn't have a -b option, --file-offsets which prints offset and length is probably the closest).

perl -l -0777 -ne 'print "$-[0]:$_" while /\x23.\xab/gs' < file

Or:

perl -l -0777 -ne 'print $-[0] if /\x23.\xab/s' < file

To only print the byte offset of the first match.

perl loads the whole file in memory, pcregrep doesn't but has internal limits that would likely prevent you from processing files where 0xA bytes are far apart.

---

^{¹ or NUL-delimited records with --null/-z}

^{¹ on ASCII based systems. I don't even know if libpcre was ever ported to EBCDIC systems, I doubt many people will ever come across some of those these days.}

Answer 2

You can include a rang instead of using a wild card to match all ASCII characters, like so:

grep -Pe '\x00\x50\x53\x46[\x00-\x7F]\x01\x00\x00\x34\x01\x00\x00'