1

I'm trying to find the offset of a hex pattern in a file. This works for one specific value:

$ grep -obUaP -m1 "\x00\x50\x53\x46\x01\x01\x00\x00\x34\x01\x00\x00" file.bin
3088:PSF4

However, this pattern includes a few bytes that will change, so I need to include wildcards in my grep. I can't figure out how to do that. Here's everything I've tried so far:

  • \x.., \x., .., and every similar form I can think of does not match
  • \x[0-9][0-9] does not match
  • \x.* does not match
  • just .* (ie., \x00.*\x01) does match, but it's greedy and matches more than the pattern

Probably overlooking something silly, but I'm running into a wall here.

How do you specify a wildcard in hex, or at least when using grep with perl-regex to search for hex?

5
  • 1
    Using text tools to process binary data never ends well. Will the grep "Match any single character", "." work? "\x01.\x03".
    – waltinator
    Sep 16 at 22:14
  • @waltinator That sounds like a reasonable solution. Does . match something like \x00 (nul) too, or possibly some multi-byte character?
    – Kusalananda
    Sep 16 at 22:20
  • 1
    If you're using GNU grep, the -U option has no effect on non-MS-DOS/Windows systems. Also, using -U with -a seems odd.
    – Kusalananda
    Sep 16 at 22:25
  • @waltinator - you nailed it. I could sworn I tried that, but apparently not. Searching for this, matches the pattern in all my files: "\x00\x50\x53\x46..\x00\x00..\x00\x00". Thanks! If you post as an answer I'll be happy to accept. And while I don't disagree with your assessment about grep and binary files, it's the best tool I know of for this job and does the trick, though I'm certainly open to other suggestions.
    – Jared
    Sep 16 at 22:36
  • @Kusalananda - agree with -U and -a being an odd combination. I had picked this up from cobbling together answers about searching for offsets in binary files a while back and it worked, so didn't question it. Just verified that it works without the -U, so going with that now. Appreciate the heads up.
    – Jared
    Sep 16 at 22:40
2

grep -P '\xAB' doesn't look for a hex character. There is no such thing as a hex character. \xAB is PCRE syntax to match a character whose codepoint value expressed in hexadecimal is 0xAB (171 in decimal).

codepoint here would be the Unicode codepoint in locales that use UTF-8 and byte value in locales that use a single byte charset (GNU grep -P doesn't support multibyte charsets other than UTF-8).

So \xAB would match on the U+00AB character («) in a UTF-8 locale (where that character is encoded on 2 bytes: 0xc2 and 0xab) and the 0xAB byte in single-bytes locales (for instance, which represents the Ћ in a locale using the iso8859-5 charset).

If you want to match on byte value, you should make sure the locale uses a single-byte charset, the C locale is probably your best bet.

LC_ALL=C grep -P '\xAB'

matches on the 0xAB (171) byte, regardless of what character if any it represents in any charset.

To match on any single byte, again, you can use . (assuming the C locale or any local with single byte per character charset).

To match on a byte value within a range, as @Angle115 already said: [\x01-\x45] (here for byte values 1 to 0x45 / 69)

But bear in mind that grep matches on the contents of text lines¹, so it will never find the newline character which is the line delimiter and, regardless of the locale always has value 0x0A² (10 in decimal).

So LC_ALL=C grep -P '\x23.\xab' would match on a sequence of 3 bytes, the first one with value 0x23, the second with any value except 0xA and the third one with value 0xAB.

To be able to search for bytes with arbitrary values including 0xA, you'd need to treat the whole input as a whole, not one line or nul-delimited record at a time like grep does.

For that, you could use pcregrep with its -M (multiline) option along with the (?s) flag (for newline not to be treated specially by .) or use perl with its slurp-mode:

LC_ALL=C pcregrep --file-offsets -Ma '(?s)\x23.\xab' < file

(pcregrep doesn't have a -b option, --file-offsets which prints offset and length is probably the closest).

perl -l -0777 -ne 'print "$-[0]:$_" while /\x23.\xab/gs' < file

Or:

perl -l -0777 -ne 'print $-[0] if /\x23.\xab/s' < file

To only print the byte offset of the first match.

perl loads the whole file in memory, pcregrep doesn't but has internal limits that would likely prevent you from processing files where 0xA bytes are far apart.


¹ or NUL-delimited records with --null/-z

¹ on ASCII based systems. I don't even know if libpcre was ever ported to EBCDIC systems, I doubt many people will ever come across some of those these days.

1
  • Thanks for the detailed response and clarifying some terminology. In my script I am setting LC_ALL=C (have run into that issue previously), just didn't think to show that here. Also very interested in what you noted about the 0x0A issue, which I also ran into after moving forward with @waltinator's answer. Took a little while to connect the dots on that; eventually realized what was happening, but couldn't come up with a good way to match, so ended up having to refactor a bit to workaround that. Very good to know about pcregrep -M in the future if needed. Thanks!
    – Jared
    Sep 20 at 22:15
1

You can include a rang instead of using a wild card to match all ASCII characters, like so:

grep -Pe '\x00\x50\x53\x46[\x00-\x7F]\x01\x00\x00\x34\x01\x00\x00'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.