I have a feeling that it is strings which is thwarting my efforts here. The binary files I want to use strings on unfortunately yield several matches per file, even though it is a 100% rule that exactly one of these contains the string with a NUL character at the end. And THIS is the string I want. I'm also working this way to make sure that false positives are eliminated from the beginning.

Unfortunately, it seems that strings cannot be taught to preserve the null-bytes so I can grep for some_expression\0.

Sample line: (simplified)

$ find . -maxdepth 1 -type f -size +1M -print0 | xargs -0 strings -fwn 3 | grep -w 'XYZ'

If strings could be told to keep the \0 characters, it would even allow something like ... | grep -w 'XYZ[^[:print:]]' at the end of the pipe later. But of course, grep is powerless whenever the preceding command in the pipe has already eliminated the '\0'.

I've even thought of a (fairly ugly) way to tackle this problem by tr'ing each '\0' character to something like '\177' (decimal 255) in the whole file that's being processed. But that would probably yield too many false positives.

Any better solutions out there?

Additional note: Although this question only covers the most common case (NUL), an optimum solution would define as one that is suitable for easy adaptation on other non-printable characters used for string termination as well.

  • I can't test/write it all up right now, but what about piping the files through 'od -c' or xxd, from which you could then grep the corresponding codes? – Jeff Schaller Feb 7 '16 at 1:28
  • Sounds like a good idea, but I should see some sample line first instead of blind fiddling. Plus, it would be great to keep using strings, because with big files it will already have filtered so much garbage before grep even gets its first attempt to process the content. – syntaxerror Feb 7 '16 at 1:31
  • 1
    Would omitting strings and using something like grep -waoP 'XYZ\x00' work? – Mark Plotnick Feb 7 '16 at 11:05
  • @MarkPlotnick Indeed it would, however there appears to be a general problem with \xAA hex codes in the grep regex. Spotted lots of similar questions here on SE. Users occasionally had to work around it by greping for 'XYZ'$'\x00', i. e. using the built-in trickery of some (not all!) shells. Maybe hex codes in grep will only work reliably in GNU grep? Besides: -a is discouraged, as it might wreak havoc on a terminal due to the shell interpreting some of the control codes as actual ones. So these might clear your screen, deface output strings, insult your mother...;-) – syntaxerror Feb 7 '16 at 18:08
  • Yes, some of these options are only supported by GNU grep. -a isn't a problem here, since we give grep the -o option; -a would only cause control codes to be displayed if your grep regexps matched control codes, which they won't. The -P option allows NUL to be specified as \x00 in the regexp, as 4 ordinary characters, no shell magic required. – Mark Plotnick Feb 7 '16 at 19:30

Make grep do the work of strings. If you have GNU grep, pass the -z option to make it read null-delimited records instead of newline-delimited records. This will also match at the end of the file, but that should be ok in practice.

find . -maxdepth 1 -type f -size +1M -print0 |
xargs -0 grep -Eoz '[[:print:]]{3,}$'

If you don't have GNU utilities, pass the file through tr to swap null bytes with newlines. While you're at it, make tr trash non-printable characters.

find . -type d -prune -o -type f -size +1024k -exec sh -c '
  for x; do
    <"$x" tr \\0\\n \\n\\0 | tr -dsC "[:print:]" \\n |
    grep ...
' _ {} +
  • As usual, you're simply a magician. :-O Thank you so much. Had not the slightest idea that tr would even understand a whole set like [[:print:]]instead of mere dumb single characters (i. e. the purpose it would normally serve). Might take me a while to have your partially "golfed" solution broken down to things I can follow, but nevermind: I think I can master that on my own. Thanks again. – syntaxerror Feb 8 '16 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.