Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have some database dumps from a Windows system on my box. They are text files. I'm using cygwin to grep through them. These appear to be plain text files; I open them with text editors such as notepad and wordpad and they look legible. However, when I run grep on them, it will say binary file foo.txt matches.

I have noticed that the files contain some ascii NUL characters, which I believe are artifacts from the database dump.

So what makes grep consider these files to be binary? The NUL character? Is there a flag on the filesystem? What do I need to change to get grep to show me the line matches?

share|improve this question
    
--null-data may be useful if NUL is the delimiter. – Steve-o Sep 1 '11 at 13:27
up vote 44 down vote accepted

If there is a NUL character anywhere in the file, grep will consider it as a binary file.

There might a workaround like this cat file | tr -d '\000' | yourgrep to eliminate all null first, and then to search through file.

share|improve this answer
32  
... or use -a/--text, at least with GNU grep. – derobert Nov 26 '12 at 20:44
    
@derobert: actually, on some (older) systems, grep see lines, but its output will truncate each matching line at the first NUL (probably becauses it calls C's printf and gives it the matched line?). On such a system a grep cmd .sh_history will return as many empty lines as there are lines matching 'cmd', as each line of sh_history has a specific format with a NUL at the begining of each line. (but your comment "at least on GNU grep" probably comes true. I don't have one at hand right now to test, but I expect they handle this nicely) – Olivier Dulac Nov 25 '13 at 11:46
    
+1 for using -a / --text with GNU grep, because you can mix this easily with recursive search, e.g. egrep -r -a mystring . Thanks @derobert – phil_w Jul 14 '15 at 19:08
2  
Is the presence of a NUL character the only criteria? I doubt it. It's probably smarter than that. Anything falling outside the Ascii 32-126 range would be my guess, but we'd have to look at the source code to be sure. – Michael Martinez Aug 14 '15 at 16:58
1  
I had a file which grep on cygwin considered binary because it had a long dash (0x96) instead of a regular ASCII hyphen/minus (0x2d). I guess this answer resolved the OP's issue, but it appears it is incomplete. – cp.engr Feb 15 at 16:15

You can use the strings utility to extract the text content from any file and then pipe it through grep, like this: strings file | grep pattern.

share|improve this answer
1  
Ideal for grepping log files that might be partly corrupted – Hannes R. Feb 27 '15 at 7:43

grep -a worked for me:

$ grep --help
[...]
 -a, --text                equivalent to --binary-files=text
share|improve this answer

The file /etc/magic or /usr/share/misc/magic has a list of sequences that the command file uses for determining the file type.

Note that binary may just be a fallback solution. Sometimes files with strange encoding are considered binary too.

grep on Linux has some options to handle binary files like --binary-files or -U / --binary

share|improve this answer
    
More precisely, encoding error according to C99's mbrlen(). Example and source interpretation at: unix.stackexchange.com/a/276028/32558 – Ciro Santilli 巴拿馬文件 六四事件 法轮功 Apr 12 at 20:51

GNU grep 2.24 RTFS

Conclusion: 2 and 2 cases only:

  • NUL, e.g. printf 'a\0' | grep 'a'

  • encoding error according to the C99 mbrlen(), e.g.:

    export LC_CTYPE='en_US.UTF-8'
    printf 'a\x80' | grep 'a'
    

    because \x80 cannot be the first byte of an UTF-8 Unicode point: https://en.wikipedia.org/wiki/UTF-8#Description

Only up to the first buffer read

As mentioned by Stéphane Chazelas, those checks are only done up to the first buffer read, so if a NUL or encoding error happens in the middle of a very large file, it might be grepped anyways.

I imagine this is for performance reasons.

E.g.: this prints the line:

printf '%10000000s\n\x80a' | grep 'a'

but this does not:

printf '%10s\n\x80a' | grep 'a'

The actual buffer size depends on how the file is read. E.g. compare:

export LC_CTYPE='en_US.UTF-8'
(printf '\n\x80a') | grep 'a'
(printf '\n'; sleep 1; printf '\x80a') | grep 'a'

With the sleep, the first line gets passed to grep even if it is only 1 byte long because the process goes to sleep, and the second read does not check if the file is binary.

RTFS

git clone git://git.savannah.gnu.org/grep.git 
cd grep
git checkout v2.24

Find where the stderr error message is encoded:

git grep 'Binary file'

Leads us to /src/grep.c:

if (!out_quiet && (encoding_error_output
                    || (0 <= nlines_first_null && nlines_first_null < nlines)))
    {
    printf (_("Binary file %s matches\n"), filename);

If those variables were well named, we basically reached the conclusion.

encoding_error_output

Quick grepping for encoding_error_output shows that the only code path that can modify it goes through buf_has_encoding_errors:

clen = mbrlen (p, buf + size - p, &mbs);
if ((size_t) -2 <= clen)
  return true;

then just man mbrlen.

nlines_first_null and nlines

Initialized as:

intmax_t nlines_first_null = -1;
nlines = 0;

so when a null is found 0 <= nlines_first_null becomes true.

TODO when can nlines_first_null < nlines ever be false? I got lazy.

POSIX

Does not define binary options http://pubs.opengroup.org/onlinepubs/9699919799/utilities/grep.html , and GNU grep does not document it, so RTFS is the only way.

share|improve this answer
1  
Impressive explication! – user394 Apr 13 at 2:02
1  
Note that the check for valid UTF-8 only happens in UTF-8 locales. Also note that the check is only done on the first buffer read from the file which for a regular file seems to be 32768 bytes on my system, but for a pipe or socket can be as small as one byte. Compare (printf '\n\0y') | grep y with (printf '\n'; sleep 1; printf '\0y') | grep y for instance. – Stéphane Chazelas Apr 13 at 12:18
    
@StéphaneChazelas "Note that the check for valid UTF-8 only happens in UTF-8 locales": do you mean about the export LC_CTYPE='en_US.UTF-8' as in my example, or something else? Buf read: amazing example, added to answer. You have obviously read the source more than me, reminds me of those hacker koans "The student was enlightened" :-) – Ciro Santilli 巴拿馬文件 六四事件 法轮功 Apr 13 at 13:05
1  
I didn't look into great detail either, but did very recently – Stéphane Chazelas Apr 13 at 13:09
1  
@CiroSantilli巴拿馬文件六四事件法轮功 what version of GNU grep did you test against? – jrw32982 Jun 8 at 23:33

One of my text files was suddenly being seen as binary by grep:

$ file foo.txt
foo.txt: ISO-8859 text

Solution was to convert it by using iconv:

iconv -t UTF-8 -f ISO-8859-1 foo.txt > foo_new.txt
share|improve this answer
1  
This happened to me as well. In particular, the cause was an ISO-8859-1-encoded non-breaking space, which I had to replace with a regular space in order to get grep to search in the file. – Gallaecio Jun 9 '15 at 13:50
3  
grep 2.21 treats ISO-8859 text files as if they are binary, add export LC_ALL=C before grep command. – netawater Aug 17 '15 at 2:52

Actually answering the question "What makes grep consider a file to be binary?", you can use iconv:

$ iconv < myfile.java
iconv: (stdin):267:70: cannot convert

In my case there were Spanish characters that showed up correctly in text editors but grep considered them as binary; iconv output pointed me to the line and column numbers of those characters

In the case of NUL characters, iconv will consider them normal and will not print that kind of output so this method is not suitable

share|improve this answer

I had the same problem. I used vi -b [filename] to see the added characters. I found the control characters ^@ and ^M. Then in vi type :1,$s/^@//g to remove the ^@ characters. Repeat this command for ^M.

Warning: To get the "blue" control characters press Ctrl+v then Ctrl+M or Ctrl+@. Then save and exit vi.

share|improve this answer

One of my students had this problem. There is a bug in grep in Cygwin. If the file has non-Ascii characters, grep and egrep see it as binary.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.