Currently I have netcat piping output to tee which is writing to output.txt with

nc -l -k -p 9100 | tee output.txt

I want to monitor this output, so I'm watching it with tail -f | egrep -i 'regex' via PuTTY so that I only see relevant bits.

Every now and then I want to clear the output file. The issue arises that if I do > output.txt and then again try to tail -f | egrep ... I get no output. If I grep through the file, I get no matches, despite knowing that there should be matches (as cat output.txt spits out the file properly)

mitch@quartz:~$ grep output.txt -e 'regex'
Binary file output.txt matches

While the same command on output.txt before emptying it works fine.

Basically: > makes grep think my file is a binary file and it won't properly search. Is there a better way to clear the file?

  • Instead of > output.txt why not try something along the lines of echo Start > output.txt. Hopefully, grep won't think it's a binary file. – garethTheRed Oct 15 '14 at 13:46
  • 2
    Best would be to use tee -a. See there for details (tee -a is the equivalent of shells using >>). – Stéphane Chazelas Oct 15 '14 at 15:08
  • 1
    @StéphaneChazelas, I think that's the best answer in this thread – Mitch Oct 15 '14 at 16:56
  • If you truncate the file while open, the next write (by | tee) will refill it with zeros and the write will occur at the previous offset. grep treats an occurrence of 0 within the first buffer read (32kiB) as a binary file by default. – mr.spuratic Oct 15 '14 at 19:48

If the only problem is that grep treats it as binary, tell grep to search it regardless:

$ head /bin/bash > out
$ echo "test" >> out 
$ grep test out 
Binary file out matches
$ grep -a test out 

From man grep:

   -a, --text
          Process  a binary file as if it were text; this is equivalent to
          the --binary-files=text option.

It might answer your question, so here are the results of a few tests I just ran:

$ > output.txt
$ file output.txt
output.txt: empty

$ echo "" > output.txt
$ file output.txt
output.txt: very short file (no magic)

$ echo " " > output.txt
$ file output.txt
output.txt : ASCII text

As you can see, the file isn't categorised the same way according to what you actually "put" in it when you try to clear it. Therefore, you might want to use an empty string instead of just nothing.

  • I'll give it a shot in a minute and report back – Mitch Oct 15 '14 at 13:50
  • It works, but only if it's ASCII beforehand. That is, if I had previously done > output and then echo " " > output it still gets seen as a binary file – Mitch Oct 15 '14 at 14:24
  • The difference between the three is that the first is empty, the second contains a newline (not really an empty string) and the third contains a space and a newline. – Dennis Williamson Oct 15 '14 at 21:27

> makes grep think the file is binary because it is binary. The thing is, you emptied the file, but didn't stop the program that was filling it.

>output.txt creates output.txt if it doesn't exist, and truncates it to zero length if it does.

At the point you run >output.txt, there is a tee process which has the file open. Truncating the file doesn't affect the position at which tee is writing. Let's say it had written N bytes before the truncation. The next time tee writes after the truncation, it will start writing at position N. Writing at a position beyond the current end of a file is allowed and fills the beginning of the file with null bytes.¹ That's what happened here.

Grep sees a file that begins with some null bytes. It correctly reports the file as binary.

You can tell GNU grep to treat the file as text by calling grep -a. It will search the whole file, including the null bytes (which don't match, so they don't affect the result unless there's a match on the first line, but they may cause a slowdown if there are a lot of them).

A better solution is to tell tee to always write at the current end of the file. Fortunately (as Stephane Chazelas remarked), there's an option for that: tee -a (present on all POSIX-compliant systems). You'll need to truncate the file first.

nc -l -k -p 9100 | tee -a output.txt

¹ Most filesystems allow blocks that would entirely consist of null bytes to remain unallocated. This specialized method of compression is called making a sparse file.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.