1

Task:

I'm using grep to search some text files, piping the results from one grep (excluding some lines) to another (matching some lines) + displaying some context using the -C parameter as shown below:

grep -v "Chapter" *.txt | grep -nE  -C1 " leaves? " 

Problem:

This works very well when printing the results, but produces very large files (~ several GB) and takes forever when I write it to file like so:

grep -v "Chapter" *.txt | grep -nE  -C1 " leaves? " > out.txt

Troubleshoot:

  1. The grep returns only 1345 lines (according to wc), printout takes a few seconds

  2. The output in the large output files looks legit, aka actual results from input files.

  3. Replacing the -C operator with -A or -B produces good output files in the KB size.

Questions:

  • Why is this happening?
  • Is there something about the -C that breaks things this way?
  • Or is there a different issue I am overlooking?

Any hints appreciated! Running this in the MacOS terminal. I was following this man.

  • 2
    "Or is there a different issue I am overlooking?" well, you are writing to file out.txt while also reading from any files matching *.txt, maybe that results in a kind of race condition? – steeldriver Aug 11 '18 at 23:48
  • @steeldriver that is a very good point that I have not considered. I don't know why -A and -B would work under these circumstances then but I'll give it a shot as soon as I am back on that machine – patrick Aug 12 '18 at 0:50
3

Try changing the directory where you're writing out.txt. For example change this command to this:

$ grep -v "Chapter" *.txt | grep -nE  -C1 " leaves? " > /tmp/out.txt

Example

Here you can see what's happening when you enable verbose output in your Bash shell.

$ set -x
$ grep -v "Chapter" *.txt | grep -nE  -C1 " leaves? " > out.txt
+ grep --color=auto -nE -C1 ' leaves? '
+ grep --color=auto -v Chapter file01.txt file02.txt file03.txt file04.txt file05.txt file06.txt file07.txt file08.txt file09.txt file10.txt out.txt

Notice that it's taking the argument *.txt and expanding it, and it includes the the file out.txt. So you're literally parsing this file as you write out to it.

Why?

If you think about what a shell does when output from 1 command is piped into the next it makes sense. The shell parses the commands you just gave it, looking for pipes (|). When it encounters them it has to run the ones from the right in order to set up the redirection of STDIN/STDOUT between the commands occuring within the pipes.

You can use the sleep command to see how the shell parses things as more pipes are added:

$ sleep 0.1 | sleep 0.2 | sleep 0.3 | sleep 0.4
+ sleep 0.2
+ sleep 0.3
+ sleep 0.4
+ sleep 0.1

$ sleep 0.1 | sleep 0.2 | sleep 0.3 | sleep 0.4 | sleep 0.5
+ sleep 0.2
+ sleep 0.3
+ sleep 0.4
+ sleep 0.5
+ sleep 0.1

Doing this with echo + writing to a file also shows the order via the file accesses & the stat command:

$ echo "1" > file1 | echo "2" > file2 | echo "3" > file3 | echo "4" > file4
+ echo 2
+ echo 3
+ echo 4
+ echo 1

$ stat file* | grep -E "File|Access: [[:digit:]]+"
+ grep --color=auto -E 'File|Access: [[:digit:]]+'
+ stat file1 file2 file3 file4
  File: ‘file1’
Access: 2018-08-11 23:55:20.868220474 -0400
  File: ‘file2’
Access: 2018-08-11 23:55:20.865220576 -0400
  File: ‘file3’
Access: 2018-08-11 23:55:20.866220542 -0400
  File: ‘file4’
Access: 2018-08-11 23:55:20.867220508 -0400
  • thank you so much for the detailed answer. I changed the file location / name and it works no prob. I still don't understand why I would not run into this problem when doing -A or -B but I see the error of my thinking: since the file output is the right-most command, in my mind this happened "after everything else"... – patrick Aug 12 '18 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.