22

Consider a text file with the following entries:

aaa
bbb
ccc
ddd
eee
fff
ggg
hhh
iii

Given a pattern (e.g. fff), I would like to grep the file above to get in the output:

all_lines except (pattern_matching_lines  U (B lines_before) U (A lines_after))

For example, if B = 2 and A = 1, the output with pattern = fff should be:

aaa
bbb
ccc
hhh
iii

How can I do this with grep or other command line tools?


Note, when I try:

grep -v 'fff'  -A1 -B2 file.txt

I don't get what I want. I instead get:

aaa
bbb
ccc
ddd
eee
fff
--
--
fff
ggg
hhh
iii
9

don's might be better in most cases, but just in case the file is really big, and you can't get sed to handle a script file that large (which can happen at around 5000+ lines of script), here it is with plain sed:

sed -ne:t -e"/\n.*$match/D" \
    -e'$!N;//D;/'"$match/{" \
            -e"s/\n/&/$A;t" \
            -e'$q;bt' -e\}  \
    -e's/\n/&/'"$B;tP"      \
    -e'$!bt' -e:P  -e'P;D'

This is an example of what is called a sliding window on input. It works by building a look-ahead buffer of $B-count lines before ever attempting to print anything.

And actually, probably I should clarify my previous point: the primary performance limiter for both this solution and don's will be directly related to interval. This solution will slow with larger interval sizes, whereas don's will slow with larger interval frequencies. In other words, even if the input file is very large, if the actual interval occurrence is still very infrequent then his solution is probably the way to go. However, if the interval size is relatively manageable, and is likely to occur often, then this is the solution you should choose.

So here's the workflow:

  • If $match is found in pattern space preceded by a \newline, sed will recursively Delete every \newline that precedes it.
    • I was clearing $match's pattern space out completely before - but to easily handle overlap, leaving a landmark seems to work far better.
    • I also tried s/.*\n.*\($match\)/\1/ to try to get it in one go and dodge the loop, but when $A/$B are large, the Delete loop proves considerably faster.
  • Then we pull in the Next line of input preceded by a \newline delimiter and try once again to Delete a /\n.*$match/ once again by referring to our most recently used regular expression w/ //.
  • If pattern space matches $match then it can only do so with $match at the head of the line - all $Before lines have been cleared.
    • So we start looping over $After.
    • Each run of this loop we'll attempt to s///ubstitute for &itself the $Ath \newline character in pattern space, and, if successful, test will branch us - and our whole $After buffer - out of the script entirely to start the script over from the top with the next input line if any.
    • If the test is not successful we'll branch back to the :top label and recurse for another line of input - possibly starting the loop over if $match occurs while gathering $After.
  • If we get past a $match function loop, then we'll try to print the $last line if this is it, and if !not try to s///ubstitute for &itself the $Bth \newline character in pattern space.
    • We'll test this, too, and if it is successful we'll branch to the :Print label.
    • If not we'll branch back to :top and get another input line appended to the buffer.
  • If we make it to :Print we'll Print then Delete up to the first \newline in pattern space and rerun the script from the top with what remains.

And so this time, if we were doing A=2 B=2 match=5; seq 5 | sed...

The pattern space for the first iteration at :Print would look like:

^1\n2\n3$

And that's how sed gathers its $Before buffer. And so sed prints to output $B-count lines behind the input it has gathered. This means that, given our previous example, sed would Print 1 to output, and then Delete that and send back to the top of the script a pattern space which looks like:

^2\n3$

...and at the top of the script the Next input line is retrieved and so the next iteration looks like:

^2\n3\n4$

And so when we find the first occurrence of 5 in input, the pattern space actually looks like:

^3\n4\n5$

Then the Delete loop kicks in and when it's through it looks like:

^5$

And when the Next input line is pulled sed hits EOF and quits. By that time it has only ever Printed lines 1 and 2.

Here's an example run:

A=8 B=7 match='[24689]0'
seq 100 |
sed -ne:t -e"/\n.*$match/D" \
    -e'$!N;//D;/'"$match/{" \
            -e"s/\n/&/$A;t" \
            -e'$q;bt' -e\}  \
    -e's/\n/&/'"$B;tP"      \
    -e'$!bt' -e:P  -e'P;D'

That prints:

1
2
3
4
5
6
7
8
9
10
11
12
29
30
31
32
49
50
51
52
69
70
71
72
99
100
  • I'm actually working with huge files, and don's answer was noticeably slower than this solution. I was initially hesitant to change my accepted answer, but the speed difference is quite visible. – Amelio Vazquez-Reina Jul 1 '15 at 23:20
  • 4
    @Amelio - this will work with a stream of any size, and it need not read the file through to work. The biggest performance factor is the size of $A and/or $B. The larger you make those numbers, the slower it will get - but you can make them reasonably large. – mikeserv Jul 1 '15 at 23:25
  • 1
    @AmelioVazquez-Reina - if you're using the older one, this is better, I think. – mikeserv Jul 2 '15 at 8:17
11

You could use gnu grep with -A and -B to print exactly the parts of the file you want to exclude but add the -n switch to also print the line numbers and then format the output and pass it as a command script to sed to delete those lines:

grep -n -A1 -B2 PATTERN infile | \
sed -n 's/^\([0-9]\{1,\}\).*/\1d/p' | \
sed -f - infile

This should also work with files of patterns passed to grep via -f e.g.:

grep -n -A1 -B2 -f patterns infile | \
sed -n 's/^\([0-9]\{1,\}\).*/\1d/p' | \
sed -f - infile

I think this could be slightly optimized if it collapsed any three or more consecutive line numbers into ranges so as to have e.g. 2,6d instead of 2d;3d;4d;5d;6d... though if the input has only a few matches it's not worth doing it.


Other ways that don't preserve line order and are most likely slower:
with comm:

comm -13 <(grep PATTERN -A1 -B2 <(nl -ba -nrz -s: infile) | sort) \
<(nl -ba -nrz -s: infile | sort) | cut -d: -f2-

comm requires sorted input which means the line order would not be preserved in the final output (unless your file is already sorted) so nl is used to number the lines before sorting, comm -13 prints only lines unique to 2nd FILE and then cut removes the part that was added by nl (that is, the first field and the delimiter :)
with join:

join -t: -j1 -v1 <(nl -ba -nrz -s:  infile | sort) \
<(grep PATTERN -A1 -B2 <(nl -ba -nrz -s:  infile) | sort) | cut -d: -f2-
  • Thanks Don! Quick question, would you expect the solution with comm to be faster than the original one with sed and grep ? – Amelio Vazquez-Reina Jul 1 '15 at 23:22
  • 1
    @AmelioVazquez-Reina - I don't think so as it still reads the input file twice (plus it does some sorting) as opposed to Mike's solution which only processes the file once. – don_crissti Jul 1 '15 at 23:27
7

If you don't mind using vim:

$ export PAT=fff A=1 B=2
$ vim -Nes "+g/${PAT}/.-${B},.+${A}d" '+w !tee' '+q!' foo
aaa
bbb
ccc
hhh
iii
  • -Nes turns on non-compatible, silent ex mode. Useful for scripting.
  • +{command} tell vim to run {command} on the file.
  • g/${PAT}/ - on all lines matching /fff/. This gets tricky if the pattern contains regular expression special characters that you didn't intend to treat that way.
  • .-${B} - from 1 line above this one
  • .+${A} - to 2 lines below this one (see :he cmdline-ranges for these two)
  • d - delete the lines.
  • +w !tee then writes to standard output.
  • +q! quits without saving changes.

You can skip the variables and use the pattern and numbers directly. I used them just for clarity of purpose.

2

How about (using GNU grep and bash):

$ grep -vFf - file.txt < <(grep -B2 -A1 'fff' file.txt)
aaa
bbb
ccc
hhh
iii

Here we are finding the lines to be discarded by grep -B2 -A1 'fff' file.txt, then using this as an input file to find the desired lines discarding these.

  • Hmm, this does not output anything on my machine (OS X) – Amelio Vazquez-Reina Jul 1 '15 at 20:10
  • @AmelioVazquez-Reina sorry about that..i did not know your OS before..anyway i have tested this on Ubuntu.. – heemayl Jul 1 '15 at 20:12
  • 2
    This would have the same problem as kos's (now deleted) solution as if there are duplicate lines in the input file and some of them fall outside the range and others are inside that range this will delete them all. Also, with multiple occurrences of pattern, if there are lines like -- in the input file (outside the ranges) this will delete them because the delimiter -- appears in grep's output when more then one line is matching pattern (the latter is highly unlikely but worth mentioning I guess). – don_crissti Jul 1 '15 at 21:41
  • @don_crissti Thanks..you are right..although i was taking OP's example literally..i am gonna leave it in case someone find it helpful later.. – heemayl Jul 2 '15 at 14:14
1

You can reach a good-enough result by using temporary files:

my_file=file.txt #or =$1 if in a script

#create a file with all the lines to discard, numbered
grep -n -B1 -A5 TBD "$my_file" |cut -d\  -f1|tr -d ':-'|sort > /tmp/___"$my_file"_unpair

#number all the lines
nl -nln "$my_file"|cut -d\  -f1|tr -d ':-'|sort >  /tmp/___"$my_file"_all

#join the two, creating a file with the numbers of all the lines to keep
#i.e. of those _not_ found in the "unpair" file
join -v2  /tmp/___"$my_file"_unpair /tmp/___"$my_file"_all|sort -n > /tmp/___"$my_file"_lines_to_keep

#eventually use these line numbers to extract lines from the original file
nl -nln $my_file|join - /tmp/___"$my_file"_lines_to_keep |cut -d\  -f2- > "$my_file"_clean

The result is good-enough because you can loose some indentation in the process, but if it is an xml or indentation insensitive file it should not be a problem. Since this script uses a ram drive, writing and reading those temp files is as fast as working in memory.

0

Also, if you just want to exclude some lines ahead of a given marker, you could use:

awk -v nlines=2 '/Exception/ {for (i=0; i<nlines; i++) {getline}; next} 1'

(glenn jackman at https://stackoverflow.com/a/1492538 )

By piping some commands you can get the before/after behaivour:

awk -v nlines_after=5 '/EXCEPTION/ {for (i=0; i<nlines_after; i++) {getline};print "EXCEPTION" ;next} 1' filename.txt|\
tac|\
awk -v nlines_before=1 '/EXCEPTION/ {for (i=0; i<nlines_before; i++) {getline}; next} 1'|\
tac
0

One way to accomplish this, maybe the easiest way would be to create a variable and do the following:

grep -v "$(grep "fff" -A1 -B2 file.txt)" file.txt

This way you still have your structure. And you can easily see from the one liner what are you trying to remove.

$ grep -v "$(grep "fff" -A1 -B2 file.txt)" file.txt
aaa
bbb
ccc
hhh
iii

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.