5

I have a bunch of gz files and unzipped version of them contains the patterns A and B=1 (these are certainly on different lines where A appears first).

I want to write a command that gives me the content of lines where A is present and where B=1 is present. Or at least the content between A and B=1 inclusively.

Input file1 :

..A ...
...
...B=0..
...

Input file2 :

..A ...
...
...B=1..
...

My command must output A ....B=1 for file2 and nothing for file1.

I did something like this, but is not working as expected:

find . -name \*.gz -print0 | xargs -0 zcat | sed -n -e '/A/,/B=1/p'

What is the problem here?

13
  • Why not simply zgrep?
    – simlev
    Dec 13, 2017 at 13:35
  • Because zgrep does not work on multiple lines.
    – cehptr
    Dec 13, 2017 at 13:35
  • 1
    So some lines contain A (you want those) and some lines contain B=1 (you want those), some lines contain neither (you don't want those) and some lines contain both (you want those)? The pattern /A/,/B=1/ says "find the first line that contains A and the first line after it that contains B=1 and all the lines in between - then start looking for A again".
    – NickD
    Dec 13, 2017 at 13:36
  • All files will contain both A and B=1 or both A and B=0 but A will be placed on some before line of B=. I want to print exactly these lines with A and B=1
    – cehptr
    Dec 13, 2017 at 13:37
  • zgrep 'A\|B=1' * shows "the content of lines where A is present and where B=1 is present".
    – simlev
    Dec 13, 2017 at 13:39

4 Answers 4

4

Let's ignore the compression for now. You want to output the lines between A and B=1, but only if both appear. The sed you used will not do that, since it starts outputting as soon as A is seen, and doesn't check for B=1. We could use the hold buffer in sed to keep everything until B=1 is found, but I'm more comfortable with awk, so here:

$ echo -en 'not this\nA\nthis\nB=1\nnot this\n' | 
  awk '/A/ {save=1} save {data = data $0 ORS} /B=0/ {save=0; data=""}  /B=1/ {save=0; printf "%s", data; data=""} '
A
this
B=1

The B=0 rule handles blocks that should not be printed.

Then, handling the compression and multiple files. The find+xargs you did works, though if some files can have partial blocks (A without B), concatenating the files together will lead to problems. Assuming that's not the case, we can just stick the awk to the end:

$ find . -name foo\*.gz -print0 | xargs -0 zcat | \
  awk '/A/ {s=1} s {d = d $0 ORS} /B=0/ {s=0; d=""} 
  /B=1/ {s=0; printf "%s", d; d=""} '      

If we do need to deal with partial blocks, we'll have to handle each file separately:

$ find . -name foo\*.gz -print0 | xargs -0 sh -c '
  for f; do zcat "$f" |  awk '\''/A/ {s=1} s {d = d $0 ORS} 
    /B=0/ {s=0; d=""} /B=1/ {s=0; printf "%s", d; d=""} '\''; done' sh

The quoting is horrible, so the awk script should probably to a file of its own.

Or just do it in the shell (Bash/ksh/zsh):

$ shopt -s globstar    # set -o globstar in ksh
$ for f in **/*.gz ; do zcat "$f" |
  awk '/A/ {s=1} s {d = d $0 ORS} /B=0/ {s=0; d=""} 
  /B=1/ {s=0; printf "%s", d; d=""} ' ; done

If you want to print only the intervening lines (not the A and B=1 lines), exchange the positions of the /A/ {...} and /B=.../ {...} blocks.

3
  • That`s what I want. Thanks and well done.
    – cehptr
    Dec 13, 2017 at 14:28
  • @cehptr Of course, this would not work as intended e.g. if a file had A then B=0 and the subsequent one had A and B=1 (it'd print from the first A to B=1. I'd filter out files first to avoid this.
    – simlev
    Dec 13, 2017 at 14:57
  • @simlev, oh dear, you're right. It doesn't work too well with the files concatenated.
    – ilkkachu
    Dec 13, 2017 at 15:49
2

Certainly NOT the best way, but it works for me:

find -name "*.gz" | xargs zgrep -l A | xargs zgrep -l "B=1" | xargs zcat | sed -n '/A/,/B=1/p

First, we get a list of files, then we filter files containing A, then we filter files containing B=1, the resulting files we zcat to sed.

Danger: if one file contains both B=1 and A, in that order, the contents of that file is written to the end.

Example:

$ ls /tmp/file*gz
/tmp/filea.gz  /tmp/fileb.gz
$ zcat /tmp/filea.gz
one
two
three
A
four five
six
B=1
seven
eight
nine
$ zcat /tmp/fileb.gz
one
two
three
A
four five
six
B=0
seven
eight
nine
$ find /tmp -type f -name "file*.gz" | xargs zgrep -l A | xargs zgrep -l "B=1" | xargs zcat | sed -n '/A/,/B=1/p'
A
four five
six
B=1
1

If you have pcregrep and it has been built with libz support, you can do:

pcregrep --include='\.gz$' -rM '(?s)A.*?B=1' .

Example:

$ pcregrep --help | grep zlib
Files whose names end in .gz are read using zlib.
Files whose names end in .bz2 are read using bzlib2.
$ pcregrep --include='\.gz$' -rM '(?s)A.*?B=1' .
./1/2/3/x.gz:AAA
blih
BOB=123
./b.gz:A
blah
B=1
1
  • 1
    Cool! (+1) libz support is relevant! But: in my debian distribution the pcregrep has no support for gz/bz2 and recommends zpcregrep, which works fine zpcregrep -M '(?s)A.*?B=1' *.gz but has no option --include nor -r; in my fedora (pcre-tools with pcregrep 8.41 2017-07-05) -- no support for zlibs . Are compiling pcregrep locally?
    – JJoao
    Dec 15, 2017 at 7:02
0
zcat *.gz | \
sed  's/B=[0-9].*/&\x00/'  | \
grep  -zo 'A.*B=1' | \
sed 's/\x00/\n=====\n/'
  • line 1 (may be replaced by the find-command)
  • line 2 adds a null after "B=..." lines to explicitly separate registers
  • line 3 greps in a null-separated sequence of registers, the A...B=1 patterns
  • line 4 (if useful) translate null to a more visible separator

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .