1

I have a 9.8GB gzip file A.gz and other file i have is 79MB B.txt which has some text in each line. I want to grep B's text in A.gz and write to a new file.

Initially, I used this command

zgrep -f B.txt A.gz > C.xml

But this command was hanged and it created an empty C.xml for very long time.

Then after googling I came to know that because B.txt is huge it hangs when it keeps the text in buffer.

So I splitted the text file in 20000 text each

split -l 20000 -a 4 B.txt B

I created files like Baaaa, Baaab....

and then iterated over each file

cd B
for f in B*; do
  zgrep -f "$f" ../A.gz >> C.xml
done

It is very slow and still running.

Any better approach for this?

Will gunzipping the gz file will improve performance?

Update

I tried with -F

zgrep -F -f "$f" ../A.gz >> C.xml

this is little faster but still want other option

I have xml like

<root>
   <source>source1</source>
   <Id>123</Id>
   <category>ABC</category>
</root>
<root>
    <source>source2</source>
    <Id>123</Id>
    <category>XYZ</category>
</root>

Here id is same 123 but category is different ABC and XYZ

(Input is limited set of categories e.g ABC,DEF,GHI,JKLM,NOP) Initially I have category as ABC and So based on the category ABC I find out its id i.e. 123 like this i keep on writing all id's that belong to these input categories to a new file i.e. B.txt(list of id's) like below

zgrep -E 'ABC|DEF|GHI|JKLM|NOP' A.gz | sed -n 's:.*<Id>\(.*\)</Id>.*:\1:p' | uniq > B.txt

later on I iterate over this id's and fetch all xmls, this way I get both xml tags with category ABC and XYZ which belongs to id 123

2
  • @Anon I have updated the question with sample xml
    – happy
    Jun 3 '20 at 7:31
  • Assuming that your file structure really is that regular could measure how big the output of a processed version is via zgrep '\<\(category\|Id\)>' A.gz | wc -c?
    – Anon
    Jun 3 '20 at 8:14
2
+50

A 79MByte grep "string" is going to painful to work with. Are the lines of B.txt really regular expressions or are they fixed identical strings? If they are fixed strings do they appear identically in A.gz as a whole line? How many lines in the uncompressed A.gz are expected to be matched by lines in B.txt?

Pattern matching suggestion

If lines in B.txt really are regular expressions or substrings of lines in A.gz you may be forced to use something like HyperScan which is designed to deal with huge regexes. If you have the disk space you could decompress A.gz and just let HyperScan get to work on that (you may even be able to let the shell decompress on the fly while HyperScan searches through it). Another alternative to try is ripgrep.

Full line matching suggestion

If you're dealing with fixed full line strings in B.txt and the uncompressed A.gz contains a comparatively small (let's say on the order of 100MB) of matching lines you may be better served writing a program to pre-process A.gz:

  • You could hash each line of B.txt and remember the hashes
  • You then check if any line in the uncompressed A.gz hashes to the same thing as any of your previous hashes. If so you print out the line (e.g. into C.txt) ready for further processing
  • You now do final pass where you more rigorously check each line whether each line in B.txt is in C.txt (or vice versa - depending on which file was smaller)

Some code for do the initial approximate filtering could be this:

# Do a quick APPROXIMATE filter of lines in FILENEEDLES that are also in
# FILEHAYSTACK
import sys

def main():
    if len(sys.argv) < 2:
        print("usage: %s FILENEEDLES FILEHAYSTACK" % sys.argv[0])
        exit(1)

    first_filename = sys.argv[1]
    second_filename = sys.argv[2]

    line_hashes = set()

    with open(first_filename, "r") as f:
        for line in f:
            line_hashes.add(hash(line))

    with open(second_filename, "r") as f:
        for line in f:
            if hash(line) in line_hashes:
                sys.stdout.write(line)

if __name__ == "__main__":
    main()

For example:

$ echo -e '1\n2\n3' > B.txt
$ echo -e '2\n3\n4\5' | gzip > A.gz
$ ./approxfilter.py B.txt <(gzip -dc A.gz) > candidates.txt
$ cat candidates.txt
2
3

You would now need to check candidates.txt to see if lines output EXACTLY matched those of B.txt (but this is hopefully a smaller and far easier problem and you could even modify the program above to do it all if the number of candidate lines is "small" and well within what can be held in memory). (The questioner later clarified in comments that they are not working with full line length strings so this approach won't work)

3
  • B.txt has fixed identical strings, that is why later on i added -F in the grep command. And I am expecting 2086446 lines to be matched lines
    – happy
    Jun 3 '20 at 4:43
  • Sorry, my mistake...B.txt has 1\n2\n3 content but A.gz has xml tags, each nested xml which has <Id>2</Id> tag. And I am grepping this <Id> tag and fetching the xml
    – happy
    Jun 3 '20 at 5:43
  • @happy Argh. I think your next best bet is to try and think of ways to cut down A.gz before you have to search it. For example, do all the lines you're interested in within A.gz all contain some tag not seen in other lines? If so how much smaller does that make what you have to search through?
    – Anon
    Jun 3 '20 at 6:29
2

Your second attempt would very likely be improved by unzipping, otherwise each iteration of the loop will have the full unzipping overhead — unzipping beforehand will mean you only have that overhead one time.

If that is still not fast enough, you could try multi threading it as well (assumes unzipped A).

find B -type f -name 'B*' -print0 \
  | xargs -0 -t -n1 -P8 \
  grep -f {} A >> C.xml

This example should run 8 processes simultaneously, you might need to tune this value depending on how many processors/cores you have.

I’m not sure what sort of results you’re expecting speed wise; frankly, it sounds like you are giving it a lot of work to do, and lots of work takes time.

2
  • it goes out of memory when i try to unzip it as it contains 400-600gb data..What if I split this gzip file into smaller ones?
    – happy
    Jun 2 '20 at 7:31
  • Splitting the gz as it is will not yield positive results due to the headers it will contain. You would have to decompress the file and split the raw text, and recompress. But I’m not sure if this would improve your lot.
    – bxm
    Jun 2 '20 at 20:57
1

zgrep is just a shell script wrapper for grep. It simply runs whatever grep you have installed on your system and uses gzip to decompress the input file.

If you were using GNU grep version 3.5 or 3.6, there is bug recently uncovered which seems to slow down processing of the pattern file as you described with extreme performance degradation.

The bug example pattern file in the release notes of grep 3.7 is about 48 Mb worth of patterns, so size should not be a problem as far as i can see.

https://www.theregister.com/2021/08/16/gnu_grep_37/

2
  • OK @terdon, thank you for trying to improve my terse answer. Zgrep is an interesting shell script, that is why adding .sh to the name still refers to the same application. It may come from gzip package or FreeBSD grep source itself. However the problem could be grep itself. Grep should not take more than a dozen seconds to complete compiling his pattern file and the described bug affects that precisely. He tries to fix slow processing of his pattern file by splitting it and then running a grep for loop, which is but a desperate approach.What is his grep version? IDK
    – user487306
    Aug 21 at 19:22
  • Gotcha. I thought you were referring to some other tool called zrep.sh. I knew zgrep was a wrapper but had never seen it with an sh extension so that threw me. I edited to clarify. Thanks!
    – terdon
    Aug 23 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.