I have a 9.8GB gzip file A.gz and other file i have is 79MB B.txt which has some text in each line. I want to grep B's text in A.gz and write to a new file.
Initially, I used this command
zgrep -f B.txt A.gz > C.xml
But this command was hanged and it created an empty C.xml for very long time.
Then after googling I came to know that because B.txt is huge it hangs when it keeps the text in buffer.
So I splitted the text file in 20000 text each
split -l 20000 -a 4 B.txt B
I created files like Baaaa, Baaab....
and then iterated over each file
cd B for f in B*; do zgrep -f "$f" ../A.gz >> C.xml done
It is very slow and still running.
Any better approach for this?
Will gunzipping the gz file will improve performance?
I tried with -F
zgrep -F -f "$f" ../A.gz >> C.xml
this is little faster but still want other option
I have xml like
<root> <source>source1</source> <Id>123</Id> <category>ABC</category> </root> <root> <source>source2</source> <Id>123</Id> <category>XYZ</category> </root>
Here id is same 123 but category is different ABC and XYZ
(Input is limited set of categories e.g ABC,DEF,GHI,JKLM,NOP) Initially I have category as ABC and So based on the category ABC I find out its id i.e. 123 like this i keep on writing all id's that belong to these input categories to a new file i.e. B.txt(list of id's) like below
zgrep -E 'ABC|DEF|GHI|JKLM|NOP' A.gz | sed -n 's:.*<Id>\(.*\)</Id>.*:\1:p' | uniq > B.txt
later on I iterate over this id's and fetch all xmls, this way I get both xml tags with category ABC and XYZ which belongs to id 123