I want to look for the presence of a string on all gziped files on server. I am afraid if I run something like gunzip -c *.gz | grep "string to be searched" server get overloaded with cache for ungziped files.

I didn't run the script already, because it's a one shot request that I don't have space to experiment. Is the script sure ? like it does sequentially look-ups and saves no cache in between?

  • 1
    gunzip uncopresses a file in a streaming fashion, it doesn't load it whole in the memory. It won't put any more stress on the cache than a simple cat. – mosvy Oct 9 '18 at 10:23

Because of the way pipes work, gunzip -c *.gz won’t be able to feed (much) more data into the pipe than grep can process, so the pipe itself won’t cause memory issues. gunzip streams data as it uncompresses it, so likewise it won’t decompress more data than it can output, and won’t use much memory.

Reading all that data from disk will cause data to be cached, but the kernel manages that quite well. There’s nothing much to worry about there.

On my system,

/bin/time sh -c "gunzip -c *.gz | grep test"

on 3.1GiB of compressed files (6.1GiB uncompressed) shows that the whole pipe consumes at most 3MiB of RAM; cache usage increased by 1.3GiB.

  • very concise and clear to me. I had no need anymore in my use case functionally, but I learned something. – Curcuma_ Oct 10 '18 at 6:47

You can analyse file by file, and monitor system load in between. You can also use zgrep which is more immediate to type:

zgrep "string" file.gz

There is nothing "sure" in the world, but unless you have a colossal zipfile or a very low amount of RAM in the machine, you should be fine. Keep an eye on RAM usage while running this.

  • 1
    Why would gunzip -c *.gz | grep "string to be searched" cause more stress on the system than zgrep? – Stephen Kitt Oct 9 '18 at 9:20
  • It does not in fact, my suggestion was to analyse file by file to check system load in between. I clarified my answer. – dr01 Oct 9 '18 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.