I want to look for the presence of a string on all gziped files on server.
I am afraid if I run something like gunzip -c *.gz | grep "string to be searched" server get overloaded with cache for ungziped files.
I didn't run the script already, because it's a one shot request that I don't have space to experiment. Is the script sure ? like it does sequentially look-ups and saves no cache in between?
Because of the way pipes work, gunzip -c *.gz won’t be able to feed (much) more data into the pipe than grep can process, so the pipe itself won’t cause memory issues. gunzip streams data as it uncompresses it, so likewise it won’t decompress more data than it can output, and won’t use much memory.
Reading all that data from disk will cause data to be cached, but the kernel manages that quite well. There’s nothing much to worry about there.
On my system,
/bin/time sh -c "gunzip -c *.gz | grep test"
on 3.1GiB of compressed files (6.1GiB uncompressed) shows that the whole pipe consumes at most 3MiB of RAM; cache usage increased by 1.3GiB.
You can analyse file by file, and monitor system load in between.
You can also use zgrep which is more immediate to type:
zgrep "string" file.gz
There is nothing "sure" in the world, but unless you have a colossal zipfile or a very low amount of RAM in the machine, you should be fine. Keep an eye on RAM usage while running this.
gunzip -c *.gz | grep "string to be searched" cause more stress on the system than zgrep?
Oct 9, 2018 at 9:20
gunzipuncopresses a file in a streaming fashion, it doesn't load it whole in the memory. It won't put any more stress on the cache than a simplecat.