Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
I am using a script to regularly download my gmail messages that compresses the raw .eml into .gz files. The script creates a folder for each day, and then compresses every message into its own file.
I would like a way to search through this archive for a "string."
Grep alone doesn't appear to do it. I also tried SearchMonkey.
If you want to grep recursively in all .eml.gz files in the current directory, you can use:
find . -name \*.eml.gz -print0 | xargs -0 zgrep "STRING"
You have to escape the first * so that the shell does not interpret it. -print0 tells find to print a null character after each file it finds; xargs -0 reads from standard input and runs the command after it for each file; zgrep works like grep, but uncompresses the file first.
zgrep actually seems faster than grep run on uncompressed files. It must be because compressed files can be read off the HD and decompressed faster than reading an uncompressed file from the HD.
xargs uses blanks (whitespace) by default. Sure, files almost never have newlines in them, but spaces are not unheard of (even if most UNIXy types frown on them). That said, you can simplify without worrying about whitespace even more easily: find . -name '*.eml.gz' -exec zgrep "STRING" {} + That gets the same many arguments per-launch of xargs, the safety of -print0/-0, and all without the overhead of an extra process launch and piping, and fairly concisely. -exec with + is POSIX specified, so it should be on most semi-recent UNIX-like systems to my knowledge.
Dec 9, 2016 at 18:38
ABCLog04_18_18_2_21.gz Is there a way to recursively look for files beginning with ABC*. I tried replacing \*.eml.gz in your example above with ABCLog* and get an error about file format.: find: paths must precede expression: ABCLog-2018-03-12-10-16-1.log.gz Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
Apr 18, 2018 at 19:21
There's a lot of confusion here because there isn't just one zgrep. I have two versions on my system, zgrep from gzip and zgrep from zutils. The former is just a wrapper script that calls gzip -cdfq. It doesn't support the -r, --recursive switch.1
The latter is a c++ program and it supports the -r, --recursive option.
Running zgrep --version | head -n 1 will reveal which one (if any) of them is the default:
zgrep (gzip) 1.6
is the wrapper script,
zgrep (zutils) 1.3
is the cpp executable.
If you have the latter you could run:
zgrep 'pattern' -r --format=gz /path/to/dir
Anyway, as suggested, find + zgrep will work equally well with either version of zgrep:
find /path/to/dir -name '*.gz' -exec zgrep -- 'pattern' {} +
If zgrep is missing from your system (highly unlikely) you could try with:
find /path/to/dir -name '*.gz' -exec sh -c 'gzip -cd "$0" | grep -- "pattern"' {} \;
but there's a major downside: you won't know where the matches are as there's no file name prepended to the matching lines .
zgrep from zutils is not available you can install it in Ubuntu with sudo apt-get install zutils.
Jul 27, 2015 at 1:46
grep -n, zgrep -n will print line no.s. It's in the manual...
Nov 9, 2017 at 22:55
ag is a variant of grep, with some nice extra features.
So:
ag -r -z your-pattern-goes-here folder
If not installed,
apt-get install silversearcher-ag (debian and friends)
yum install the_silver_searcher (fedora)
brew install the_silver_searcher (mac)
(edit in Sep 2021 \thanks(x-yuri))
Also consider rg (recursive grep) that has -z option
rg -z your-pattern-goes-here folder
rg has also a large set of useful options. If necessary:
apt install ripgrep
ag: truncated file: Success as a result. Any other flag should I add?
If your system has zgrep, you can simply
zgrep -irs your-pattern-goes-here the-folder-to-search-goes-here/
If your system does not have zgrep, you can use the find command to run zcat and grep against each file like so:
find the-folder-to-search-goes-here/ -name '*.gz' \
-exec sh -c 'echo "Searching {}" ; zcat "{}" | grep your-pattern-goes-here ' \;
Searching ~/gmvault-db/db/2015-02/03/whatever.gz
Mar 2, 2015 at 16:29
zgrep won't take the -r flag for some reason. That's mention in man zgrep (also see my answer).
Recursion alone is easy:
-r, --recursive
Read all files under each directory, recursively, following
symbolic links only if they are on the command line. This is
equivalent to the -d recurse option.
-R, --dereference-recursive
Read all files under each directory, recursively. Follow all
symbolic links, unlike -r.
However, for compressed files you need something like:
shopt globstar
for file in /path/to/directory/**/*gz; do zcat ""$file" | grep pattern; done
path/to/directory should be the parent directory that contains the subdirectories for each day.
zgrep is the obvious answer but, unfortunately, it does not support the -r flag. From man zgrep:
These grep options will cause zgrep to terminate with an error code: (-[drRzZ]|--di*|--exc*|--inc*|--rec*|--nu*).
xzgrep -l "string" ./*/*.eml.gz
xzgrep is a derivative of the zgrep utils (less /bin/xzgrep)
From the Man page:
xzgrep invokes grep(1) on files which may be either uncompressed or compressed with xz(1), lzma(1), gzip(1), bzip2(1), or lzop(1). All options specified are passed directly to grep(1).
-l print the matching file name
-R for recursion will not work as it's specifically prohibited in the script, however simple shell globbing should get us there
./*/*.eml.gz
from a relative path where ./today/sample.eml.gz, match on all instances of that are one level below our relative position in the shell, that ends with ".eml.gz"
