0

I have a list of folders in each folder I have around 600 .gz files which are automatically generated by the system every 15 min. I need to display the particular string which is repeated in that .gz files. And find the .gz file which the string is located?

  • 3
    What do you mean exactly by "the particular string which is repeated"? In general, you can use zgrep to search within GZIPed data. Can you give a more complete description of what you are trying to do? – steeldriver Apr 26 at 0:57
  • a compressed string of bytes, or a string of bytes that has now been compressed to something else? Also, what does "repeated" mean? is 2 instances sufficient, or what is? – Jeff Schaller Apr 26 at 1:04
1

Given some top-level directory, topdir, under which all your compressed files are located, and assuming that you are using a GNU implementation of zgrep:

zgrep -R -l -F --include='*.gz' -e 'string' topdir

This would search for string as a substring in any of the gzip-compressed files under topdir and would return the pathnames of all matching files.

Using zgrep on a system without GNU zgrep, you likely can't use --include. This may not be an issue if your topdir only contains compressed files, but if you want to only search the file with .gz filename suffix, you may do

find topdir -type f -name '*.gz' -exec zgrep -l -F -e 'string' {} +

This would find all regular files with names ending in .gz under topdir and will run the given zgrep command on as large batches of these as possible.

If you don't even have zgrep, you could do

find topdir -type f -name '*.gz' -exec sh -c '
    for pathname do
        if gzip -dc "$pathname" | grep -q -F -e 'string'; then
            printf "%s\n" "$pathname"
        fi
    done' sh {} +

This extracts each file and passes it through ordinary grep. If a line is found that contains the given string, the pathname of the file is printed.

Notice how the "internal" shell script is a loop that could easily be modified to work on all *.gz-files in a single directory:

for pathname in ./*.gz; do
    if gzip -dc "$pathname" | grep -q -F -e 'string'; then
        printf "%s\n" "$pathname"
    fi
done

If you want to match string as a complete word and not as a substring, then add -w to the invocations of grep above. If string is something that you'd like to be interpreted as a regular expression, remove -F.

0

In case you are looking for duplicate strings that are not known beforehand, here's a script that will look for duplicate words in a set of files (a word here means anything the has spaces around it):

#!/bin/bash

dup_words() {
    zcat $1 | tr ' ' '\n' | sed '/^$/d' | sort | uniq -c | awk '{if ($1 > 1) m=1; } END { exit(!m) } '
}

for i in *.gz
do
    dup_words $i && echo $i
done

It will split lines at every occurance of a space, then remove empty lines, find and count occurrences, and finally check for more than 1 occurrence.

The loop just outputs the filename, and you can adjust the action after && as you want.

0

You can use this command to find whether the string appears in some .gz files under one folder:

zgrep the_string  *.gz -l

To do this for a list of folders that are all in one folder:

find . -name "*.gz" | xargs zgrep the_string -l

For example, the output is like this:

./tmp/2/F.tar.gz
./tmp/2/F1.tar.gz
./tmp/1/F.tar.gz
./tmp/1/F1.tar.gz

Now you get the list of files that contain the string.

  • zgrep likely has a -R option to do recursive grepping across a directory hierarchy, and also a -l option to return only the pathnames of the matching files. Most of yeur pipeline is not needed. – Kusalananda Apr 26 at 7:12
  • @Kusalananda Thanks. Yes, -l option is good. I have updated my answer. However, the -R option is not supported with zgrep – Bin Li Apr 30 at 2:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.