Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm trying to find out what modules use Test::Version in cpan. So I've used minicpan to mirror it. My problem is that I need to iterate through the archives that are downloaded, and grep the files that are in the archives. Can anyone tell me how I might do this? preferably in a way that tells me which file in the archive and what line it's on.

(note: they aren't all tarballs some are zip files)

share|improve this question

5 Answers 5

It appears that I can do it this way

find authors/ -type f -exec zgrep "Test::Version" '{}' +  

However, this gives results like:

authors/id/J/JO/JONASBN/Module-Info-File-0.11.tar.gz:Binary file (standard input) matches

which is not very specific to where in the tarball. Hopefully someone can come up with a better answer.

share|improve this answer

Thanks for the challenge, I came up with:

#!/bin/bash
#

# tarballs to check in
find authors/ -type f | while read tarball; do

    # get list of files in tarball (not dirs ending in /):
    tar tzf $tarball | grep -v '/$' | while read file; do       

        # get contents of file and look for string
        tar -Ozxf conform.tar.gz $file | grep -q 'Text::Version' && echo "Tar ($tarball) has matching File ($file)"

    done

done
share|improve this answer
    
Just saw your line number requirement. That can probably work with some combination of grep -n and awk to capture the line number. Can't be as simple as grep -H to list filename since it's always stdin, so might require more lines. –  Kyle Smith May 25 '11 at 14:24
    
errors out when run on my system, infinite repeated : tar (child): conform.tar.gz: Cannot open: No such file or directory tar (child): Error is not recoverable: exiting now tar: Child returned status 2 tar: Error is not recoverable: exiting now –  xenoterracide May 25 '11 at 14:43
    
also I didn't realize when I first posted this that some of the archives on cpan are zip files. –  xenoterracide May 25 '11 at 14:45
    
Hm, I tested with a structure of only .tar.gz files -- it could be made more robust to take appropriate actions based on file type, but this should give a decent starting point. –  Kyle Smith May 25 '11 at 18:35

Ok, let's apply the unix philosophy. What are the components of this task?

  • Text search: you need a tool to search text in a file, such as grep.
  • Recursive: you need a tool to go looking for files in a directory tree, such as find.
  • Archives: you need a tool to read them.

Most unix programs operate on files. So to operate easily on archive components, you need to access them as files, in other words you need to access them as directories.

The AVFS filesystem presents a view of the filesystem where every archive file /path/to/foo.zip is accessible as a directory ~/.avfs/path/to/foo/zip#. AVFS provides read-only access to most common archive file formats.

mountavfs
find ~/.avfs"$PWD" \( -name '*.zip' -o -name '*.tar.gz' -o -name '*.tgz' \) \
     -exec sh -c '
                  find "$0#" -name "*.pm" -exec grep "$1" {\} +
                 ' {} 'Test::Version' \;
fusermount -u ~/.avfs   # optional

Explanations:

  • Mount the AVFS filesystem.
  • Look for archive files in ~/.avfs$PWD, which is the AVFS view of the current directory.
  • For each archive, execute the specified shell snippet (with $0 = archive name and $1 = pattern to search).
  • $0# is the directory view of the archive $0.
  • {\} rather than {} is needed in case the outer find substitutes {} inside -exec ; arguments (some do it, some don't).
  • Optional: finally unmount the AVFS filesystem.

Or in zsh ≥4.3:

mountavfs
grep 'Test::Version' ~/.avfs$PWD/**/*.(tgz|tar.gz|zip)(e\''
     reply=($REPLY\#/**/*.pm(.N))
'\')

Explanations:

  • ~/.avfs$PWD/**/*.(tgz|tar.gz|zip) matches archives in the AVFS view of the current directory and its subdirectories.
  • PATTERN(e\''CODE'\') applies CODE to each match of PATTERN. The name of the matched file is in $REPLY. Setting the reply array turns the match into a list of names.
  • $REPLY\# is the directory view of the archive.
  • $REPLY\#/**/*.pm matches .pm files in the archive.
  • The N glob qualifier makes the pattern expand to an empty list if there is no match.
share|improve this answer
    
this creates the other intesting problem of having to mount and then unmount all of the archives, as part of the the problem is that there are 22k archives that need to be searched through –  xenoterracide May 28 '11 at 14:24
    
@xenoterracide: How is that a problem? With AVFS, you have a single mount point (~/.avfs), and access to each archive is automatic (~/.avfs/path/to/archive.zip\# is an ordinary directory on the AVFS filesystem, not a mount point). Sure, each archive you access means a little performance hit, but that's intrinsic to the problem. –  Gilles May 28 '11 at 14:31
    
@gilles only the fact that now I have to go through and figure out how to mount them first, which seems like a bit of a bad idea, better to mount them as I go and unmount after being searched. –  xenoterracide May 28 '11 at 15:14
    
@xenoterracide: Again: no, you don't need to mount them individually. The full workflow (apart from installing AVFS if needed) is in my code snippets. –  Gilles May 28 '11 at 15:18
    
@gilles well I'll have to dig into this a bit... because I get find: missing argument to -exec'` and lots of this from zsh zsh: Input/output error: Data-Maker-0.27 –  xenoterracide May 28 '11 at 15:36

Maybe my answer will helpfull for someone:

#!/bin/bash

findpath=$(echo $1 | sed -r 's|(.*[^/]$)|\1/|')

# tarballs to check in
find $findpath -type f | while read tarball; do

    # get list of files in tarball (not dirs ending in /):
    if [ -n "$(file --mime-type $tarball | grep -e "application/jar")" ]; then

        jar tf $tarball | grep -v '/$' | while read file; do
            # get contents of file and look for string
            grepout=$(unzip -q -c $tarball $file | grep $3 -e "$2")

            if [ -n "$grepout" ]; then
                echo "*** $tarball has matching file ($file):"
                echo $grepout
            fi

        done

    elif tar -tf $tarball 2>/dev/null; then

        tar -tf $tarball | grep -v '/$' | while read file; do
            # get contents of file and look for string
            grepout=$(unzip -q -c $tarball $file | grep $3 -e "$2")

            if [ -n "$grepout" ]; then
                echo "*** $tarball has matching file ($file):"
                echo $grepout
            fi

        done

    else
        file=""
        grepout=$(grep $3 -e "$2" $tarball)

        if [ -n "$grepout" ]; then
            echo "*** $tarball has matching:"
            echo $grepout
        fi

    fi

done
share|improve this answer

After installing p7zip-* you are able to do this:

ls | xargs -I {} 7z l {} | grep whatever | less

You don't have to use ls before the first pipe, whatever list the compressed files will work. The final less only will show the PATH of the listet life inside the compressed archive, but not the name of this.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.