Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I find myself needing to find and identify extraneous files (out of about 900K files on a 2T drive). There are lots of files that I want to keep, and I have filename patterns for these known good files. What I want is to locate those files that don't fit any of the patterns.

How do I find files not matching a list of filename patterns?

I can run find to get a list of all files, and I could use grep -v on the result, using a list of patterns stored in a file. Is this the canonical method, or do you have a concise way to find these non-conforming files?


Clarification - based upon the answers, here is a little more information. I expect to have numerous patterns (>20, maybe >100), I want to store them in a file and certainly want an easy way to add new ones. I would prefer to avoid directly editing a large list of find parameters (fragile), but building that list might work.

share|improve this question
    
The Perl answer assumes you have the patterns stored in a separate file and reads from that. It tries to match the file names against the patterns either literally or by interpreting the patterns as globs. –  Joseph R. Oct 17 '13 at 18:03
    
I've edited my answer to address the "patterns in a file" requirement –  Warren Young Oct 17 '13 at 19:10
add comment

2 Answers

up vote 3 down vote accepted

Since you mention Perl...

#!/usr/bin/perl

use strict;
use warnings;
use File::Find qw{find};

my %patterns;
while (<>) {
  chomp;
  $patterns{$_}++;
}

die "No pattern supplied\n" unless keys %patterns;

find( 
    sub{
           my $matches_a_pattern=0;
           for my $pattern (keys %patterns){
               my $glob_pattern = $pattern;
               for($glob_pattern){
                   s/\./\\./g;
                   s/\*/.*/g;
                   s/\?/./g;
               }
               $matches_a_pattern++ if ( /\Q$pattern\E/ or /$glob_pattern/);
           }

           print "$File::Find::name\n" unless $matches_a_pattern;
     }
    , '.' )

Invoke this as

/path/to/my/script file_with_patterns

Replace the . at the end with the top of the tree you want to walk.

share|improve this answer
add comment

find(1) is powerful enough to do what you need. Simply collect all of the conforming names into an expression using parentheses, then negate it to show non-conforming file names. For example, to show all files not named *.txt, *.bz2, or *.zip:

$ find . \! \( -name \*.txt -o -name \*.bz2 -o -name \*.zip \)

You can use -not instead of \! with GNU and BSD find. It isn't POSIX compliant, but it doesn't require an escape to prevent the shell from interpreting it.

To build the expression from patterns in a file, it's a small matter of shell scripting:

#!/bin/sh
set --
while IFS= read -r pattern
do
    set -- "$@" -o "$pattern"
done < .fnpatterns
if [ $# -ne 0 ]; then
  shift
  set -- -not \( "$@" \)
fi
find . "$@"

This expects a file in the current directory called .fnpatterns with one pattern per line. To mimic the one-liner above, it would need to contain:

*.txt
*.bz2
*.zip

Note that the shell script escapes the * characters in the patterns for you.

You can make this arbitrarily complex. Some ideas:

  • Add -type f to the find command so it only shows normal files, not directories.

  • Pass the pattern file name in as an argument instead of expecting it in a fixed location

  • Keep the pattern file where it is, but add -o -name .fnpatterns to the built-up find command so it doesn't show up in the output. (This would also avoid the need for the shift hack to "eat" the lead -o in the built-up expression.)

  • Add actions to the find command via -exec or similar.

  • Allow blank lines or comments in the patterns file

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.