Searching for terms in a set of files

Question

I have two files:

files_to_search.out
terms_to_search.out

I'd like to create a command that identifies terms in terms_to_search.out that are not used in any of the files in files_to_search.out

Is there an easy way to do this?

Answer 1

This is a bit tricky if you want to account for terms that can overlap, e.g. a single line containing banana is enough to count as a use of both ban and nan.

Here's a minimally-tested, quick-and-dirty perl script. It reads the strings to search (the needles) and the file names, then builds a regular expression that matches any of the needles. When it finds a match, it removes the matched string from the set of needles and rebuilds the regex. The needles that are left over at the end are the ones you're after.

#! /usr/bin/env perl
open FILENAMES, "<", "files_to_search.out" or die $!;
@filenames = <FILENAMES>;
close FILENAMES;
chomp foreach @filenames;
open NEEDLES, "<", "terms_to_search.out" or die $!;
@needles = <NEEDLES>;
close NEEDLES;
chomp foreach @needles;
%needles = map {$_, 1} @needles;
sub build_re {
    $re = qr/(@{[join("|", map quotemeta, keys %needles)]})/;
}
@ARGV = @filenames;
while (<ARGV>) {
    while (/$re/) {
        delete $needles{$1};
        exit if !%needles;
        build_re();
    }
}
print map "$_\n", sort keys %needles;

Answer 2

quick ugly attempt at a one liner (with GNU grep for the -o option):

grep -of terms_to_search_out $(cat files_to_search.out | tr '\n' ' ') | sort | uniq | grep  -vf terms_to_search_out