Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have two files:

files_to_search.out
terms_to_search.out

I'd like to create a command that identifies terms in terms_to_search.out that are not used in any of the files in files_to_search.out

Is there an easy way to do this?

share|improve this question
add comment

2 Answers

up vote 1 down vote accepted

This is a bit tricky if you want to account for terms that can overlap, e.g. a single line containing banana is enough to count as a use of both ban and nan.

Here's a minimally-tested, quick-and-dirty perl script. It reads the strings to search (the needles) and the file names, then builds a regular expression that matches any of the needles. When it finds a match, it removes the matched string from the set of needles and rebuilds the regex. The needles that are left over at the end are the ones you're after.

#! /usr/bin/env perl
open FILENAMES, "<", "files_to_search.out" or die $!;
@filenames = <FILENAMES>;
close FILENAMES;
chomp foreach @filenames;
open NEEDLES, "<", "terms_to_search.out" or die $!;
@needles = <NEEDLES>;
close NEEDLES;
chomp foreach @needles;
%needles = map {$_, 1} @needles;
sub build_re {
    $re = qr/(@{[join("|", map quotemeta, keys %needles)]})/;
}
@ARGV = @filenames;
while (<ARGV>) {
    while (/$re/) {
        delete $needles{$1};
        exit if !%needles;
        build_re();
    }
}
print map "$_\n", sort keys %needles;
share|improve this answer
add comment

quick ugly attempt at a one liner (with GNU grep for the -o option):

grep -of terms_to_search_out $(cat files_to_search.out | tr '\n' ' ') | sort | uniq | grep  -vf terms_to_search_out 
share|improve this answer
1  
Note that all of \n, \t and space are used by default to split command substitutions, so there's no point changing '\n' to ' '. Restricting IFS to newline and do the reverse conversion would even make more sense since newline characters are less likely to occur in file names than space characters. Also note that filename generation is also performed (except in zsh) upon command substitution so blank characters in filenames are not the only problematic ones (see set -f to disable filename generation). –  Stephane Chazelas Nov 7 '12 at 22:27
    
learn something new everyday - wasn't aware about the \n defaults. So maybe pass through sed to add quotes? or go with a for loop? –  Drake Clarris Nov 8 '12 at 15:00
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.