12

I am trying to find a way to determine if a text file is a subset of another..

For example:

foo
bar

is a subset of

foo
bar
pluto

While:

foo
pluto

and

foo
bar

are not a subset of each other...

Is there a way to do this with a command?

This check must be a cross check, and it has to return:

file1 subset of file2 :    True
file2 subset of file1 :    True
otherwise             :    False
11

If those file contents are called file1, file2 and file3 in order of apearance then you can do it with the following one-liner:

 # python -c "x=open('file1').read(); y=open('file2').read(); print x in y or y in x"
 True
 # python -c "x=open('file2').read(); y=open('file1').read(); print x in y or y in x"
 True
 # python -c "x=open('file1').read(); y=open('file3').read(); print x in y or y in x"
 False
|improve this answer|||||
  • Thanks for your answer.. +1 .. I don't know if accept my answer because yours is not unix-linux specific and my answer is a bit faster, as far as I tested it.. what do you think? – gc5 Feb 12 '14 at 13:12
  • You welcome, there are of course other solutions with more unix specific tools. But this seems a good use of Python's in operator. – Timo Feb 12 '14 at 13:21
  • There is python command line wrapper to make it more unix like, with piping built in, named pyp: code.google.com/p/pyp I think it is trivial to make this solution more unix like one liner tool. – IBr Nov 14 '14 at 9:15
3

With perl:

if perl -0777 -e '$n = <>; $h = <>; exit(index($h,$n)<0)' needle.txt haystack.txt
then echo needle.txt is found in haystack.txt
fi

-0octal defines the record delimiter. When that octal number is greater than 0377 (the maximum byte value), that means there's no delimiter, it's equivalent to doing $/ = undef. In that case, <> returns the full content of a single file, that's the slurp mode.

Once we have the content of the files in two $h and $n variables, we can use index() to determine if one is found in the other.

That means however that the whole files are stored in memory which means that method won't work for very large files.

For mmappable files (usually includes regular files and most seekable files like block devices), that can be worked around by using mmap() on the files, like with the Sys::Mmap perl module:

if 
  perl -MSys::Mmap -le '
    open N, "<", $ARGV[0] || die "$ARGV[0]: $!";
    open H, "<", $ARGV[1] || die "$ARGV[1]: $!";
    mmap($n, 0, PROT_READ, MAP_SHARED, N);
    mmap($h, 0, PROT_READ, MAP_SHARED, H);
    exit (index($h, $n) < 0)' needle.txt haystack.txt
then
  echo needle.txt is found in haystack.txt
fi
|improve this answer|||||
2

I found a solution thanks to this question

Basically I am testing two files a.txt and b.txt with this script:

#!/bin/bash

first_cmp=$(diff --unchanged-line-format= --old-line-format= --new-line-format='%L' "$1" "$2" | wc -l)
second_cmp=$(diff --unchanged-line-format= --old-line-format= --new-line-format='%L' "$2" "$1" | wc -l)

if [ "$first_cmp" -eq "0" -o "$second_cmp" -eq "0" ]
then
    echo "Subset"
    exit 0
else
    echo "Not subset"
    exit 1
fi

If one is subset of the other the script return 0 for True otherwise 1.

|improve this answer|||||
  • What does %L do? This script doesn't seem to work, and I am trying to debug it... – Alex May 24 '17 at 16:18
  • I actually don't remember the meaning of %L, it was three years ago. From man diff (current version) %L means "contents of line". – gc5 May 24 '17 at 18:56
  • %L prints the contents of the "new" line. IOW, don't print anything for unchanged-lines or old-lines, but print the contents of the line for new-lines. – PLG Sep 26 '17 at 11:44
  • This script works for me, out of the box! – PLG Sep 26 '17 at 17:56
2

If f1 is a subset of f2 then f1 - f2 is an empty set. Building on that we can write an is_subset function and a function derived from it. As per Set difference between 2 text files



sort_files () {
  f1_sorted="$1.sorted"
  f2_sorted="$2.sorted"

  if [ ! -f $f1_sorted ]; then
    cat $1 | sort | uniq > $f1_sorted
  fi

  if [ ! -f $f2_sorted ]; then
    cat $2 | sort | uniq > $f2_sorted
  fi
}

remove_sorted_files () {
  f1_sorted="$1.sorted"
  f2_sorted="$2.sorted"
  rm -f $f1_sorted
  rm -f $f2_sorted
}

set_union () {
  sort_files $1 $2
  cat "$1.sorted" "$2.sorted" | sort | uniq
  remove_sorted_files $1 $2
}

set_diff () {
  sort_files $1 $2
  cat "$1.sorted" "$2.sorted" "$2.sorted" | sort | uniq -u
  remove_sorted_files $1 $2
}

rset_diff () {
  sort_files $1 $2
  cat "$1.sorted" "$2.sorted" "$1.sorted" | sort | uniq -u
  remove_sorted_files $1 $2
}

is_subset () {
  sort_files $1 $2
  output=$(set_diff $1 $2)
  remove_sorted_files $1 $2

  if [ -z $output ]; then
    return 0
  else
    return 1
  fi

}

|improve this answer|||||
  • Should this script start with #!/bin/bash? – Alex May 24 '17 at 16:20
2

From http://www.catonmat.net/blog/set-operations-in-unix-shell/:

Comm compares two sorted files line by line. It may be run in such a way that it outputs lines that appear only in the first specified file. If the first file is subset of the second, then all the lines in the 1st file also appear in the 2nd, so no output is produced:

$ comm -23 <(sort subset | uniq) <(sort set | uniq) | head -1
# comm returns no output if subset ⊆ set
# comm outputs something if subset ⊊ set
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.