I am generating all the common/duplicate lines between 5 files all within the same directory
sort *.txt | uniq -d > output.txt
output.txt to be merged with File1.txt (so anything that isn't a duplicate is added) however elsewhere, in File2.txt if the duplicate line appears it needs to be removed.
Any pointers? Using Debian
EDIT: Merging output with File1 is easy as
cat output.txt >> File1.txt
then just removing the duplicates after with a regular expression.