0

I am generating all the common/duplicate lines between 5 files all within the same directory

sort *.txt | uniq -d > output.txt

I need output.txt to be merged with File1.txt (so anything that isn't a duplicate is added) however elsewhere, in File2.txt if the duplicate line appears it needs to be removed.

Any pointers? Using Debian

EDIT: Merging output with File1 is easy as

cat output.txt >> File1.txt

then just removing the duplicates after with a regular expression.

closed as unclear what you're asking by Kusalananda, don_crissti, sam, GAD3R, Anthon Jan 23 '17 at 20:31

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Is the order of the lines in the output of any importance? – Kusalananda Jan 23 '17 at 9:23
  • 2
    Can you add an example please ? – Wissam Roujoulah Jan 23 '17 at 9:25
  • @Kusalananda No, it isnt. – James Berrisford Jan 23 '17 at 9:48
  • @MhdWissamAl-Roujoulah In file 1 if anything is within output.txt that is additional and not a duplicate gets added. Yet in file 2 if a duplicate line from output.txt appears it is removed. These files are in different locations – James Berrisford Jan 23 '17 at 9:50
  • 2
    @JamesBerrisford Please update the question with all the relevant information. – Kusalananda Jan 23 '17 at 10:11
2

You can cat both files (output and file1.txt) then find uniq lines using just uniq (no switces) and finally write the result back to file1.txt

Example:

$ cat output.txt
Line1
$ cat file1.txt
Line1
Line3
Line4
$ echo "$(cat output.txt file1.txt |sort |uniq)" >file1.txt
$ cat file1.txt
Line1
Line3
Line4

If you want "Line1" to be completelly removed and not kept even once, then use uniq -u.

$ echo "$(cat output.txt file1.txt |sort |uniq -u)" >file1.txt && cat file1.txt
Line3
Line4

Similar operation for your file2.

PS: You can not cat and write on the same file at one operation. You have either to use the echo $(...) trick above or send the cat |uniq output to another file (temp file) and then either rename tempfile to file1 (mv tempfile file1.txt) or cat tmpfile >file1.txt && rm tmpfile.

Another solution would be to append output.txt to file1.txt (cat output.txt >>file1.txt) and use sed -i for removing the duplicate lines

  • sed -i File2.txt isn't removing duplicate lines – James Berrisford Jan 23 '17 at 11:52
  • @JamesBerrisford No , sed will need a regex to remove duplicate lines, does not work like this. Also it might require file to be sorted first. See this post: stackoverflow.com/questions/1444406/… . Solution other than sed which supports inplace / in file replacements (-i switch) also might have the restriction that output file can not be the same as input file. – George Vasiliou Jan 23 '17 at 11:55
  • @JamesBerrisford also my solution requires sort. I will update my answer to be accurate and correct. – George Vasiliou Jan 23 '17 at 12:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.