-1

Remove duplicate words from between lists example:

We have two lists; the first list contains:

a
b
c
d

The second list contains:

a
b
c
d
e
f

I want to make a comparison between the first and second lists, removing the matches contained in both, resulting in this:

e
f

I couldn't find a solution to do that with bash, but I did find one in python: https://stackoverflow.com/questions/7961363/removing-duplicates-in-lists/7961390#7961390

  • 4
    Have you tried something so far? Folks here will be more inclined to help if you show some effort. – schrodigerscatcuriosity Oct 17 at 14:16
  • I couldn't find a solution to do that with bash But a solution was found in Python stackoverflow.com/a/7961390/13079083 It's similar to what I want to do but I don't know a function like set in BASH – abdallaEG Oct 17 at 14:25
  • Does this answer your question? Two files comparison in bash script? – schrodigerscatcuriosity Oct 17 at 14:31
  • It looks like it but the list must be similar to give correct results list a a \ nb \ nc \ nd ... list b a \ nb \ nc \ nd \ ne ... If the letters are not in order, it will give false and incomplete results Especially if you use long words in the comparison, all results will be incorrect – abdallaEG Oct 17 at 15:24
  • The grep solution linked will remove all duplicate lines. The files do not need to be "similar" to each other. – markasoftware Oct 18 at 3:32
1

You can use diff using --GTYPE-group-format=GFMT option. From man diff :

--GTYPE-group-format=GFMT
    format GTYPE input groups with GFMT

LTYPE is 'old', 'new', or 'unchanged'.
    GTYPE is LTYPE or 'changed'.

    GFMT (only) may contain:

%<     lines from FILE1

%>     lines from FILE2

%=     lines common to FILE1 and FILE2

In your case, you can use diff --new-group-format='%>' --unchanged-group-format='' list1 list2

$ cat list1
a
b
c
d

$ cat list2
a
b
c
d
e
f

$ diff --new-group-format='%>' --unchanged-group-format='' list1 list2
e
f

Explanation

  • The --new-group-format='%>' will output any new entries from FILE2 (%>) that doesn't exist in FILE1.
  • The --unchanged-group-format='' will prevent diff to print any identical lines.
| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.