4

Let's say I have two lines( in a txt file)

Monday, Tuesday, Week
Tuesday, Monday, Week

They contain the same information, but the information(the first and second column) is in a different order in each line, so i cannot simply use sort, or uniq to get rid of them.

How can i delete all the duplicate lines, that contain the same information?

5

With perl:

perl -lne 'print unless $seen{join ",", sort split /,\h*/}++'
1

If you don't care if you preserve an order within the line seen within the file, you would put each line in a standard (or canonical) format, and then use sort/uniq or similar.

Since you've added that you do care about preserving one of the orders represented in the file, the approach I'd take is to sort each line into canonical order, and output that along with the original line, then sort on that first canonicalized representation as a key, and drop lines with duplicate keys, then trim off the key.

This is the first part. It prepends each original line with a canonicalized representation of the line contents:

( while read f; do  echo $f | tr -d "," | tr " " "\n" | sort | tr "\n" " " ; echo ':' $f ; done  < data ) | awk -F":" '!_[$1]++' | cut -f2 -d: |cut -c2-

You can run pieces of that to see what it does. The first part emits the original records with a canonicalized representation (lexically sorted) with a : delimiter:

 $ cat data
Monday, Tuesday, Week
Tuesday, Monday, Week
Tuesday, Thursday, Week
Week, Thursday, Tuesday

 $ ( while read f; do  echo $f | tr -d "," | tr " " "\n" | sort | tr "\n" " " ; echo ':' $f ; done  < data )
Monday Tuesday Week : Monday, Tuesday, Week
Monday Tuesday Week : Tuesday, Monday, Week
Thursday Tuesday Week : Tuesday, Thursday, Week
Thursday Tuesday Week : Week, Thursday, Tuesday

Then I use awk to build a hash with a count of each time a key was seen, with an implicit print for the first instance of each, due to the not with '!'

 $ ( while read f; do  echo $f | tr -d "," | tr " " "\n" | sort | tr "\n" " " ; echo ':' $f ; done  < data ) | awk -F":" '!_[$1]++' | cut -f2 -d: |cut -c2-
Monday, Tuesday, Week
Tuesday, Thursday, Week
  • I do care about the first order. – sadboy Nov 24 '16 at 3:49
  • I mean the order of first occurence – sadboy Nov 24 '16 at 3:55
1

Tricky solution (this is just a sample) - works ok in my bash. All lines / words can be split in characters and then those characters can be sorted. If sorted line 1 = sorted line 2 then you have the duplicate.

word1+=( $(echo "this is my life" |fold -w1) )
sortedword1=($(echo ${word1[@]} | tr " " "\n" | sort))
word2+=( $(echo "is this my life" |fold -w1) )
sortedword2=($(echo ${word1[@]} | tr " " "\n" | sort))
echo "${sortedword1[@]}"
echo "${sortedword2[@]}"

if [[ $sortedword1 == $sortedword2 ]]; then
echo "Word 1 and Word 2 are the same, delete one of them"
fi

Output:

e f h i i i l m s s t y
e f h i i i l m s s t y
Word 1 and Word 2 are the same, delete one of them

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.