0

I have seen command "uniq" to remove duplicate entries in file but My purpose is to remove duplicate output before saving in file

#!/bin/sh
echo "aaa"
echo "bbb"
echo "aaa"
echo "aaa"
echo "bbb"
echo "ccc"

I want to the output to be

aaa
bbb
ccc

and then redirect to text file

Is it possible to do this way? Thank you.

  • why not removing duplicated commands echo …. in your script then? – αғsнιη Jan 8 at 9:14
4

uniq only works on sorted input.

./my_script.sh | sort -u > output_file

is essentially the same as

./my_script.sh | sort | uniq > output_file
2

awk '!a[$0]++' is a compact way of removing unsorted duplicated lines, keeping original order.

Using this we can for example:

  • filter the output: ./my_script |awk '!a[$0]++' > output
  • filter the script: awk '!a[$0]++' my_script > new_script
  • filter and execute the script: awk '!a[$0]++' my_script | sh > output
  • filter inside vim : :%!awk '\!a[$0]++'

The idea behind this is to keep track of patterns already found by storing the number of occurences in an array a and only printing a line if the value of a for the specific content of the line (represented by $0 in awk) is still zero. So,

  • a[$0]++ increments the array entry for the "index" $0 (= the content of the current input line) by one
  • !a[$0] is a filter rule that instructs awk to only print a line if the array entry for the content is still zero (which is implicitly the case when it is uninitialized)
  • !a[$0]++ combines the two in one line (a.k.a. "code golfing")

...and we can also make it more readable with a bash alias:

alias myuniq='awk '\''!a[$0]++'\'


\thanks{@AdminBee}

  • @AdminBee: thank you! – JJoao Jan 8 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.