2 added 6 characters in body
source | link

As Belmin showed, sort is great. His answer is best for unsorted data, and it's easy to remember and use.

However, it is also volatile, as it changes the order of the input. If you absolutely need to have the data go through in the same order but removing later duplicates, awk may be better.

$ cat myfile.csv
c
a
c
b
b
a
c 


$ awk '{if (!($0 in x)) {print $0; x[$0]=1} }' myfile.csv
c
a
b

Weird edge case, but it does come up from time to time.

Also, if your data is already sorted when you are poking at it, you can just run uniq.

$ cat myfile.csv 
a
a
a
b
b
c
c
c
c
c 


$ uniq myfile.csv 
a
b
c

Drawback to both of my suggestions is that you need to use a temporary file and copy that back in.

As Belmin showed, sort is great. His answer is best for unsorted data, and it's easy to remember and use.

However, it is also volatile, as it changes the order of the input. If you absolutely need to have the data go through in the same order but removing later duplicates, awk may be better.

$ cat myfile.csv
c
a
c
b
b
a
c
$ awk '{if (!($0 in x)) {print $0; x[$0]=1} }' myfile.csv
c
a
b

Weird edge case, but it does come up from time to time.

Also, if your data is already sorted when you are poking at it, you can just run uniq.

$ cat myfile.csv 
a
a
a
b
b
c
c
c
c
c
$ uniq myfile.csv 
a
b
c

Drawback to both of my suggestions is that you need to use a temporary file and copy that back in.

As Belmin showed, sort is great. His answer is best for unsorted data, and it's easy to remember and use.

However, it is also volatile, as it changes the order of the input. If you absolutely need to have the data go through in the same order but removing later duplicates, awk may be better.

$ cat myfile.csv
c
a
c
b
b
a
c 


$ awk '{if (!($0 in x)) {print $0; x[$0]=1} }' myfile.csv
c
a
b

Weird edge case, but it does come up from time to time.

Also, if your data is already sorted when you are poking at it, you can just run uniq.

$ cat myfile.csv 
a
a
a
b
b
c
c
c
c
c 


$ uniq myfile.csv 
a
b
c

Drawback to both of my suggestions is that you need to use a temporary file and copy that back in.

1
source | link

As Belmin showed, sort is great. His answer is best for unsorted data, and it's easy to remember and use.

However, it is also volatile, as it changes the order of the input. If you absolutely need to have the data go through in the same order but removing later duplicates, awk may be better.

$ cat myfile.csv
c
a
c
b
b
a
c
$ awk '{if (!($0 in x)) {print $0; x[$0]=1} }' myfile.csv
c
a
b

Weird edge case, but it does come up from time to time.

Also, if your data is already sorted when you are poking at it, you can just run uniq.

$ cat myfile.csv 
a
a
a
b
b
c
c
c
c
c
$ uniq myfile.csv 
a
b
c

Drawback to both of my suggestions is that you need to use a temporary file and copy that back in.