Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a lot CSV files that I have combined. However, there are duplicates, but the entire line is not duplicated. I do have a column that I want to use as the criteria to search for a duplicate. And if there is a duplicate in that column from the entire column, then delete the rows that contain the duplicates in the columns until you have all unique values in this column.

Does anyone know the best way to accomplish this in Bash, sed or awk?

share|improve this question
Could you please provide some sample data and expected output of it to help pinpoint what you are after? Either way it seams easier to accomplish with a script language. See e.g. Ruby's CSV class. – N.N. Mar 22 '13 at 17:36
awk -F, '!seen[$1]++'

$1 is the first column, change as appropriate; you can use multiple columns separated by commas ([$1,$3]), or $0 for the whole row.

share|improve this answer
Wow! This is amazingly simple! Thanks. This works perfectly. For any super noobs like me. You simply need to put the csv after the command above & if you want to output it, then use this >. It would look like this: awk -F, '!seen[$1]++' inputfile.csv > outputfile.csv. – Peaceful_Warrior Mar 23 '13 at 21:00

Bash is hard but you can you call Perl from Bash? If your fields are separated by commas, and your key field is the second one,

$ cat a.csv

this one liner will print if it hasn't seen the second column already:

$ perl -ne '$value = (split /,/)[1]; print unless $x{$value}++;' a.csv 

The [1] is the column number and is zero-based.

share|improve this answer

If there are fields that contain delimiter characters inside quotes (doesn't work with Ruby 1.8):

$ ruby -rcsv -e 'puts CSV.parse($<).uniq{|l|l[1]}.map(&:to_csv)' <<< $'"1,9",3\n4,8\n7,3'
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.