2

I have a tab-separated CSV file like this

C1      C2
A       3
B       4
C       5
B       4
A       3

How can I get rid of duplicates in col 1 and output the values from col 2 for each unique col1 value? Example output:

3
4
5

I've tried awk -F '\t' '{print $1}' file_name.csv | sort | uniq -c which sorts the data in the first column and 'drops' duplicates but this gets rid of column 2. I'm just a beginner to shell scripting so any help will do.

2
  • 1
    hi llamar25, is your CSV file formatted like that or is it comma-separated?
    – BakaKuna
    May 23, 2020 at 8:26
  • @BakaKuna it's tab separated. I just drew the table to make it easier to visualise
    – llamaro25
    May 23, 2020 at 8:27

4 Answers 4

4
awk -F '\t' 'NR>1 && !seen[$1]++{print $2}' file_name.csv
  • NR>1 is false for first line, then it is skipped.
  • !seen[$1]++ is only 1 (or true) for a 1st field value that has been parsed for the first time (duplicates evaluate to 0, or false).
  • print $2 prints the 2nd field if both conditions were met.
1

A non awk version using your sort and uniq

tail -n +2 file | sort -ugk2 | column -t -H 1

lose the header by printing the tail from line -number +2 of the file

tail -n +2 file | 

sort -unique using -general numeric on -key2

sort -ugk2 | 

and output as columnar -table but -Hide column 1

column -t -H 1
1

If you want only the first two columns, you could use:

awk -F '\t' '!seen[$1]++{print $2}' testfile
3
  • And notice this does not get rid of duplicates, but it rather prints every duplicate, even though for the sample file, it's all the same.
    – Quasímodo
    May 23, 2020 at 10:46
  • @roaima: Right! I forgot to remove the first column.
    – BakaKuna
    May 23, 2020 at 18:33
  • @Quasímodo: Thanks, I must have copy pasted the wrong line.
    – BakaKuna
    May 25, 2020 at 9:41
0

Miller has a groupwise head that allows you to take the first n records matching a particular column value (in this case, n=1):

$ mlr --tsv head -g C1 -n 1 then cut -f C2 file.tsv
C2
3
4
5

If you don't want the column headers, keep the tsv input format but change the output format to indexed:

mlr --itsv --onidx head -g C1 -n 1 then cut -f C2 file.tsv

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .