0

I have a bash script to combine many files into a single csv file. This csv file has 47 fields and several 10000 rows.

To remove duplicates I was using awk '!seen[$0]++' however, for sorting I have added a reference to the originating source in column 47.

I still only want to index on column 1 to 46 and print all including 47.

Must I list all 46 as in awk '!seen[$1, $2, $3, ,..etc.., $45, $46]++' or is there an easier way?

0

Try this:

$ cat file
1 2 3 4
5 6 7 8
1 2 3 x
a b c d
5 6 7 y

Looking at the first 3 fields as the "key", we want to drop the 2nd "1 2 3" and the 2nd "5 6 7" lines

awk '
    {
        line = $0     # remember the original state of this line
        NF--          # forget about the last field
    }
    !seen[$0]++ {print line}   # if the "new" line is unique, print the "old" line
' file
1 2 3 4
5 6 7 8
a b c d
  • I see I'm duplicating @hauke's answer. making community wiki – glenn jackman Aug 20 '17 at 15:23
  • I had to change line = $0 to line = $47 to get this to work and as I am working with a comma separated file I also amended the print line to {print $0 "," line} – triedonce Aug 21 '17 at 2:32
1
'sort_field=$47; $47=""; !seen[$0]++ { print $0 " " sort_field }' 
1

Use sort and print unique lines with its uniqe -u option based on 1~46 fields k1,46?

sort -uk1,46 infile.txt

if your input files is a .csv comma separated, you can specify that with -t','.

sort -t',' -uk1,46 infile.txt

For below input as sample:

1,2,3,4
5,6,7,8
1,2,3,x
a,b,c,d
5,6,7,y

The output is:

1,2,3,4
5,6,7,8
a,b,c,d

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.