How to remove duplicates using awk in 1th~46th column of 47 field csv file?

Question

I have a bash script to combine many files into a single csv file. This csv file has 47 fields and several 10000 rows.

To remove duplicates I was using awk '!seen[$0]++' however, for sorting I have added a reference to the originating source in column 47.

I still only want to index on column 1 to 46 and print all including 47.

Must I list all 46 as in awk '!seen[$1, $2, $3, ,..etc.., $45, $46]++' or is there an easier way?

Answer 1

'sort_field=$47; $47=""; !seen[$0]++ { print $0 " " sort_field }' 

Answer 2

Use sort and print unique lines with its uniqe -u option based on 1~46 fields k1,46?

sort -uk1,46 infile.txt

if your input files is a .csv comma separated, you can specify that with -t','.

sort -t',' -uk1,46 infile.txt

For below input as sample:

1,2,3,4
5,6,7,8
1,2,3,x
a,b,c,d
5,6,7,y

The output is:

1,2,3,4
5,6,7,8
a,b,c,d

Answer 3

Try this:

$ cat file
1 2 3 4
5 6 7 8
1 2 3 x
a b c d
5 6 7 y

Looking at the first 3 fields as the "key", we want to drop the 2nd "1 2 3" and the 2nd "5 6 7" lines

awk '
    {
        line = $0     # remember the original state of this line
        NF--          # forget about the last field
    }
    !seen[$0]++ {print line}   # if the "new" line is unique, print the "old" line
' file

1 2 3 4
5 6 7 8
a b c d