I have a
.csv file of 6.5 GB. I need to sort the files with 1st, 2nd, 3rd, 4th, 5th and 13th column and need to get the unique records. Also I need to get the duplicate records and redirect it to some other file.
I am using the following command to get the unique lines:
awk -F',' 'a[$1,$2,$3,$4,$5,$13]++' /var/tmp/Data.txt > unique.txt
But the above process is consuming more than 3.5 GB memory and eventually gets terminated. Producing the following error in the log file:
sh: line 1: 7895 Killed awk -F',' 'a[$1,$2,$3,$4,$5,$13]++' /var/tmp/Data.txt > unique.txt
How can I produce the required output?