I have two csv file of the following format:
column1,column2,user,column4,column5,column...column14
I would like to sort the files based on user as first key and then based on column14 in descending order which is a number representing a timestamp.
The result would be like:
file1:
A,AA,jim,XX,YY,....,1485771395
A,AA,jim,XX,YY,....,1485771395
etc
I would like then from these files to keep only the first row of each cluster i.e. only
A,AA,jim,XX,YY,....,1485771395
How can I do that?
Update:
Example input:
"1/30/2017 11:14:55 AM",Valid customer,jim.smith,NY,1485771295
"1/26/2017 5:06:11 AM",New customer,john.doe,CA,1485403571
"1/26/2017 8:02:01 PM",Valid customer,jim.smith,NY,1485457321
"1/30/2017 4:14:30 AM",New customer,tim.jones,CO,1485746070
"1/30/2017 11:09:36 AM",New customer,tim.jones,CO,1485770976
"1/30/2017 11:14:50 AM",Valid customer,jim.smith,NY,1485771290
"1/22/2017 11:51:51 AM",New customer,tim.jones,CO,1485082311
The last column is the epoch of the first column to be able to sort the rows using a number instead of a string.
So the expected output would be:
a) Sorting on name and on timestamp in reverse order so we get:
"1/30/2017 11:14:55 AM",Valid customer,jim.smith,NY,1485771295
"1/30/2017 11:14:50 AM",Valid customer,jim.smith,NY,1485771290
"1/26/2017 8:02:01 PM",Valid customer,jim.smith,NY,1485457321
"1/26/2017 5:06:11 AM",New customer,john.doe,CA,1485403571
"1/30/2017 11:09:36 AM",New customer,tim.jones,CO,1485770976
"1/30/2017 4:14:30 AM",New customer,tim.jones,CO,1485746070
"1/22/2017 11:51:51 AM",New customer,tim.jones,CO,1485082311
So we have a cluster of rows for jim.smith
another for tim.jones
sorted by the last column in reverse order (i.e. the latest date of the first row is first in the cluster) and a row for john.doe
that has only 1 record.
Then I would like to keep only the first row of each cluster. I.e.
"1/30/2017 11:14:55 AM",Valid customer,jim.smith,NY,1485771295
"1/26/2017 5:06:11 AM",New customer,john.doe,CA,1485403571
"1/30/2017 11:09:36 AM",New customer,tim.jones,CO,1485770976
sort -u
oruniq
?sort -t, -k3,3 -k14,14n file.csv | awk -F, '!seen[$3]++'
maybe?