This is a follow up question from Sort large CSV files (90GB), Disk quota exceeded. So now I have two CSV files sorted, as file1.csv and file2.csv each CSV file has 4 columns, e.g.
file 1:
ID Date Feature Value
01 0501 PRCP 150
01 0502 PRCP 120
02 0501 ARMS 5.6
02 0502 ARMS 5.6
file 2:
ID Date Feature Value
01 0501 PRCP 170
01 0502 PRCP 120
02 0501 ARMS 5.6
02 0502 ARMS 5.6
Ideally, I want to diff the two files in such a way that if two rows in the two files have the same ID, Date and Feature, but different values, then output something like:
ID Date Feature Value1 Value2
Of course, this might be asking too much. Something like
ID1 Date1 Feature1 Value1 ID2 Date2 Feature2 Value2
also works. In the above example, I would like to output
01 0501 PRCP 150 170
or
01 0501 PRCP 150 01 0501 PRCP 150
I think the main question is how to compare in such a way and how to output to a csv file. Thanks.
Sample output from Gilles answer: The output from comm is
$ head -20 comm_output.txt
ACW00011604,19490101,PRCP,0
AE000041196,20070402,TAVG,239
AE000041196,20070402,TAVG,244
AE000041196,20080817,TMIN,282
AE000041196,20130909,TAVG,350
AE000041196,20130909,TMAX,438
AE000041196,20130909,TMIN,294
AE000041196,20130910,TAVG,339
AE000041196,20130910,TAVG,341
AE000041196,20150910,TAVG,344
The output of awk is
$ head awk_output.csv
,
ACW00011604,19490101,PRCP,0,,,
AE000041196,20070402,TAVG,239,,,
AE000041196,20070402,TAVG,244,,,
AE000041196,20080817,TMIN,282,,,
AE000041196,20130909,TAVG,350,,,
AE000041196,20130909,TMAX,438,,,
AE000041196,20130909,TMIN,294,,,
AE000041196,20130910,TAVG,339,,,
AE000041196,20130910,TAVG,341,,,
AE000041196,20150910,TAVG,344,,,
Here is the sample input, if you insist
head file1.csv
ACW00011604,19490101,PRCP,0
ACW00011604,19490101,SNOW,0
ACW00011604,19490101,SNWD,0
ACW00011604,19490101,TMAX,289
ACW00011604,19490101,TMIN,217
ACW00011604,19490102,PRCP,30
ACW00011604,19490102,SNOW,0
ACW00011604,19490102,SNWD,0
ACW00011604,19490102,TMAX,289
ACW00011604,19490102,TMIN,228
head file2.csv
ACW00011604,19490101,SNOW,0
ACW00011604,19490101,SNWD,0
ACW00011604,19490101,TMAX,289
ACW00011604,19490101,TMIN,217
ACW00011604,19490102,PRCP,30
ACW00011604,19490102,SNOW,0
ACW00011604,19490102,SNWD,0
ACW00011604,19490102,TMAX,289
ACW00011604,19490102,TMIN,228
ACW00011604,19490102,WT16,1
paste file1 file2
might work.