1

I have two csv files where I want to figure out the deleted, modified and added lines. (The first field is an unique id.)

The first file looks like:

0, WILL_BE_REMOVED
1, OLD
2, SAME
3, SAME

and a newer csv file:

1, CHANGED
2, SAME
3, SAME
4, ADDED
5, ADDED

I was thinking that diff might tell me that, yet when I run:

diff newer_file.csv  older_file.csv 
1c1,2
< 1, CHANGED
---
> 0, WILL_BE_REMOVED
> 1, OLD
4,5c5
< 4, ADDED
< 5, ADDED
---
> 

I see a changed line similar as an added line. Is there a way to distinguish between changes and additions?

Can I achieve what I want with diff? If not, could some other bash tool fit my use case?

DELETED: 0, WILL_BE_REMOVED
UPDATED: 1, CHANGED
ADDED 4,
ADDED 5,
1
  • diff will not know that the ID in the first field is what decides the "identity" of a line. If like in your example the ID fields are unique and monotonically increasing, and the first file is small enough to fit in memory, it should be easy to create a simple Awk script which prints your expected output.
    – tripleee
    Feb 4 '19 at 12:38
5

Using the -c option (copied context) will allow you to distinguish between changes and additions:

diff -c older_file.csv newer_file.csv

produces (after the header)

*** 1,4 ****
! 0, WILL_BE_REMOVED
! 1, OLD
  2, SAME
  3, SAME
--- 1,5 ----
! 1, CHANGED
  2, SAME
  3, SAME
+ 4, ADDED
+ 5, ADDED

Changes lines start with !, added lines start with +. Deleted lines start with -.

diff can “merge” changes, so it might end up considering that added lines are part of a larger change, and mark everything with !. You can see this above where the deleted line is considered part of the larger change affecting the first two lines, and thus ends up marked ! instead of -.

Side-by-side comparisons (diff -y) help highlight some of the issues with what you’re trying to achieve:

0, WILL_BE_REMOVED    | 1, CHANGED
1, OLD                <
2, SAME                 2, SAME
3, SAME                 3, SAME
                      > 4, ADDED
                      > 5, ADDED

diff isn’t aware of the internal structure of individual lines, so it can’t determine which of the first two lines in the older file is deleted, or even whether it should consider that two lines were deleted and one added, instead of one deleted and one changed.

1
  • Nice to learn something new. I'd not realised the context diff showed changed lines as I've long since moved on to unified diffs.
    – roaima
    Feb 4 '19 at 12:29
1

A great tool to do it is daff http://paulfitz.github.io/daff/

A sort of diff specialized on CSV.

0

diff works on lines. If the line is different it is expressed in terms of a deletion and an addition.

I don't know of a tool that can comprehend the concept of "changed". Consider, if you changed a line from red to yellow how would you (as a person) determine whether the red line had been deleted and a yellow one added, or the same line changed in its entirety?

On a related side issue, personally I prefer the unified diff. I find it far more readable than an ed script. With your example data it would be expressed like this:

diff -u older_file.csv newer_file.csv
--- older_file.csv      2019-02-04 12:23:32.416529000 +0000
+++ newer_file.csv      2019-02-04 12:23:41.551570700 +0000
@@ -1,4 +1,5 @@
-0, WILL_BE_REMOVED
-1, OLD
+1, CHANGED
 2, SAME
 3, SAME
+4, ADDED
+5, ADDED

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.