1

how can I print difference line number ignoring 1st column for tab separeated files?

example- compare File 1 vs File2 by ignoring 1st column and print line no. for differnce record present in file 2.

In file1:

user1   fistname    Lastnamename
user2   Johnny            Depp
user3   Tom               Cruise
user4   Leonardo          DiCaprio
user5   Sylvester         Stallone

and

In file2:

user10   fistname    Lastnamename
user2   Johnny            Depp
user30  Tom'               Cruise
user4   Nicolas           Cage
user50  Sylvester         Stallone

expected result:- difference in file2 is for line number 3,4

Note file size to be compare is in GB and File is tab separated

2

4 Answers 4

1

You can use diff command along with cut to find the difference in the two file.

diff <(cut -f2 -d$'\t' file1) <(cut -f2 -d$'\t' file2)

Output will be

3,4c3,4
< Tom
< Leonardo
---
> Tom'
> Nicolas

If you concern about the more duplicate records, then you can use above command with sort -u for removing duplicate before finding diff from the other file. Command will be

diff <(cut -f2 -d$'\t' file1|sort -u) <(cut -f2 -d$'\t' file2|sort -u)
1
  • Note that TAB is the default delimiter in cut, so you can omit the -d$'\t' May 13, 2019 at 6:11
0

Try with this code, don't know if this will work as I don't have enough data:

diff --unchanged-line-format="" --old-line-format="" --new-line-format=":%dn: %L" file1 file2 | sed 1d | cut -d':' -f2 |tr '\n' ','|sed 's/,$//g'
0

This could answer your question:

awk 'NR==FNR{++a[$2,$3];next} {line++;if(!(a[$2,$3])){print line}}' record1 record2

Explanation:

FNR==NR

When you have two (or more) input files to awk, FNR will reset back to 1 on the first line of the next file whereas NR will continuing incrementing from where it left off. By checking FNR==NR we are essentially checking to see if we are currently parsing the first file.

++a[$2,$3]

If we are parsing the first file (see above) then create an associative array with the first field $2 and second field $3 as the key and post increment the value by 1.This essentially lets us create a 'seen' list.

next

This command tells awk not to process any further commands and to read in the next record and start over. We do this because file1 is only meant to set the associative array

!(a[$2,$3])

This line only executes when FNR==NR is false, i.e. we are not parsing file1 and thus must be parsing file2. We then use the first field $1 and second field $2 of file2 as the key to index into our 'seen' list created earlier. If the value returned is 0 it means we didn't see it in file1 and therefore we should print this line. Conversely, if the value is non-zero then we did see it in file1 and thus we should not print its value. Note that !(a[$2,$3]) is equivalent to !(a[$2,$3]){print} because the default action when one is not given is to print the entire line.

1
  • I see two problems with this one. op said that the files are in GB. Since each line looks to be a few bytes, there would be too many lines Creating an in memory associative array could be heavy. Also multiple lines with same $2 and different $3 would be merged in the first file, but not in second one
    – amisax
    Feb 21, 2017 at 6:20
-1
$ echo -n 'difference in file2 is for line number ';diff --unchanged-line-format="" --old-line-format="" --new-line-format="%dn " <(tail +2 file1|unexpand -a|cut -f2-) <(tail +2 file2|unexpand -a|cut -f2-)|grep -o "[0-9]*" | while read i;do echo $((i+1));done|paste -sd,
difference in file2 is for line number 3,4

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.