compare two files ignoring 1st column and print line number

Question

how can I print difference line number ignoring 1st column for tab separeated files?

example- compare File 1 vs File2 by ignoring 1st column and print line no. for differnce record present in file 2.

In file1:

user1   fistname    Lastnamename
user2   Johnny            Depp
user3   Tom               Cruise
user4   Leonardo          DiCaprio
user5   Sylvester         Stallone

and

In file2:

user10   fistname    Lastnamename
user2   Johnny            Depp
user30  Tom'               Cruise
user4   Nicolas           Cage
user50  Sylvester         Stallone

expected result:- difference in file2 is for line number 3,4

Note file size to be compare is in GB and File is tab separated

Answer 1

You can use diff command along with cut to find the difference in the two file.

diff <(cut -f2 -d$'\t' file1) <(cut -f2 -d$'\t' file2)

Output will be

3,4c3,4
< Tom
< Leonardo
---
> Tom'
> Nicolas

If you concern about the more duplicate records, then you can use above command with sort -u for removing duplicate before finding diff from the other file. Command will be

diff <(cut -f2 -d$'\t' file1|sort -u) <(cut -f2 -d$'\t' file2|sort -u)

Answer 2

Try with this code, don't know if this will work as I don't have enough data:

diff --unchanged-line-format="" --old-line-format="" --new-line-format=":%dn: %L" file1 file2 | sed 1d | cut -d':' -f2 |tr '\n' ','|sed 's/,$//g'

Answer 3

This could answer your question:

awk 'NR==FNR{++a[$2,$3];next} {line++;if(!(a[$2,$3])){print line}}' record1 record2

Explanation:

FNR==NR

When you have two (or more) input files to awk, FNR will reset back to 1 on the first line of the next file whereas NR will continuing incrementing from where it left off. By checking FNR==NR we are essentially checking to see if we are currently parsing the first file.

++a[$2,$3]

If we are parsing the first file (see above) then create an associative array with the first field $2 and second field $3 as the key and post increment the value by 1.This essentially lets us create a 'seen' list.

next

This command tells awk not to process any further commands and to read in the next record and start over. We do this because file1 is only meant to set the associative array

!(a[$2,$3])

This line only executes when FNR==NR is false, i.e. we are not parsing file1 and thus must be parsing file2. We then use the first field $1 and second field $2 of file2 as the key to index into our 'seen' list created earlier. If the value returned is 0 it means we didn't see it in file1 and therefore we should print this line. Conversely, if the value is non-zero then we did see it in file1 and thus we should not print its value. Note that !(a[$2,$3]) is equivalent to !(a[$2,$3]){print} because the default action when one is not given is to print the entire line.

Answer 4

$ echo -n 'difference in file2 is for line number ';diff --unchanged-line-format="" --old-line-format="" --new-line-format="%dn " <(tail +2 file1|unexpand -a|cut -f2-) <(tail +2 file2|unexpand -a|cut -f2-)|grep -o "[0-9]*" | while read i;do echo $((i+1));done|paste -sd,
difference in file2 is for line number 3,4