Compare two files based on key, print difference in value in another file in Bash Shell Script

Question

Need help on shell script, I have 2 large files around 1.2 GB data, with key and values, I need to compare both files based on the key and store difference in the value in the third file and the unique one in file 1,

file 1:

test1 marco;polo;angus
test2 mike;zen;liza
test3 tom;harry;alan
test4 bob;june;janet

file 2:

test1 polo;angus
test2 mike
test4 bob;janet

I would like to compare the first two columns of file1 with file2 (search through the entire contents of file2 in first two columns) if they match print the difference of values. Then search for the second line of file 1 and so on. Also keys unique in file 1 should be printed.

Expected Output:

test1 marco
test2 zen;liza
test3 tom;harry;alan
test4 june

The files I have are huge, containing about 100,000 lines, so I would like to make the execution fast. This is running in shell script, using #!/usr/bin/env bash. Eg:

1332239_44557576_CONTI Lased & Micro kjd $353.50_30062020_lsdf3_no-rule 343323H;343434311H;454656556H;343343432H 

It is simple Text file, with this as key (1332239_44557576_CONTI Lased & Micro kjd $353.50_30062020_lsdf3_no-rule) and these as values : (343323H;343434311H;454656556H;343343432H)

File 2 will always be a subset of file 1, just need to find values(against key) which are not present in file 2 and unique ones in File 1.

Answer 1

Using perl script :

Assumed that you are searching file2 based on file1 and not vice-versa . If you want to search file1 based on file2 also then you have to add another for loop for the file2 dictionary(hash) .
DataFiles:

$ cat file1.txt 
test1 marco;polo;angus
test2 mike;zen;liza
test3 tom;harry;alan
test4 bob;june;janet

$ cat file2.txt 
test1 polo;angus
test2 mike
test4 bob;janet

Script :

#!/usr/bin/perl

use warnings;
use strict;

my $file1=$ARGV[0];
my $file2=$ARGV[1];
my %dict1;  #Stores file1 unique key and value pairs in this dictionary ( HASH in perl )
my %dict2;  #Stores file2 unique key and value pairs in this dictionary ( HASH in perl )
my %output;     #This is output dictionary after processing all the data to print it out

open(F1,'<',$file1) or die "File not found $file1";
open(F2,'<',$file2) or die "File not found $file2";

#Store both file's contents in %dict1 and %dict2 respectively 
while(<F1>)
{
    my ($key,$value) = split(/\s+/,$_);
    $dict1{$key} = $value;
}

while(<F2>)
{
    my ($key,$value) = split(/\s+/,$_);
    $dict2{$key} = $value;
}

#Get the unique(difference) value from file2 based in the values in file1

foreach my $k ( keys %dict1 )
{
    if ( defined $dict2{$k} )
    {
        my @dict1values=split(";",$dict1{$k});
        my @dict2values=split(";",$dict2{$k});
        foreach (@dict1values)
        {
            if (   $dict2{$k} !~ /[;]*?$_[;]*?/) {

                $output{$k} .=$_.";";

             }
        }
    } else { 
        $output{$k}=$dict1{$k};
    }
}

foreach my $ke (sort(keys(%output)))
{
    print "$ke $output{$ke}\n" if ( defined($output{$ke}) );
}  

Output :

$ ./testing.pl file1.txt file2.txt 
test1 marco;
test2 zen;liza;
test3 tom;harry;alan
test4 june;

Answer 2

Here's an awk version which should be very fast.

Works on anything that follows the requested field pattern [string:key][space|";"][string][space|";"] etc.

$ cat file1;echo "";cat file2
test1 marco;polo;angus
test2 mike;zen;liza
test3 tom;harry;alan
test4 bob;june;janet

test1 polo;angus
test2 mike
test4 bob;janet            

$ awk -F '[ ;]' '
  NR==FNR{ for(i=2;i<=NF;i++){ k[$1,$i]++ } }
  NR!=FNR{ for(i=2;i<=NF;i++){ k[$1,$i]++ } }
  END{ for(i in k){
         if(k[i]==1){
           split(i,arr_i,SUBSEP); 
           k_e[arr_i[1]]=k_e[arr_i[1]]""arr_i[2]";"
         }
       }
       for(i in k_e){
         print i" "k_e[i]
       }
  }' file1 file2 | sort | sed 's/.$//'

test1 marco
test2 liza;zen
test3 harry;alan;tom
test4 june