0

Need help on shell script, I have 2 large files around 1.2 GB data, with key and values, I need to compare both files based on the key and store difference in the value in the third file and the unique one in file 1,

file 1:

test1 marco;polo;angus
test2 mike;zen;liza
test3 tom;harry;alan
test4 bob;june;janet

file 2:

test1 polo;angus
test2 mike
test4 bob;janet

I would like to compare the first two columns of file1 with file2 (search through the entire contents of file2 in first two columns) if they match print the difference of values. Then search for the second line of file 1 and so on. Also keys unique in file 1 should be printed.

Expected Output:

test1 marco
test2 zen;liza
test3 tom;harry;alan
test4 june

The files I have are huge, containing about 100,000 lines, so I would like to make the execution fast. This is running in shell script, using #!/usr/bin/env bash. Eg:

1332239_44557576_CONTI Lased & Micro kjd $353.50_30062020_lsdf3_no-rule 343323H;343434311H;454656556H;343343432H 

It is simple Text file, with this as key (1332239_44557576_CONTI Lased & Micro kjd $353.50_30062020_lsdf3_no-rule) and these as values : (343323H;343434311H;454656556H;343343432H)

File 2 will always be a subset of file 1, just need to find values(against key) which are not present in file 2 and unique ones in File 1.

3
  • 1
    I removed the shell tags you had added since this is not something you should ever do in the shell. – terdon Apr 16 '20 at 15:26
  • How are columns defined? Is the first column of your file1 test1 marco? Or is it test1 and marco is the second column? – terdon Apr 16 '20 at 15:43
  • How would you present the difference if second column differs like mike;zen;liza versus mike;liza;tom ? – Tagwint Apr 16 '20 at 15:50
1

Using perl script :

Assumed that you are searching file2 based on file1 and not vice-versa . If you want to search file1 based on file2 also then you have to add another for loop for the file2 dictionary(hash) .
DataFiles:

$ cat file1.txt 
test1 marco;polo;angus
test2 mike;zen;liza
test3 tom;harry;alan
test4 bob;june;janet

$ cat file2.txt 
test1 polo;angus
test2 mike
test4 bob;janet

Script :

#!/usr/bin/perl

use warnings;
use strict;

my $file1=$ARGV[0];
my $file2=$ARGV[1];
my %dict1;  #Stores file1 unique key and value pairs in this dictionary ( HASH in perl )
my %dict2;  #Stores file2 unique key and value pairs in this dictionary ( HASH in perl )
my %output;     #This is output dictionary after processing all the data to print it out

open(F1,'<',$file1) or die "File not found $file1";
open(F2,'<',$file2) or die "File not found $file2";

#Store both file's contents in %dict1 and %dict2 respectively 
while(<F1>)
{
    my ($key,$value) = split(/\s+/,$_);
    $dict1{$key} = $value;
}

while(<F2>)
{
    my ($key,$value) = split(/\s+/,$_);
    $dict2{$key} = $value;
}

#Get the unique(difference) value from file2 based in the values in file1

foreach my $k ( keys %dict1 )
{
    if ( defined $dict2{$k} )
    {
        my @dict1values=split(";",$dict1{$k});
        my @dict2values=split(";",$dict2{$k});
        foreach (@dict1values)
        {
            if (   $dict2{$k} !~ /[;]*?$_[;]*?/) {

                $output{$k} .=$_.";";

             }
        }
    } else { 
        $output{$k}=$dict1{$k};
    }
}

foreach my $ke (sort(keys(%output)))
{
    print "$ke $output{$ke}\n" if ( defined($output{$ke}) );
}  

Output :

$ ./testing.pl file1.txt file2.txt 
test1 marco;
test2 zen;liza;
test3 tom;harry;alan
test4 june;
0

Here's an awk version which should be very fast.

Works on anything that follows the requested field pattern [string:key][space|";"][string][space|";"] etc.

$ cat file1;echo "";cat file2
test1 marco;polo;angus
test2 mike;zen;liza
test3 tom;harry;alan
test4 bob;june;janet

test1 polo;angus
test2 mike
test4 bob;janet            
$ awk -F '[ ;]' '
  NR==FNR{ for(i=2;i<=NF;i++){ k[$1,$i]++ } }
  NR!=FNR{ for(i=2;i<=NF;i++){ k[$1,$i]++ } }
  END{ for(i in k){
         if(k[i]==1){
           split(i,arr_i,SUBSEP); 
           k_e[arr_i[1]]=k_e[arr_i[1]]""arr_i[2]";"
         }
       }
       for(i in k_e){
         print i" "k_e[i]
       }
  }' file1 file2 | sort | sed 's/.$//'

test1 marco
test2 liza;zen
test3 harry;alan;tom
test4 june                    

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.