Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have 2 files. If field 1, 2, 4 and 5 of both file1 and file2 matches, then I want to print the whole line of file1 and file2 one after another in my output file.


sc2/80         20      .        A       T         86   F=5;U=4
sc2/60         55      .        G       T         76   F=5;U=4 
sc2/68         20      .        T       C         71   F=5;U=4
sc2/24         24      .        T       G         31   F=5;U=4


sc2/99         84      .        C       G         61   F=5;U=4
sc2/80         20      .        A       T         30   F=5;U=4
sc2/60         40      .        G       T         76   F=5;U=4 
sc2/30         20      .        T       C         71   F=5;U=4
sc2/24         24      .        T       G         91   F=5;U=4

Expected Output:

sc2/80         20      .        A       T         86       F=5;U=4
sc2/80         20      .        A       T         30       F=5;U=4
sc2/24         24      .        T       G         31       F=5;U=4
sc2/24         24      .        T       G         91       F=5;U=4

I am new in the field and I appreciate your help.

share|improve this question
up vote 1 down vote accepted

You can use a multidimensional array:

awk 'FNR==NR{a[$1,$2,$4,$5]=$0;next}{if(b=a[$1,$2,$4,$5]){print b;print}}' file1 file2

FNR (file number of record) is equal to NR when awk is processing the first file.

a[$1,$2]=$0 is the same as a[$1 SUBSEP $2]=$0 or a[$1"\034"$2]=$0, and ($1,$2)in a is the same as ($1 SUBSEP $2)in a or ($1"\034"$2)in a.

You could also replace if(b=a[$1,$2,$4,$5]){print b;print} with if(($1,$2,$4,$5)in a){print a[$1,$2,$4,$5];print}. If !("index" in a), a["index"] is like a["index"]="".

share|improve this answer
It works. Thanks :) – Namrata Aug 27 '13 at 10:50
As I understand, in the second file you check if(b=a[$1,$2,$4,$5]) but where is b defined? Is that an implicit definition? – mike Aug 27 '13 at 11:59
= is an assignment, not a comparison. – user495470 Aug 27 '13 at 12:12
Okay, then I don't understand it. Could you please elaborate on the control flow? – mike Aug 27 '13 at 16:12
If a has an index like $1"\034"$2"\034"$4"\034"$5, a[$1,$2,$4,$5] returns the value of the element at that index. Otherwise a[$1,$2,$4,$5] adds a new element at that index whose value is an empty string, and b is assigned to an empty string, which evaluates to false. – user495470 Aug 28 '13 at 4:16

I know you said you didn't want a Perl or Python solution but it might be useful to someone else (and you really really should learn one of those languages if you are doing bioinformatics).

perl -ane '$f=$F[0].$F[1]; print "$k{$f}$_" if $k{$f}; $k{$f}=$_;' file1 file2 


The -a option will cause Perl to split input into the @F array, -n means read input files line by line and -e means "run the script I give on the command line".

So, $f is set to the concatenation of the first ($F[0]) and second ($F[1]) fields. $k{$f}=$_ means save the current line ($_) as the value in a hash (associative arrays in Perl) called k with the key $f. As we read through the files, print the current line and the value of $k{$f} if that value exists. In other words, if we have already seen a line that has the same two first fields, print that line and the current one.

share|improve this answer
Thanks. Yes, I started with Python. – Namrata Aug 27 '13 at 14:30

You want to print the intersection of the two files without reordering them (so it's not a set intersection)? I would look up string similarity algorithms and treat each line as a letter. You'll need to modify the algorithm to keep track which letters (lines) are the same and which differ. The big problem is that order is significant but position is not. Also, you may find it easier to massage the data by rewriting it to drop the fields you don't care about. (Or to write a comparison function that ignores these fields.)

Have you considered python or perl? I've heard they're popular in the bioinformatics field. And this really seems like a programming task.

share|improve this answer
True, I need the intersection of two files based on 4 files. Even if the other fields doesnot match, I don't care. I would prefer Awk and unix. – Namrata Aug 27 '13 at 10:19

If you can guarantee that each file is supposed to have unique entries. Sort the files with sort -u concatenate the files. Sort again without the -u and then search for duplicate entries.

I would write the little script, but I can't do it out of my head. But it shouldn't be to hard given my approach.

I have a console in front of me now. Here you go:

rm -rf all; sort -u file1 > all; sort -u file2 >> all
sort all | uniq --all-repeated=separate -w 32

If file1 file2 are already sorted and do not contain duplicate entries you can use this command:

sort -m file1 file2 | uniq --all-repeated=separate -w 32

Oh, looks like this is not quite what you asked, since I compare whole lines. Maybe someone else will find this useful though.

share|improve this answer
I want to match only 4 fields in both the files. If they match they are redirected to output. About other fields I dont care. They can be different. – Namrata Aug 27 '13 at 14:27

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.