Reading values from a file and printing those lines present in another file

Question

I have two different files: file1 with 1 million entries and file 2 with 1.3 millions entries. My file 1 looks like this

A  54 54 67 abcd
C 54 4 1.2 lmno
D 43 2 22.2 asdasd

File2 looks similarly

A 0 1.1 1 qr
D 2 1 1 rerlm

However file1 has more entries than file2. Not all column 1 values are present in file2. All other columns are different.

I need to print all such lines file1 that match the first column in file2. In a nutshell print all lines in file1 and store it in file3 based on whether or not the first column entry is present in file2.

Answer 1

A simple solution is to use the join command to accomplish this.

This requires that both files are sorted, since join only works in those cases... You can use sort to sort the files if they're not sorted.

It also assumes the number of fields in the first file (the one you want to print) is fixed, since you need to specify the output fields one by one.

This solution also assumes you're OK with changing the whitespace between fields in the solution. And that you're OK with the end result being sorted (and not necessarily preserving the order of file1), if file1 was not sorted to start with...

So, assuming you're OK with these constraints, all you need to do is:

$ join -o 1.1,1.2,1.3,1.4,1.5 file1 file2
A 54 54 67 abcd
D 43 2 22.2 asdasd

If files are not sorted, a simple approach is to use bash process substitution to sort them on-the-fly and feed them to join:

$ join -o 1.1,1.2,1.3,1.4,1.5 <(sort file1) <(sort file2)

(Though if the files are large and have millions of lines, you might want to store the sorted results, so you don't have to sort again.)

A small note that this depends on your locale (which defines sort order), you might also want to pass sort the -b option to skip leading whitespace if you have that in your files... The man page of join also suggests using sort -k 1b,1 and also discusses using LC_COLLATE to control order in sort and join... Hopefully this is a good start to resolve this question!