Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have 2 set of files.

File one contains ID's ex:

1111
2222
6666
3333
4444

File two contains ID and username:

1873 Neil
1111 Roger
7632 Tim
3333 Oscar
8723 Greg
4444 Roy
6666 Patrick

I want to extract the ID and username, but only the ones that has the same ID as in file 1. I did the normal grep -f file1 file2, on two test files I made, with few Id's such as the ones i just posted. However, when i apply this to the two proper files, where file1 contains 3500 ID's and File2 contains 12000 ID's + Username, instead of extracting the 3500 lines that occur in both files, it extracts 12000 lines. However, with the 2 test files, and a few dummy ID's it will only extract the correct ids and leave the others.

Any tip on what's wrong?

share|improve this question
2  
Perhaps empty lines (just spaces) in the id file, that match any line in the user file? –  vonbrand Feb 14 '13 at 13:16
add comment

2 Answers

up vote 4 down vote accepted

Try doing this using instead of grep, this will be more suitable :

$ join  <(sort file1) <(sort file2)

1111 Roger
3333 Oscar
4444 Roy
6666 Patrick

If your shell lack process substitutions <( ), you can do :

sort file1 > new_file1
sort file2 > new_file2
join new_file1 new_file2

Doc said :

join writes to standard output a line for each pair of input lines that have identical join fields.

See http://www.gnu.org/software/coreutils/manual/html_node/join-invocation.html

Notes :

The file need to be sorted for join to work properly, that's why we uses some file descriptors in the background using process substitutions See http://mywiki.wooledge.org/ProcessSubstitution , or http://mywiki.wooledge.org/BashFAQ/024 for a common use.

share|improve this answer
    
First one worked like a charm! Can you explain quickly how the join function operates on these two files? –  Arne Feb 14 '13 at 12:48
    
It's not a function but an executable, see gnu.org/software/coreutils/manual/html_node/… –  sputnick Feb 14 '13 at 12:56
    
See my edited post. –  sputnick Feb 14 '13 at 12:59
    
Thanks allot sputnick! –  Arne Feb 14 '13 at 13:26
1  
Probably not a problem in this case, but it's safer to sort on the join field, even when that field is the first, because sort (depending on the locale) will consider extra space before the field, while join will ignore it and possibly report an "unsorted input error". join <(sort -k1,1 file1) <(sort -k1,1 file2). –  Stephane Chazelas Feb 14 '13 at 13:59
show 1 more comment

grep matches lines where a username is the same as an id. join correctly limits the match to the first field, but requires sorted input. Depending on the input awk might be a good alternative:

awk 'FNR == NR { ids[$1]++; next } ids[$1]' ids users

Or more readable:

awk 'FNR == NR { ids[$1]; next } $1 in ids' ids users

Output:

1111 Roger
3333 Oscar
4444 Roy
6666 Patrick

Explanation

The awk program is divided into two parts: one that is evaluated for the first file and one that is evaluated for the second file.

The first block is only evaluated for the first file and saves ids in the ids array. When these ids are encountered when reading users the default block is invoked ({print $0}).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.