Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have two files as follows File 1 has list of Ids


File 2 has

id=ram;*between*hkjhdswklf     kjsdklhglshglkhgklsdgkhdfg
id=taikah;jhdjfkhg**kjdfkjksfdjk dfkjskdfjkljdflkjsdflj

Basically i want to match Ids from file 1 to file 2 and paste the complete line of file 2

Help would be appreciated.

share|improve this question

Awk was designed for this (here with GNU awk for the capturing of brace matched patterns into an array in match()):

$ awk '
    NR == FNR { Ids[$1]++; next; }
    match($1, /id=([^;]+);/, Id) && Id[1] in Ids
' /file1 /file2
id=ram;*between*hkjhdswklf     kjsdklhglshglkhgklsdgkhdfg
id=taikah;jhdjfkhg**kjdfkjksfdjk dfkjskdfjkljdflkjsdflj

Several of the alternate responses propose solutions which make various assumptions about the input which are not exactly evident in the sample given by OP. One thing that is clear, though, is that there is a legitimate word in the data following id=xxx;, so it's not safe to assume that one of the ids from File 1 might not be somewhere in the data as well.

This is why Awk is much better than fgrep(1). Awk will break up the input into fields which can limit the possible match, which has been done in the example above by specifying that the match must be in the first field ($1).

The example input contains some data right next to the id, so if you wanted an absolute bullet-proof way of matching the ids in File 1, you could define the field separator (FS) as ; and only print if there is an exact match.

Assume File 2 contains:

id=ram;*between*hkjhdswklf     kjsdklhglshglkhgklsdgkhdfg
id=taikah;jhdjfkhg**kjdfkjksfdjk dfkjskdfjkljdflkjsdflj
id=notinlist;jhdjfkhg**kjdfkjksfdjk ram*dfkjskdfjkljdflkjsdflj
id=notinlist;*between*taikah*hkjhdswklf     kjsdklhglshglkhgklsdgkhdfg

Then the following will still work

$ awk -v FS=";" '
    NR == FNR {Id = "id=" $0; Ids[Id]++; next; }
    $1 in Ids
' /file1 /file2
id=ram;*between*hkjhdswklf     kjsdklhglshglkhgklsdgkhdfg
id=taikah;jhdjfkhg**kjdfkjksfdjk dfkjskdfjkljdflkjsdflj

Note that the other Awk solution will be very slow for large files since it is O(N2). The sed version will work, but personally, I find sed to be unreadable for mere mortals.

share|improve this answer
May not work if there is more than one id=xxx; per line. – Stéphane Chazelas Mar 15 '14 at 10:15
There are probably dozens of cases where it might not work, but those cases are not specified in the question nor in the sample input. So absent more than one id=xxx; in the first field, then it will work as intended with the sample provided. It's the poster's responsibility to provide representative input. – myq Mar 15 '14 at 10:23
That was not intended as a negative critique on your answer, I was the 1st to upvote your answer and it most certainly fits the OP's need. Just a comment indicating a limitation for anyone coming here with a similar need (which is what SE is about) – Stéphane Chazelas Mar 15 '14 at 10:56

That is exactly what fgrep(1) was created for... or more modernly (may use GNU exensions):

grep -Fwf file1 file2

From man grep:

   -F, --fixed-strings
          Interpret PATTERN as a  list  of  fixed  strings,  separated  by
          newlines,  any  of  which is to be matched.  (-F is specified by
   -f FILE, --file=FILE
          Obtain  patterns  from  FILE,  one  per  line.   The  empty file
          contains zero patterns, and therefore matches nothing.   (-f  is
          specified by POSIX.)
   -w, --word-regexp
          Select  only  those  lines  containing  matches  that form whole
          words.  The test is that the matching substring must  either  be
          at  the  beginning  of  the  line,  or  preceded  by  a non-word
          constituent character.  Similarly, it must be either at the  end
          of  the  line  or  followed by a non-word constituent character.
          Word-constituent  characters  are  letters,  digits,   and   the
share|improve this answer
This. We were all wrong. – mikeserv Mar 15 '14 at 18:47
Not quite: this will also match e.g. id=baram if file1 contains tam. See Stephane's answer for how do handle this. – Gilles Mar 15 '14 at 23:24
@Gilles not with grep -Fwf file1 file2. – terdon Mar 16 '14 at 5:02

Here is one way to do it:

awk 'FNR==NR{ids[$0]=$0;next}{for(id in ids){if($0 ~ id){print}}}' file1 file2
share|improve this answer

Another convoluted sed one:

echo | sed -e '1,/^$/{H;d;}
               s/\n.*//' file1 - file2

A POSIX version of myq's, with the same limitation:

awk '
  NR == FNR { Ids[$1]++; next; }
  match($0, /id=[^;]+/) && substr($0, RSTART+3, RLENGTH-3) in Ids
' file1 file2

An approach using grep:

sed 's/.*/id=&;/' file1 | grep -Ff - file2
share|improve this answer
    sed -rfn - <<-HD ./file2
        $(sed -r 's|.*|s/id=&/\&/p|' ./file1)

This works. It first reads only file1 in and transforms each of its lines into a sed regular expression for file2, writing each of those lines to its stdout which happens to be the subshell run in the here-document assigned to the second sed's file-descriptor <<-HD 0. So sed invocation the second pulls in sed the first's output as its regex pattern from - stdin. Its script looks like this:


The above is the script the second sed invocation runs. Because every line in file1 is transformed into a literal id=* match for file2 without relying on any reference address before the i in id or after the following ;semicolon; I don't think it could miss anything, and should (hopefully now) be very easy to alter to handle any variations on the original need as necessary.

And when sed two is through, the output looks like this:

    id=ram;*between*hkjhdswklf     kjsdklhglshglkhgklsdgkhdfg
    id=taikah;jhdjfkhg**kjdfkjksfdjk dfkjskdfjkljdflkjsdflj

P.S. If you want a good laugh, have a look at the edit history on this answer - my path to this solution was more than just a little convoluted - it was a trainwreck.

share|improve this answer
@Stephane Chazelas : I know - had to rewrite the whole command. It was awful. It's better now though. I did have some experience with 'R' just the other day - but it was different in that the lines were literally 1 for 1 collated. Also, I think it was eating my |pipe anyway, so it was easier to do the processing through twice - still streamed - but end to end to end again, if you take my meaning. And that's why the working command just pulls them in together and deletes the first file's lines from the pattern space after writing them out to the pattern file. – mikeserv Mar 15 '14 at 10:53

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.