I have a short list of 1000 gene names and a full list of 20000 gene names with ID numbers.

I have tried to set up a for loop to search for the shorter list in the full list to return the numbers.

This is the code:


LIST=$(cat ShortList.txt)

for i in ${LIST}
    RESULT=$(grep -i ${i} FullList.txt)
    echo "${RESULT}" >> Final_List_With_Numbers

The resulting file is empty- any advice on what to change?

Many thanks for the help!

Some troubleshooting I've already done:

  • checked cat Shortlist works in command line
  • checked cat FullList works in command line
  • checked for loop works to just echo "${i}"
  • checked grep "gene" FullList works in command line
  • checked genes are in FullList
  • Do consider showing a representative sample of entries from both files.
    – Kusalananda
    Commented May 12, 2020 at 17:04
  • Shortlist: AP4B1 HSPB11 CIRBP-AS1 PRKRIP1 RAB4B CCDC130 TREX1 TOR2A IZUMO4 LY6G5C CENPT TARBP2 LCAT FullList: A1BG|1 A1CF|29974 A2BP1|54715 A2LD1|87769 A2ML1|144568 A2M|2 A4GALT|53947 A4GNT|51146 AAA1|404744 AAAS|8086
    – Cait
    Commented May 12, 2020 at 17:18
  • What does the FxF option in grep do? just trying to understand
    – Cait
    Commented May 12, 2020 at 17:19
  • @Cait don't add extra information in comments please. As you can see, they don't render clearly. Instead, edit your question and add anything you need there.
    – terdon
    Commented May 12, 2020 at 18:00

1 Answer 1


First change is to not do this in a shell loop! That means you search the file once per gene name and will take much, much longer than necessary. Instead, use the -f option of grep to give it the list of names as input:

grep -iFxf ShortList.txt  FullList.txt > Final_List_With_Numbers

The options used are:

  -i, --ignore-case
          Ignore  case  distinctions in patterns and input data, 
          so that characters that differ only in case match each other.

   -F, --fixed-strings
          Interpret PATTERNS as fixed strings, not regular expressions.
   -f FILE, --file=FILE
          Obtain patterns from FILE, one per line.  If this option is 
          used multiple times or is combined with the -e (--regexp) option,
          search for all patterns given.  The empty file contains zero patterns, 
          and therefore matches nothing.
   -x, --line-regexp
          Select  only  those matches that exactly match the whole line.  
          For a regular expression pattern, this is like parenthesizing
          the pattern and then surrounding it with ^ and $.

The -x is particularly important since you don't want to find LOC12345 when you search for LOC1. However, if your FullList.txt doesn't have only the gene name on each line, then you probably want to use -w instead of -x:

   -w, --word-regexp
          Select only those lines containing matches that form whole  words.   The  test  is  that  the
          matching  substring  must  either  be at the beginning of the line, or preceded by a non-word
          constituent character.  Similarly, it must be either at the end of the line or followed by  a
          non-word  constituent  character.   Word-constituent  characters are letters, digits, and the
          underscore.  This option has no effect if -x is also specified.

Now, the code you show should actually work. It would just be very, very slow and inefficient and likely return wrong results if one of the names in Shortlist can be a substring of one of the names in FullList. If you never got any results, I am guessing that your ShortList.txt was created in Windows and has Windows-style line endings (\r\n). That means that each i in your for i in ${LIST} loop won't be geneName but instead it will be geneName\r which doesn't exist in FullList.txt so no results are found.

If you test on a *nix file, it will work as expected:

$ cat ShortList.txt 

$ cat FullList.txt 

Now, run your exact code on those examples:

$ LIST=$(cat ShortList.txt); for i in ${LIST}; do 
   RESULT=$(grep -i ${i} FullList.txt);     
   echo "${RESULT}" >> Final_List_With_Numbers;
$ cat Final_List_With_Numbers 


Of course, it also contains empty lines since when you don't find a match, $RESULT is empty, but you are still echoing it which means only an empty line will be printed. Yet another reason why using a shell loop here is a bad idea.

  • The definitely will want -w rather than -x I think since FullList.txt apparently contains "names with ID numbers" and the OP wants to "return the numbers" Commented May 12, 2020 at 18:17
  • This is all great thanks very much!
    – Cait
    Commented May 13, 2020 at 22:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .