I have a large file with around 15 million lines. The odd lines contains the keys (but not only the keys) and the even lines are the data. That is, the file is something like:

WRTZ Tyrosine
ASDF Proline
TYSR Serine
ASDF Cytosine

The keys here are ASDF, TYSR, and WRTZ.

I have a list of around 100000 keys. I want to extract the data (both lines) corresponding to these keys.

One approach I tried was to grep the lines containing the keys to get the line numbers and then extract the line and the next line using head and tail in a loop. However, this seems to take a long time to run.

Is there a more efficient way to do this?

| improve this question | | | | |
  • Must you do this using only shell scripting, or can you write a program to do it? – Greg Hewgill Jul 2 '14 at 22:10
  • I can write a python script to do it. I was wondering if it could be done more efficiently using some shell commands. – Devil Jul 2 '14 at 22:14
  • Well, if you've got a large input file, then the key to efficiency is to read the file only once. Using grep/head/tail will need to read the file a great number of times, which will end up taking a long time. A Python program to extract this information should be short and simple. – Greg Hewgill Jul 2 '14 at 22:17
  • 3
    Does your system's version of grep support context (the -A,-B, -C flags)? If so have you tried something simple such as fgrep -f keysfile -A1 file - the A1 prints a line of context after the match. – steeldriver Jul 2 '14 at 22:28
  • @steeldriver : Thanks. Yes. That seems to work very well. – Devil Jul 2 '14 at 22:37

If there is no change in even and odd lines. Then try using below command

awk 'NR%2{printf $1"-";next;}1' <Filename>

Output of the above command will be




| improve this answer | | | | |

A small awk script can do the job. Easy to understand by reading awk man manual.

    #!/usr/bin/awk -f
    /ASDF/{printf ("%s ",$1); got=1;linenum=NR+1}
    /TYSR/{printf ("%s ",$1); got=1;linenum=NR+1}
    /WRTZ/{printf ("%s ",$1); got=1;linenum=NR+1}
    /^[0-9]/{if ( ( got == 1 ) && ( linenum == NR) ) {

The output will be liek this:

    ./awk_script data_file

    WRTZ 1287998798
    ASDF 9408654860
    TYSR 9809804090
    ASDF 4950409808

You can modify it to satisfy your own need!

| improve this answer | | | | |
  • The user has a list of 100000 keys, not just the ones shown in the question. – Kusalananda Apr 8 '19 at 6:54
  • Oh, and there are 15 million lines. I think to write a C program with the same logic is better. – JinChin Apr 8 '19 at 7:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.