2

I'd like to read patterns from file1 and search them iteratively in file2, printing the matched line plus 3 rows, and print them to file3.

Now I am doing:

cat file1 | while read line; do awk '/$line/ {for(i=1; i<=3; i++) {getline; print}}' file2 > file3 ; done

Is there any more efficient way to do this? My irl files are quite big (~30GB)

6
  • 1
    You could read the lines from file1 without using cat. See here for syntax.
    – clk
    Jul 9 '16 at 18:05
  • 6
    What you're describing sounds like grep -f file1 -A3 file2 > file3 Jul 9 '16 at 18:07
  • @steeldriver, sounds good. but I have to test if runtime is acceptable. usualy grep takes longer than awk on big files.
    – dovah
    Jul 9 '16 at 19:13
  • Well, if you do decide to use awk, you should be passing $line into it using something like awk -v pat="$line" and then testing it using $0 ~ pat rather than trying to use shell variable /$line/ inside the awk regex. Jul 9 '16 at 19:47
  • There is no possible way that a shell while read loop around awk (or anything, for that matter ) is going to be faster than just using grep (in your case, grep -f file1 ...). A while read loop is one of the slowest ways to read data from a file. BTW, GNU grep is extremely fast, anyway.
    – cas
    Jul 10 '16 at 3:10
2

You scan the file once for each pattern. It would be a lot faster to go through the file only once, especially since the file doesn't fit in cache.

Your sample code actually only saves the matches from the last pattern in file1. I assume that this is an oversimplification. If you need all the matches in a single file, then it's as if the patterns were joined together with the “or” operator.

awk '
    BEGIN {
        while (getline <"file1") pattern = pattern "|" $0;
        pattern = substr(pattern, 2);
    }
    match($0, pattern) {for(i=1; i<=3; i++) {getline; print}}
' file2 > file3

Note also that this doesn't print the matched line, only the three following lines. If you actually wanted to print the matched line before the three following lines, you need an additional print; before the for loop.

If you actually wanted to print the matched line before the three following lines, and your grep command supports the -A argument, then you should use this instead. A specialized program like grep is usually faster than an interpreted language like awk. As an added bonus, the command is a lot simpler.

grep -A -E -f file1 file2 >file3

If you've found that grep is slower, it might be because you've run into a defect of GNU grep in multibyte locales: it can sometimes be very slow, even when neither the file nor the pattern contains multibyte characters. If you aren't using character classes to match non-ASCII characters (e.g. you don't use [[:alpha:]] to match all Unicode letters) then run grep in a unibyte locale:

LC_ALL=C grep -A3 -E -f file1 file2 >file3

The output of grep isn't exactly the same as the awk snippet because they behave differently if the pattern is found in one of the lines being printed due to a previous line. For example, if the pattern is a and the lines are a1,a2,b,c,d then grep -A3 prints a1,a2,b,c whereas the awk snippet above prints a2,b,c and the awk snippet with the additional print prints a1,a2,b,c.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.