New answers tagged


This works with your sample awk ' NR==FNR {patt[$0]; next} $0 in patt {getline; getline; getline; prev=$0; next} {print prev; prev=$0} END {print prev} ' fileA.txt fileB.txt You have to keep all of fileA in memory, but you only need to remember one line at a time from fileB


Next code is not optimal (because it have to read FileB.txt twice) but hope it can be more quick then awk comm --nocheck-order -23 FileB.txt <(grep -B1 -A2 -Ff FileA.txt FileB.txt) With new GNU sed command e you can try (for memory saving) sed + grep: sed 'N;h;s/.*\n//;s/.*/grep -xF "&" FileA.txt/e;/./{N;N;d};x;P;D' FileB.txt


Note: No error checking. Also, assumption is that the input in 2nd file follows the pattern mentioned exactly. awk 'NR== FNR {a[$0] = $0 ; next } {if (!($0 in a)) {b[count++] = $0; } else {count--; if (count > 0) delete b[count];getline;getline; }} END {for (i=0; i<count; i++) print b[i] }' 1 2 inputs are in 1 and 2 1 ATGCATGC GGGGGGTT TTTTT AAAA ...


The short answer is "No" - your limiting factor on this sort of operation is disk IO. There is no way to stream 25GB of a disk any faster. You might get a minor improvement if you don't inplace edit, and you write the result of the sed to a separate drive (if you have one available) - because that way you can be reading from one, whilst writing to another ...

Top 50 recent answers are included