Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

This question already has an answer here:

If I had a plain-text file containing 8 million lines and I want to print out lines 4,000,010 to 4,000,000 to the screen, which would be more efficient: awk or sed?

There is no pattern to the text, and unfortunately, a database isn't an option. I know this isn't ideal, I'm just curious on which one would complete the task quicker.

Or maybe there is even a better alternative to sed or awk?

share|improve this question

marked as duplicate by Gilles, slm, jasonwryan, vonbrand, Patrick May 19 at 1:28

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

1 Answer 1

up vote 7 down vote accepted

Neither, use tail or head instead:

$ time tail -n 4000001 foo | head -n 11
real    0m0.039s
user    0m0.032s
sys     0m0.004s

$ time head -n 4000010 foo | tail -n 11
real    0m0.055s
user    0m0.064s
sys     0m0.036s

tail is in fact consistently faster. I ran both commands 100 times and calculated their average:

tail:

real    0.03962
user    0.02956
sys     0.01456

head:

real    0.06284
user    0.07356
sys     0.07244

I imagine tail is faster because though it has to seek all the way to line 4e10, it does not actually print anything until it gets there while head will print everything until line 4e10 + 10.


Compare to some other methods sorted in order of time:

sed:

$ time sed -n 4000000,4000011p;q foo
real    0m0.312s
user    0m0.236s
sys     0m0.072s

Perl:

$ time perl -ne 'next if $.<4000000; print; exit if $.>=4000010' foo 
real    0m1.000s
user    0m0.936s
sys     0m0.064s

awk:

$ time awk '(NR>=4000000 && NR<=4000010){print} (NR==4000010){exit}' foo 
real    0m0.955s
user    0m0.868s
sys     0m0.080s

Basically, the rule is the less you parse, the faster you are. Treating the input as a stream of data which only needs to be printed to the screen (as tail does) will always be the fastest way.

share|improve this answer
    
Great! Works visibly faster than sed. Just out of curiosity however would you know the efficiency difference between sed/awk for the same task? –  Kevin Duke Oct 9 '13 at 20:30
    
@KevinDuke make sure you use my updated version, there was a mistake in the first one. I will post some tests with other tools in a second but all of them will need to parse the lines somehow which should make them slower. –  terdon Oct 9 '13 at 20:33
1  
@KevinDuke comparisons added. –  terdon Oct 9 '13 at 20:45
3  
Is there any disadvantage to doing it the other way around? I mean: head -n 4000010 foo.txt | tail -n 10? It seems the more intuitive way to do it to me, since it doesn't require you to know the exact length of the file. It should even be faster, since AFAIK tail has to read the whole file (in order to know how many lines there are), whereas head stops reading after the appropriate number of lines. –  evilsoup Oct 9 '13 at 20:47
    
Wow! Thanks a lot for spending the time to do all these tests, it is really interesting and useful. –  Kevin Duke Oct 9 '13 at 20:49

Not the answer you're looking for? Browse other questions tagged or ask your own question.