For a really big file like 1GB
wc -l happens to be slow. Do we have a faster way calculating the number of newlines for a particular file?
You can try to write in C:
Save in e.g.,
This finds newlines sprinkled in a 1GB file on my system in about 370ms (repeated runs).
(Increasing buffer sizes slightly increases the time, which is to be expected -- BUFSIZ should be close to optimal).
This is very comparable to the ~380ms I'm getting from
Mmaping gives me a better time of about 280ms, but it of course has the limitation of being limited to real files (no FIFOS, no terminal input, etc.):
I created my test file with:
and added some test newlines with:
and a hex editor.
You can improve on the solution suggested by @pskocik by reducing the number of calls to
When benchmarking the various approaches, you might keep in mind that some systems (such as Linux) use most of your machine's unused memory as a disk cache. A while back (almost 20 years ago, mentioned in the vile FAQ), I was puzzled by unexpectedly good results from a (not very good) paging algorithm which I had developed to handle low-memory conditions in a text editor. It was explained to me that it ran fast because the program was working from the memory buffers used to read the file, and that only if the file were re-read or written would there be a difference in speed.
The same applies to