About around the time this question was first asked, I had written a C99-based utility called
sample (https://github.com/alexpreynolds/sample) that does reservoir sampling on text files.
sample tool will only use 8 bytes per line, which means that very large files can be sampled from with much less memory.
In contrast, the
shuf and other tools mentioned above can read the entire file into system memory, which can often lead to
Here's the help statement to summarize features:
$ sample --help
author: Alex Reynolds
Usage: sample [--sample-size=n] [--lines-per-offset=n] [--sample-without-replacement | --sample-with-replacement] [--shuffle | --preserve-order] [--hybrid | --mmap | --cstdio] [--rng-seed=n] <newline-delimited-file>
Performs reservoir sampling (http://dx.doi.org/10.1145/3147.3165) on very large input
files that are delimited by newline characters. The approach used in this application
reduces memory usage by storing a pool of byte offsets to the start of each line, instead
of the line elements themselves.
If the sample size (--sample-size) parameter is omitted, then the sample binary will shuffle
the entire file.
For text files delimited by multiples of lines, specify a --lines-per-offset value.
--sample-size=n | -k n Number of samples to retrieve (n = positive integer; optional)
--lines-per-offset=n | -l n Number of lines per offset (n = positive integer; optional, default=1)
--sample-without-replacement | -o Sample without replacement (default)
--sample-with-replacement | -r Sample with replacement (optional)
--shuffle | -s Shuffle sample written to standard output (default)
--preserve-order | -p Preserve order of sample written to standard output (optional)
--mmap | -m Use memory mapping for handling input file (default)
--cstdio | -c Use C I/O routines for handling input file (optional)
--hybrid | -y Use hybrid of C I/O routines and memory mapping for handling input file (optional)
--rng-seed=n | -d n Initialize the Twister RNG with a specific seed value (n = positive integer; optional)
--help | -h Show this usage message
To install it:
$ git clone https://github.com/alexpreynolds/sample.git
$ cd sample
$ cp sample /usr/local/bin
To use it, in the most basic case:
$ sample -k 1234 some_file.txt > some_file.sample.txt
This draws 1234 random lines from
some_file.txt and writes them to