How to randomly sample a subset of a file

Question

Is there any Linux command one can use to sample subset of a file? For instance, a file contains one million lines, and we want to randomly sample only one thousand lines from that file.

For random I mean that every line gets the same probability to be chosen and none of the lines chosen are repetitive.

head and tail can pick a subset of the file but not randomly. I know I can always write a python script to do so but just wondering is there a command for this usage.

Answer 1

The shuf command (part of coreutils) can do this:

shuf -n 1000 file

And at least for now non-ancient versions (added in a commit from 2013), that will use reservoir sampling when appropriate, meaning it shouldn't run out of memory and is using a fast algorithm.

Answer 2

If you have a very large file (which is a common reason to take a sample) you will find that:

1. shuf exhausts memory
2. Using $RANDOM won't work correctly if the file exceeds 32767 lines

If you don't need "exactly" n sampled lines you can sample a ratio like this:

cat input.txt | awk 'BEGIN {srand()} !/^$/ { if (rand() <= .01) print $0}' > sample.txt

This uses constant memory, samples 1% of the file (if you know the number of lines of the file you can adjust this factor to sample a close to a limited number of lines), and works with any size of file but it will not return a precise number of lines, just a statistical ratio.

Note: The code comes from: https://stackoverflow.com/questions/692312/randomly-pick-lines-from-a-file-without-slurping-it-with-unix

Answer 3

Similar to @Txangel's probabilistic solution but approaching 100x faster.

perl -ne 'print if (rand() < .01)' huge_file.csv > sample.csv

If you need high performance, an exact sample size, and are happy to live with a sample gap at end of the file, you can do something like the following (samples 1000 lines from a 1m line file):

perl -ne 'print if (rand() < .0012)' huge_file.csv | head -1000 > sample.csv

.. or indeed chain a second sample method instead of head.

Answer 4

In case the shuf -n trick on large files runs out of memory and you still need a fixed size sample and an external utility can be installed then try sample:

$ sample -N 1000 < FILE_WITH_MILLIONS_OF_LINES 

The caveat is that the sample (1000 lines in the example) must fit into memory.

Disclaimer: I am the author of the recommended software.

Answer 5

I like using awk for this when I want to preserve a header row, and when the sample can be an approximate percentage of the file. Works for very large files:

awk 'BEGIN {srand()} !/^$/ { if (rand() <= .01 || FNR==1) print > "data-sample.txt"}' data.txt

Answer 6

Not aware of any single command which could do what you ask but here is a loop I put together which can do the job:

for i in `seq 1000`; do sed -n `echo $RANDOM % 1000000 | bc`p alargefile.txt; done > sample.txt

sed will pick up a random line on each of the 1000 passes. Possibly there are more efficient solutions.

Answer 7

You can save the follow code in a file (by example randextract.sh) and execute as:

randextract.sh file.txt

---- BEGIN FILE ----

#!/bin/sh -xv

#configuration MAX_LINES is the number of lines to extract
MAX_LINES=10

#number of lines in the file (is a limit)
NUM_LINES=`wc -l $1 | cut -d' ' -f1`

#generate a random number
#in bash the variable $RANDOM returns diferent values on each call
if [ "$RANDOM." != "$RANDOM." ]
then
    #bigger number (0 to 3276732767)
    RAND=$RANDOM$RANDOM
else
    RAND=`date +'%s'`
fi 

#The start line
START_LINE=`expr $RAND % '(' $NUM_LINES - $MAX_LINES ')'`

tail -n +$START_LINE $1 | head -n $MAX_LINES

---- END FILE ----

Answer 8

If you know the number of lines in the file (like 1e6 in your case), you can do:

awk -v n=1e6 -v p=1000 '
  BEGIN {srand()}
  rand() * n-- < p {p--; print}' < file

If not, you can always do

awk -v n="$(wc -l < file)" -v p=1000 '
  BEGIN {srand()}
  rand() * n-- < p {p--; print}' < file

That would do two passes in the file, but still avoid storing the whole file in memory.

Another advantage over GNU shuf is that it preserves the order of the lines in the file.

Note that it assumes n is the number of lines in the file. If you want to print p out of the first n lines of the file (which has potentially more lines), you'd need to stop awk at the n^th line like:

awk -v n=1e6 -v p=1000 '
  BEGIN {srand()}
  rand() * n-- < p {p--; print}
  !n {exit}' < file

Answer 9

Or like this:

LINES=$(wc -l < file)  
RANDLINE=$[ $RANDOM % $LINES ]  
tail -n $RANDLINE  < file|head -1  

From the bash man page:

        RANDOM Each  time this parameter is referenced, a random integer
              between 0 and 32767 is generated.  The sequence of random
              numbers  may  be initialized by assigning a value to RAN‐
              DOM.  If RANDOM is unset, it loses  its  special  proper‐
              ties, even if it is subsequently reset.

Answer 10

If you file size isn't huge, you can use Sort random. This takes a little longer than shuf, but it randomizes the entire data. So, you could easily just do the following to use head as you requested:

sort -R input | head -1000 > output

This would sort the file randomly and give you the first 1000 lines.

Answer 11

To get a single random block of adjacent lines, use shuf to get one random line, then use grep to get the block of lines after the randomly selected line.

$ shuf -n 1 file | grep -f - -A 10 file

This will access the file twice. The -f parameter tells grep to get the search pattern from file, in this case stdin (by using a dash as the value to the f parameter) which is the single randomly selected line from the file.

A simple function:

function random-block {
  shuf -n 1 $1 | grep -f - -A $(($2>0?$2-1:0)) $1
}

Example usage:

$ random-block /var/log/syslog 10

Indeed, this will not guarantee an output of the requested number of lines if the random selection is so far down that there aren't enough lines left when selecting the block.

An enhanced function might look like this:

function random-block {
  head -n $(($(wc -l | cut -f1 -d ' ')-$2+1)) $1 | shuf -n 1 | grep -f - -A $(($2>0?$2-1:0)) $1
}

This would grab all the lines in the file except the last n lines and then shuffle that list of lines. This would ensure that grep can always select the requested number of lines.

Answer 12

As mentioned in the accepted answer, GNU shuf supports simple random sampling (shuf -n) quite well. If sampling methods beyond those supported by shuf are needed, consider tsv-sample from eBay's TSV Utilities. It supports several additional sampling modes, including weighted random sampling, Bernoulli sampling, and distinct sampling. Performance is similar to GNU shuf (both are quite fast). Disclaimer: I am the author.

Answer 13

About around the time this question was first asked, I had written a C99-based utility called sample (https://github.com/alexpreynolds/sample) that does reservoir sampling on text files.

My sample tool will only use 8 bytes per line, which means that very large files can be sampled from with much less memory.

In contrast, the shuf and other tools mentioned above can read the entire file into system memory, which can often lead to out-of-memory errors.

Here's the help statement to summarize features:

$ sample --help
sample
  version: 1.0
  author:  Alex Reynolds

Usage: sample [--sample-size=n] [--lines-per-offset=n] [--sample-without-replacement | --sample-with-replacement] [--shuffle | --preserve-order] [--hybrid | --mmap | --cstdio] [--rng-seed=n] <newline-delimited-file>

  Performs reservoir sampling (http://dx.doi.org/10.1145/3147.3165) on very large input
  files that are delimited by newline characters. The approach used in this application
  reduces memory usage by storing a pool of byte offsets to the start of each line, instead
  of the line elements themselves.

  If the sample size (--sample-size) parameter is omitted, then the sample binary will shuffle
  the entire file.

  For text files delimited by multiples of lines, specify a --lines-per-offset value.

  Process Flags:

  --sample-size=n               | -k n    Number of samples to retrieve (n = positive integer; optional)
  --lines-per-offset=n          | -l n    Number of lines per offset (n = positive integer; optional, default=1)
  --sample-without-replacement  | -o      Sample without replacement (default)
  --sample-with-replacement     | -r      Sample with replacement (optional)
  --shuffle                     | -s      Shuffle sample written to standard output (default)
  --preserve-order              | -p      Preserve order of sample written to standard output (optional)
  --mmap                        | -m      Use memory mapping for handling input file (default)
  --cstdio                      | -c      Use C I/O routines for handling input file (optional)
  --hybrid                      | -y      Use hybrid of C I/O routines and memory mapping for handling input file (optional)
  --rng-seed=n                  | -d n    Initialize the Twister RNG with a specific seed value (n = positive integer; optional)
  --help                        | -h      Show this usage message

To install it:

$ git clone https://github.com/alexpreynolds/sample.git
$ cd sample
$ make 
$ cp sample /usr/local/bin

To use it, in the most basic case:

$ sample -k 1234 some_file.txt > some_file.sample.txt

This draws 1234 random lines from some_file.txt and writes them to some_file.sample.txt.