Is there any Linux command one can use to sample subset of a file? For instance, a file contains one million lines, and we want to randomly sample only one thousand lines from that file.

For random I mean that every line gets the same probability to be chosen and none of the lines chosen are repetitive.

head and tail can pick a subset of the file but not randomly. I know I can always write a python script to do so but just wondering is there a command for this usage.

  • lines in random order, or a random block of 1000 consecutive lines of that file? Commented Jan 9, 2014 at 17:49
  • Every line gets the same probability to be chosen. Don't need to be consecutive although there is a tiny probability that a consecutive block of lines be chosen together. I've updated my question to clearer about that. Thanks.
    – clwen
    Commented Jan 9, 2014 at 18:08
  • My github.com/barrycarter/bcapps/tree/master/bc-fastrand.pl does this approximately by seeking to a random location in the file and finding the nearest newlines.
    – user2267
    Commented Apr 18, 2019 at 18:18

13 Answers 13


The shuf command (part of coreutils) can do this:

shuf -n 1000 file

And at least for now non-ancient versions (added in a commit from 2013), that will use reservoir sampling when appropriate, meaning it shouldn't run out of memory and is using a fast algorithm.

  • According to documentation, it needs a sorted file as input: gnu.org/software/coreutils/manual/…
    – Ketan
    Commented Jan 9, 2014 at 19:17
  • @Ketan, doesn't seem that way Commented Jan 9, 2014 at 19:44
  • 2
    @Ketan it's just in the wrong section of the manual, I believe. Note that even the examples in the manual are not sorted. Note also that sort is in the same section, and it clearly doesn't require sorted input.
    – derobert
    Commented Jan 9, 2014 at 19:49
  • 2
    shuf was introduced to coreutils in version 6.0 (2006-08-15), and believe it or not, some reasonably-common systems (CentOS 6.5 in particular) don't have that version :-|
    – offby1
    Commented Nov 18, 2014 at 19:59
  • 4
    @petrelharp shuf -n does reservoir sampling, at least when the input is greater than 8K, which is the size they determined is benchmarks better. See the source code (e.g., at github.com/coreutils/coreutils/blob/master/src/shuf.c#L46 ). Sorry for this very late answer. Apparently that's new as of 6 years ago.
    – derobert
    Commented Nov 8, 2018 at 17:44

If you have a very large file (which is a common reason to take a sample) you will find that:

  1. shuf exhausts memory
  2. Using $RANDOM won't work correctly if the file exceeds 32767 lines

If you don't need "exactly" n sampled lines you can sample a ratio like this:

cat input.txt | awk 'BEGIN {srand()} !/^$/ { if (rand() <= .01) print $0}' > sample.txt

This uses constant memory, samples 1% of the file (if you know the number of lines of the file you can adjust this factor to sample a close to a limited number of lines), and works with any size of file but it will not return a precise number of lines, just a statistical ratio.

Note: The code comes from: https://stackoverflow.com/questions/692312/randomly-pick-lines-from-a-file-without-slurping-it-with-unix

  • 1
    If a user wants approximately 1% of the non-blank lines, this is a pretty good answer. But if the user wants an exact number of lines (e.g., 1000 out of a 1000000-line file), this fails. As the answer you got it from says, it yields only a statistical estimate. And do you understand the answer well enough to see that it is ignoring blank lines? This might be a good idea, in practice, but undocumented features are, in general, not a good idea. Commented Dec 5, 2016 at 21:47
  • 1
    P.S.  Simplistic approaches using $RANDOM won’t work correctly for files larger than 32767 lines.  The statement “Using $RANDOM doesn’t reach the entire file” is a bit broad. Commented Dec 5, 2016 at 21:48
  • 2
    @G-Man The question seems to talk about getting 10k lines from a million as an example. None of the answers around did work for me (because of the size of the files and hardware limitations) and I propose this as a reasonable compromise. It won't get you 10k lines out of a million but it might be close enough for most practical purposes. I've clarified it a bit more following your advise. Thanks.
    – Txangel
    Commented Dec 6, 2016 at 18:32
  • 1
    This is the best answer, the lines are picked randomly while respecting the chronological order of the original file, in case this is a requirement. In addition awk is more resource friendly than shuf
    – Polymerase
    Commented Apr 15, 2018 at 18:42
  • 1
    If you need an exact number, you can always… Run this with a % greater than your need. Count the result. Remove lines matching count mod difference. Commented Apr 12, 2019 at 15:17

Similar to @Txangel's probabilistic solution but approaching 100x faster.

perl -ne 'print if (rand() < .01)' huge_file.csv > sample.csv

If you need high performance, an exact sample size, and are happy to live with a sample gap at end of the file, you can do something like the following (samples 1000 lines from a 1m line file):

perl -ne 'print if (rand() < .0012)' huge_file.csv | head -1000 > sample.csv

.. or indeed chain a second sample method instead of head.

  • Successfully applied the first method to a 10GB file which was also gzipped, so I used zcat huge_file.csv.gz | perl -ne .... Took 30 minutes or so. Thanks a lot.
    – khituras
    Commented Oct 12, 2020 at 7:52

In case the shuf -n trick on large files runs out of memory and you still need a fixed size sample and an external utility can be installed then try sample:


The caveat is that the sample (1000 lines in the example) must fit into memory.

Disclaimer: I am the author of the recommended software.

  • 2
    For those who install it and have their /usr/local/bin before /usr/bin/ in their path, be wary that macOS comes with a built-in call-stack sampler called sample, which does something completely different, in /usr/bin/. Commented Jul 5, 2019 at 8:11

I like using awk for this when I want to preserve a header row, and when the sample can be an approximate percentage of the file. Works for very large files:

awk 'BEGIN {srand()} !/^$/ { if (rand() <= .01 || FNR==1) print > "data-sample.txt"}' data.txt

Not aware of any single command which could do what you ask but here is a loop I put together which can do the job:

for i in `seq 1000`; do sed -n `echo $RANDOM % 1000000 | bc`p alargefile.txt; done > sample.txt

sed will pick up a random line on each of the 1000 passes. Possibly there are more efficient solutions.

  • Is it possible to get the same line multiple times in this approach?
    – clwen
    Commented Jan 9, 2014 at 18:11
  • 1
    Yes, quite possible to get the same line number more than once. Additionally, $RANDOM has a range between 0 and 32767. So, you will not get a well spread line numbers.
    – Ketan
    Commented Jan 9, 2014 at 18:21
  • does not work - random is called once
    – Bohdan
    Commented Aug 20, 2014 at 5:20

You can save the follow code in a file (by example randextract.sh) and execute as:

randextract.sh file.txt

---- BEGIN FILE ----

#!/bin/sh -xv

#configuration MAX_LINES is the number of lines to extract

#number of lines in the file (is a limit)
NUM_LINES=`wc -l $1 | cut -d' ' -f1`

#generate a random number
#in bash the variable $RANDOM returns diferent values on each call
if [ "$RANDOM." != "$RANDOM." ]
    #bigger number (0 to 3276732767)
    RAND=`date +'%s'`

#The start line

tail -n +$START_LINE $1 | head -n $MAX_LINES

---- END FILE ----

  • 3
    I'm not sure what you're trying to do here with RAND, but $RANDOM$RANDOM does not generate random numbers in the whole range “0 to 3276732767” (for example, it will generate 1000100000 but not 1000099999). Commented Jan 9, 2014 at 22:37
  • The OP says, “Every line gets the same probability to be chosen.  … there is a tiny probability that a consecutive block of lines be chosen together.” I also find this answer to be cryptic, but it looks like it is extracting a 10-line block of consecutive lines from a random starting point. That is not what the OP is asking for. Commented Dec 5, 2016 at 21:19

If you know the number of lines in the file (like 1e6 in your case), you can do:

awk -v n=1e6 -v p=1000 '
  BEGIN {srand()}
  rand() * n-- < p {p--; print}' < file

If not, you can always do

awk -v n="$(wc -l < file)" -v p=1000 '
  BEGIN {srand()}
  rand() * n-- < p {p--; print}' < file

That would do two passes in the file, but still avoid storing the whole file in memory.

Another advantage over GNU shuf is that it preserves the order of the lines in the file.

Note that it assumes n is the number of lines in the file. If you want to print p out of the first n lines of the file (which has potentially more lines), you'd need to stop awk at the nth line like:

awk -v n=1e6 -v p=1000 '
  BEGIN {srand()}
  rand() * n-- < p {p--; print}
  !n {exit}' < file

Or like this:

LINES=$(wc -l < file)  
tail -n $RANDLINE  < file|head -1  

From the bash man page:

        RANDOM Each  time this parameter is referenced, a random integer
              between 0 and 32767 is generated.  The sequence of random
              numbers  may  be initialized by assigning a value to RAN‐
              DOM.  If RANDOM is unset, it loses  its  special  proper‐
              ties, even if it is subsequently reset.
  • This fails badly if the file has fewer than 32767 lines.
    – offby1
    Commented Nov 18, 2014 at 20:00
  • This will output one line from the file. (I guess your idea is to execute the above commands in a loop?) If the file has more than 32767 lines, then these commands will choose only from the first 32767 lines.  Aside from possible inefficiency, I don’t see any big problem with this answer if the file has fewer than 32767 lines. Commented Dec 5, 2016 at 21:27

If you file size isn't huge, you can use Sort random. This takes a little longer than shuf, but it randomizes the entire data. So, you could easily just do the following to use head as you requested:

sort -R input | head -1000 > output

This would sort the file randomly and give you the first 1000 lines.


To get a single random block of adjacent lines, use shuf to get one random line, then use grep to get the block of lines after the randomly selected line.

$ shuf -n 1 file | grep -f - -A 10 file

This will access the file twice. The -f parameter tells grep to get the search pattern from file, in this case stdin (by using a dash as the value to the f parameter) which is the single randomly selected line from the file.

A simple function:

function random-block {
  shuf -n 1 $1 | grep -f - -A $(($2>0?$2-1:0)) $1

Example usage:

$ random-block /var/log/syslog 10

Indeed, this will not guarantee an output of the requested number of lines if the random selection is so far down that there aren't enough lines left when selecting the block.

An enhanced function might look like this:

function random-block {
  head -n $(($(wc -l | cut -f1 -d ' ')-$2+1)) $1 | shuf -n 1 | grep -f - -A $(($2>0?$2-1:0)) $1

This would grab all the lines in the file except the last n lines and then shuffle that list of lines. This would ensure that grep can always select the requested number of lines.


As mentioned in the accepted answer, GNU shuf supports simple random sampling (shuf -n) quite well. If sampling methods beyond those supported by shuf are needed, consider tsv-sample from eBay's TSV Utilities. It supports several additional sampling modes, including weighted random sampling, Bernoulli sampling, and distinct sampling. Performance is similar to GNU shuf (both are quite fast). Disclaimer: I am the author.


About around the time this question was first asked, I had written a C99-based utility called sample (https://github.com/alexpreynolds/sample) that does reservoir sampling on text files.

My sample tool will only use 8 bytes per line, which means that very large files can be sampled from with much less memory.

In contrast, the shuf and other tools mentioned above can read the entire file into system memory, which can often lead to out-of-memory errors.

Here's the help statement to summarize features:

$ sample --help
  version: 1.0
  author:  Alex Reynolds

Usage: sample [--sample-size=n] [--lines-per-offset=n] [--sample-without-replacement | --sample-with-replacement] [--shuffle | --preserve-order] [--hybrid | --mmap | --cstdio] [--rng-seed=n] <newline-delimited-file>

  Performs reservoir sampling (http://dx.doi.org/10.1145/3147.3165) on very large input
  files that are delimited by newline characters. The approach used in this application
  reduces memory usage by storing a pool of byte offsets to the start of each line, instead
  of the line elements themselves.

  If the sample size (--sample-size) parameter is omitted, then the sample binary will shuffle
  the entire file.

  For text files delimited by multiples of lines, specify a --lines-per-offset value.

  Process Flags:

  --sample-size=n               | -k n    Number of samples to retrieve (n = positive integer; optional)
  --lines-per-offset=n          | -l n    Number of lines per offset (n = positive integer; optional, default=1)
  --sample-without-replacement  | -o      Sample without replacement (default)
  --sample-with-replacement     | -r      Sample with replacement (optional)
  --shuffle                     | -s      Shuffle sample written to standard output (default)
  --preserve-order              | -p      Preserve order of sample written to standard output (optional)
  --mmap                        | -m      Use memory mapping for handling input file (default)
  --cstdio                      | -c      Use C I/O routines for handling input file (optional)
  --hybrid                      | -y      Use hybrid of C I/O routines and memory mapping for handling input file (optional)
  --rng-seed=n                  | -d n    Initialize the Twister RNG with a specific seed value (n = positive integer; optional)
  --help                        | -h      Show this usage message

To install it:

$ git clone https://github.com/alexpreynolds/sample.git
$ cd sample
$ make 
$ cp sample /usr/local/bin

To use it, in the most basic case:

$ sample -k 1234 some_file.txt > some_file.sample.txt

This draws 1234 random lines from some_file.txt and writes them to some_file.sample.txt.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .