37

Is there any Linux command one can use to sample subset of a file? For instance, a file contains one million lines, and we want to randomly sample only one thousand lines from that file.

For random I mean that every line gets the same probability to be chosen and none of the lines chosen are repetitive.

head and tail can pick a subset of the file but not randomly. I know I can always write a python script to do so but just wondering is there a command for this usage.

  • lines in random order, or a random block of 1000 consecutive lines of that file? – frostschutz Jan 9 '14 at 17:49
  • Every line gets the same probability to be chosen. Don't need to be consecutive although there is a tiny probability that a consecutive block of lines be chosen together. I've updated my question to clearer about that. Thanks. – clwen Jan 9 '14 at 18:08
  • My github.com/barrycarter/bcapps/tree/master/bc-fastrand.pl does this approximately by seeking to a random location in the file and finding the nearest newlines. – barrycarter Apr 18 at 18:18

11 Answers 11

64

The shuf command (part of coreutils) can do this:

shuf -n 1000 file

And at least for now non-ancient versions (added in a commit from 2013), that will use reservoir sampling when appropriate, meaning it shouldn't run out of memory and is using a fast algorithm.

  • According to documentation, it needs a sorted file as input: gnu.org/software/coreutils/manual/… – mkc Jan 9 '14 at 19:17
  • @Ketan, doesn't seem that way – frostschutz Jan 9 '14 at 19:44
  • 2
    @Ketan it's just in the wrong section of the manual, I believe. Note that even the examples in the manual are not sorted. Note also that sort is in the same section, and it clearly doesn't require sorted input. – derobert Jan 9 '14 at 19:49
  • 2
    shuf was introduced to coreutils in version 6.0 (2006-08-15), and believe it or not, some reasonably-common systems (CentOS 6.5 in particular) don't have that version :-| – offby1 Nov 18 '14 at 19:59
  • 2
    @petrelharp shuf -n does reservoir sampling, at least when the input is greater than 8K, which is the size they determined is benchmarks better. See the source code (e.g., at github.com/coreutils/coreutils/blob/master/src/shuf.c#L46 ). Sorry for this very late answer. Apparently that's new as of 6 years ago. – derobert Nov 8 '18 at 17:44
16

If you have a very large file (which is a common reason to take a sample) you will find that:

  1. shuf exhausts memory
  2. Using $RANDOM won't work correctly if the file exceeds 32767 lines

If you don't need "exactly" n sampled lines you can sample a ratio like this:

cat input.txt | awk 'BEGIN {srand()} !/^$/ { if (rand() <= .01) print $0}' > sample.txt

This uses constant memory, samples 1% of the file (if you know the number of lines of the file you can adjust this factor to sample a close to a limited number of lines), and works with any size of file but it will not return a precise number of lines, just a statistical ratio.

Note: The code comes from: https://stackoverflow.com/questions/692312/randomly-pick-lines-from-a-file-without-slurping-it-with-unix

  • If a user wants approximately 1% of the non-blank lines, this is a pretty good answer. But if the user wants an exact number of lines (e.g., 1000 out of a 1000000-line file), this fails. As the answer you got it from says, it yields only a statistical estimate. And do you understand the answer well enough to see that it is ignoring blank lines? This might be a good idea, in practice, but undocumented features are, in general, not a good idea. – G-Man Says 'Reinstate Monica' Dec 5 '16 at 21:47
  • 1
    P.S.  Simplistic approaches using $RANDOM won’t work correctly for files larger than 32767 lines.  The statement “Using $RANDOM doesn’t reach the entire file” is a bit broad. – G-Man Says 'Reinstate Monica' Dec 5 '16 at 21:48
  • @G-Man The question seems to talk about getting 10k lines from a million as an example. None of the answers around did work for me (because of the size of the files and hardware limitations) and I propose this as a reasonable compromise. It won't get you 10k lines out of a million but it might be close enough for most practical purposes. I've clarified it a bit more following your advise. Thanks. – Txangel Dec 6 '16 at 18:32
  • This is the best answer, the lines are picked randomly while respecting the chronological order of the original file, in case this is a requirement. In addition awk is more resource friendly than shuf – Polymerase Apr 15 '18 at 18:42
  • If you need an exact number, you can always… Run this with a % greater than your need. Count the result. Remove lines matching count mod difference. – Bruno Bronosky Apr 12 at 15:17
6

Similar to @Txangel's probabilistic solution but approaching 100x faster.

perl -ne 'print if (rand() < .01)' huge_file.csv > sample.csv

If you need high performance, an exact sample size, and are happy to live with a sample gap at end of the file, you can do something like the following (samples 1000 lines from a 1m line file):

perl -ne 'print if (rand() < .0012)' huge_file.csv | head -1000 > sample.csv

.. or indeed chain a second sample method instead of head.

5

In case the shuf -n trick on large files runs out of memory and you still need a fixed size sample and an external utility can be installed then try sample:

$ sample -N 1000 < FILE_WITH_MILLIONS_OF_LINES 

The caveat is that the sample (1000 lines in the example) must fit into memory.

Disclaimer: I am the author of the recommended software.

  • 1
    For those who install it and have their /usr/local/bin before /usr/bin/ in their path, be wary that macOS comes with a built-in call-stack sampler called sample, which does something completely different, in /usr/bin/. – Denis de Bernardy Jul 5 at 8:11
2

Not aware of any single command which could do what you ask but here is a loop I put together which can do the job:

for i in `seq 1000`; do sed -n `echo $RANDOM % 1000000 | bc`p alargefile.txt; done > sample.txt

sed will pick up a random line on each of the 1000 passes. Possibly there are more efficient solutions.

  • Is it possible to get the same line multiple times in this approach? – clwen Jan 9 '14 at 18:11
  • 1
    Yes, quite possible to get the same line number more than once. Additionally, $RANDOM has a range between 0 and 32767. So, you will not get a well spread line numbers. – mkc Jan 9 '14 at 18:21
  • does not work - random is called once – Bohdan Aug 20 '14 at 5:20
2

You can save the follow code in a file (by example randextract.sh) and execute as:

randextract.sh file.txt

---- BEGIN FILE ----

#!/bin/sh -xv

#configuration MAX_LINES is the number of lines to extract
MAX_LINES=10

#number of lines in the file (is a limit)
NUM_LINES=`wc -l $1 | cut -d' ' -f1`

#generate a random number
#in bash the variable $RANDOM returns diferent values on each call
if [ "$RANDOM." != "$RANDOM." ]
then
    #bigger number (0 to 3276732767)
    RAND=$RANDOM$RANDOM
else
    RAND=`date +'%s'`
fi 

#The start line
START_LINE=`expr $RAND % '(' $NUM_LINES - $MAX_LINES ')'`

tail -n +$START_LINE $1 | head -n $MAX_LINES

---- END FILE ----

  • 3
    I'm not sure what you're trying to do here with RAND, but $RANDOM$RANDOM does not generate random numbers in the whole range “0 to 3276732767” (for example, it will generate 1000100000 but not 1000099999). – Gilles 'SO- stop being evil' Jan 9 '14 at 22:37
  • The OP says, “Every line gets the same probability to be chosen.  … there is a tiny probability that a consecutive block of lines be chosen together.” I also find this answer to be cryptic, but it looks like it is extracting a 10-line block of consecutive lines from a random starting point. That is not what the OP is asking for. – G-Man Says 'Reinstate Monica' Dec 5 '16 at 21:19
2

If you know the number of lines in the file (like 1e6 in your case), you can do:

awk -v n=1e6 -v p=1000 '
  BEGIN {srand()}
  rand() * n-- < p {p--; print}' < file

If not, you can always do

awk -v n="$(wc -l < file)" -v p=1000 '
  BEGIN {srand()}
  rand() * n-- < p {p--; print}' < file

That would do two passes in the file, but still avoid storing the whole file in memory.

Another advantage over GNU shuf is that it preserves the order of the lines in the file.

Note that it assumes n is the number of lines in the file. If you want to print p out of the first n lines of the file (which has potentially more lines), you'd need to stop awk at the nth line like:

awk -v n=1e6 -v p=1000 '
  BEGIN {srand()}
  rand() * n-- < p {p--; print}
  !n {exit}' < file
2

I like using awk for this when I want to preserve a header row, and when the sample can be an approximate percentage of the file. Works for very large files:

awk 'BEGIN {srand()} !/^$/ { if (rand() <= .01 || FNR==1) print > "data-sample.txt"}' data.txt
1

Or like this:

LINES=$(wc -l < file)  
RANDLINE=$[ $RANDOM % $LINES ]  
tail -n $RANDLINE  < file|head -1  

From the bash man page:

        RANDOM Each  time this parameter is referenced, a random integer
              between 0 and 32767 is generated.  The sequence of random
              numbers  may  be initialized by assigning a value to RAN‐
              DOM.  If RANDOM is unset, it loses  its  special  proper‐
              ties, even if it is subsequently reset.
  • This fails badly if the file has fewer than 32767 lines. – offby1 Nov 18 '14 at 20:00
  • This will output one line from the file. (I guess your idea is to execute the above commands in a loop?) If the file has more than 32767 lines, then these commands will choose only from the first 32767 lines.  Aside from possible inefficiency, I don’t see any big problem with this answer if the file has fewer than 32767 lines. – G-Man Says 'Reinstate Monica' Dec 5 '16 at 21:27
1

If you file size isn't huge, you can use Sort random. This takes a little longer than shuf, but it randomizes the entire data. So, you could easily just do the following to use head as you requested:

sort -R input | head -1000 > output

This would sort the file randomly and give you the first 1000 lines.

0

As mentioned in the accepted answer, GNU shuf supports simple random sampling (shuf -n) quite well. If sampling methods beyond those supported by shuf are needed, consider tsv-sample from eBay's TSV Utilities. It supports several additional sampling modes, including weighted random sampling, Bernoulli sampling, and distinct sampling. Performance is similar to GNU shuf (both are quite fast). Disclaimer: I am the author.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.