I have the following script to process a file with some data: First, the header is printed to the output file. Then 60000 lines from the input are randomly picked and printed to the output (the possibility of printing the same line several times is explicitly wanted).

N = 60000
gawk '{if (NR < 37) print $0}' input > output
MAX=$(gawk 'END{print NR}' input)

for ((i=1; i<=$N; i++ ))

   declare $(gawk -v min=37 -v max=$MAX -v seed=$RANDOM 'BEGIN{srand(seed); print "x="int(min+rand()*(max-min+1))}')
   gawk -v l=$x 'NR>36 && NR==l {print $0}' input >> output


I now that this is very inefficient, so I am open to any idea how to improve the performance of this code, maybe it is possible to prevent opening and closing the input file all the time?

Thank you for your time!


You want to first extract a 36 line header from the file called input, then pick 60000 random lines from the rest of the file with the possibility of randomly picking the same line multiple times. All output should go to the file named output.

Using shuf from GNU coreutils:


# Fetch header (36 first lines)
head -n 36 <input >output

# Scramble the other lines and pick 60000 (allowing for repeated lines)
tail -n +37 <input | shuf -r -n 60000 >>output


( head -n 36 <input; tail -n +37 <input | shuf -r -n 60000 ) >output

With GNU head, which leaves the input file stream at the position after the last line outputted, which means shuf can continue where head finished reading (this may not work with some non-GNU head implementations):

( head -n 36; shuf -r -n 60000 ) <input >output
  • That should work with any POSIX compliant head (some have been known not to be though). – Stéphane Chazelas May 7 '18 at 13:52
  • @StéphaneChazelas I know. I have a non-POSIX head (in this regard) on my OpenBSD system. Which is why I was careful to say that I knew it worked with at least GNU head. – Kusalananda May 7 '18 at 13:53
  • 1
    Yes, getc() uses stdio which reads by whole bufs. IIRC, GNU stdio does seek back for unconsumed buffers on exit, though don't quote me on that, as it's vague memory. In any case, GNU head does seek back at least on GNU systems as observed with strace(). – Stéphane Chazelas May 7 '18 at 14:04
  • 1
    @TankorSmash On my system, the shuf solution takes about 0.15 seconds. Spawning awk 120000 times, which is what the original code does, but without actually doing anything (awk 1 /dev/null) takes about 10 minutes on my system. – Kusalananda May 8 '18 at 4:23
  • 1
    @quadrupoltensor shuf can take "random bytes" from a file with --random-source=FILE. Giving it the same file as a source of random bytes should make it generate the same random sequence every time. – Kusalananda May 8 '18 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.