0

For simulation I am trying to do, I want a text file with numbers ranging from 0 to 2^33 which is a huge number. I have used this command:

seq 0 Number >> OUTPUT FILE

But this is very slow. The file is nearly 94 GB, so we can't use shuf.Then I have used terashuf by Alexandres which is also taking quite a lot of time. Even though I have done what I wanted to do, I wanted to know if there is a faster way to do this in a single command and whether there is any way in which we can truly randomize the order of these numbers

NOTE: Even though I have been using Linux from quite a log time, I have very limited knowledge on bash scripting. So please try to give answers which a beginner can understand.

7
  • General hint: Shell scripting is never about performance... – Hauke Laging May 2 '20 at 13:35
  • @HaukeLaging Thanks for the hint. But I guess there should be some way which does it in the fastest way possible. – Uday May 2 '20 at 13:44
  • 1
    I am wondering why. This could be an XY problem. I can not see why you would want a file full of numbers. If you show us what you will use it for, then it may help to come up with a better answer. – ctrl-alt-delor May 2 '20 at 14:37
  • @ctrl-alt-delor I am using this file as I need boolean representations of all these numbers. The thing is I need nearly 2^25 to 2^27 numbers out of 0 to 2^33 numbers. So I thought maybe writing all these numbers to file and shuffling them would do the job as I could just read each line serially. This is what I wanted to do – Uday May 2 '20 at 16:26
  • 1
    There are 32-bit pseudorandom number generators with period 2^32-1 (see schneier.com/academic/archives/1994/09/…). There may be generators for 33-bit values with period 2^33-1. Or you can extend each element of the 32-bit PNRG stream by adding a bit to it, alternating between 0 and 1. – Mark Plotnick May 3 '20 at 8:44
0

The best solution in this case is probably to modify the source code of shuf or a similar program.

Then you could throw away the input part, use just the shuffling part and write out the indexes. It may be useful to not convert the indexes to strings but write them binary without line breaks to a file because the file will be so much smaller (and thus faster to read).

4
  • Could you please elaborate on how exactly to do that – Uday May 2 '20 at 14:12
  • If you are going to do than, then just write a C program. It would be tiny. – ctrl-alt-delor May 2 '20 at 14:36
  • @ctrl-alt-delor I assume shuf to be a C program. It is already there sp why write a new one? – Hauke Laging May 2 '20 at 15:44
  • @Uday Hardly. I would make sense to make that a question on stackoverflow. – Hauke Laging May 2 '20 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.