0

For simulation I am trying to do, I want a text file with numbers ranging from 0 to 2^33 which is a huge number. I have used this command:

seq 0 Number >> OUTPUT FILE

But this is very slow. The file is nearly 94 GB, so we can't use shuf.Then I have used terashuf by Alexandres which is also taking quite a lot of time. Even though I have done what I wanted to do, I wanted to know if there is a faster way to do this in a single command and whether there is any way in which we can truly randomize the order of these numbers

NOTE: Even though I have been using Linux from quite a log time, I have very limited knowledge on bash scripting. So please try to give answers which a beginner can understand.

7
  • General hint: Shell scripting is never about performance... May 2, 2020 at 13:35
  • @HaukeLaging Thanks for the hint. But I guess there should be some way which does it in the fastest way possible.
    – Uday
    May 2, 2020 at 13:44
  • 1
    I am wondering why. This could be an XY problem. I can not see why you would want a file full of numbers. If you show us what you will use it for, then it may help to come up with a better answer. May 2, 2020 at 14:37
  • @ctrl-alt-delor I am using this file as I need boolean representations of all these numbers. The thing is I need nearly 2^25 to 2^27 numbers out of 0 to 2^33 numbers. So I thought maybe writing all these numbers to file and shuffling them would do the job as I could just read each line serially. This is what I wanted to do
    – Uday
    May 2, 2020 at 16:26
  • 1
    There are 32-bit pseudorandom number generators with period 2^32-1 (see schneier.com/academic/archives/1994/09/…). There may be generators for 33-bit values with period 2^33-1. Or you can extend each element of the 32-bit PNRG stream by adding a bit to it, alternating between 0 and 1. May 3, 2020 at 8:44

1 Answer 1

0

The best solution in this case is probably to modify the source code of shuf or a similar program.

Then you could throw away the input part, use just the shuffling part and write out the indexes. It may be useful to not convert the indexes to strings but write them binary without line breaks to a file because the file will be so much smaller (and thus faster to read).

4
  • Could you please elaborate on how exactly to do that
    – Uday
    May 2, 2020 at 14:12
  • If you are going to do than, then just write a C program. It would be tiny. May 2, 2020 at 14:36
  • @ctrl-alt-delor I assume shuf to be a C program. It is already there sp why write a new one? May 2, 2020 at 15:44
  • @Uday Hardly. I would make sense to make that a question on stackoverflow. May 2, 2020 at 15:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .