2

I want to generate some big files for some experiment.

This is my script. It creates a file then reads it to a variable and then tries to write it as many times defined in the loop in files:

#! /usr/bin/env bash

set -e
set -u

< /dev/urandom tr -dc "\t\n [:alnum:]" | head -c32768 > temp.txt
data=$(cat ./temp.txt)

for testdir in "$@"; do
    echo "create directory '$testdir'"
    mkdir -p $testdir
    for i in {1..3}; do
        counter=$(printf %02d $i)
        testfile=$testdir/test_${testdir##*/}_$counter.txt
        echo "create file '$testfile'"
        echo "$data" > $testfile
    done
done

If I try to use this script to create 3000 file (each folder will have 3 file) it takes around 19s on my system:

time generateUserData.sh TEST{0..1000}

create directory 'TEST999'
create file 'TEST999/test_TEST999_01.txt'
create file 'TEST999/test_TEST999_02.txt'
create file 'TEST999/test_TEST999_03.txt'
create directory 'TEST1000'
create file 'TEST1000/test_TEST1000_01.txt'
create file 'TEST1000/test_TEST1000_02.txt'
create file 'TEST1000/test_TEST1000_03.txt'

real    0m19.333s
user    0m14.791s
sys     0m4.784s

I recon echo can be the slow part here. Any ideas how can I make his as fast as possible?

0

2 Answers 2

3

What would be slow would rather be forking processes and executing external commands such as mkdir

counter=$(printf %02d $i)

Also forks a process in bash. That can be avoided by writing it instead as:

printf -v counter %02d "$i"

Or:

printf -v testfile %s/%s_%02d.txt "$testdir" "${testdir##*/}" "$i"

Create all the directory with one mkdir invocation (mkdir -p -- "$@"; don't forget the --) instead of running one mkdir per file.

Also no need for a temp file:

data=$(< /dev/urandom tr -dc "\t\n [:alnum:]" | head -c32768; echo .)
data=${data%.}

The adding of . is necessary if you want $data to be guaranteed to contain 32768 bytes as command substitution removes all trailing newline characters. Also note that echo without -n adds one back. printf should be used instead of echo anyway for arbitrary data:

Also beware the head -c 32768 gives you 32768 bytes, not characters so could cut characters in the middle.

printf %s "$data" > "$file"
1

Taking on some of what Stéphane Chazelas said in their fine answer, with some tweaks.

#!/usr/bin/env bash

set -e
set -u

main() {
  < /dev/urandom tr -dc "\t\n [:alnum:]" | dd iflag=fullblock of=./temp.txt bs=32K count=1
  mkdir -p -- "${@:?}"
  for testdir in "$@"; do
    for i in {1..3}; do
      printf "%s/%s_%02d.txt\n" "$testdir" "${testdir##*/}" "$i"
    done
  done | xargs -n1 -P${proc:-16} cp ./temp.txt
}

time main "${@}"

  • dd - alternate way to get the exact number of bytes (though as this only happens once, it's not going to make much difference either way)
  • all those echo added about 3 seconds on my end with a 1000 count
  • multithreaded, tuneable at runtime (via proc variable) -- play around to find the optimum value for your system

e.g.

proc=32 bash ./foo.sh {1..1000}

NB -- assuming that your original question about filling the variable was an instance of an XY problem ... if that is a hard requirement, my answer is not valid as written.

This change should do it though:

data="$(< /dev/urandom tr -dc "\t\n [:alnum:]" | dd iflag=fullblock bs=32K count=1)"
dd iflag=fullblock bs=32K count=1 of=./temp.txt <<<"${data}"

The second dd is to make sure we get only the generated data, without it, somewhere between $( ) and <<< we seem to find an extra byte from somewhere (implied newline?). Happy to take corrections on this. Looks a bit cumbersome I admit, if the size of the random data is arbitrary or otherwise unimportant, you can simplify this I'm sure

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.