Fastest way to write bash variable to a file

Question

I want to generate some big files for some experiment.

This is my script. It creates a file then reads it to a variable and then tries to write it as many times defined in the loop in files:

#! /usr/bin/env bash

set -e
set -u

< /dev/urandom tr -dc "\t\n [:alnum:]" | head -c32768 > temp.txt
data=$(cat ./temp.txt)

for testdir in "$@"; do
    echo "create directory '$testdir'"
    mkdir -p $testdir
    for i in {1..3}; do
        counter=$(printf %02d $i)
        testfile=$testdir/test_${testdir##*/}_$counter.txt
        echo "create file '$testfile'"
        echo "$data" > $testfile
    done
done

If I try to use this script to create 3000 file (each folder will have 3 file) it takes around 19s on my system:

time generateUserData.sh TEST{0..1000}

create directory 'TEST999'
create file 'TEST999/test_TEST999_01.txt'
create file 'TEST999/test_TEST999_02.txt'
create file 'TEST999/test_TEST999_03.txt'
create directory 'TEST1000'
create file 'TEST1000/test_TEST1000_01.txt'
create file 'TEST1000/test_TEST1000_02.txt'
create file 'TEST1000/test_TEST1000_03.txt'

real    0m19.333s
user    0m14.791s
sys     0m4.784s

I recon echo can be the slow part here. Any ideas how can I make his as fast as possible?

Answer 1

What would be slow would rather be forking processes and executing external commands such as mkdir

counter=$(printf %02d $i)

Also forks a process in bash. That can be avoided by writing it instead as:

printf -v counter %02d "$i"

Or:

printf -v testfile %s/%s_%02d.txt "$testdir" "${testdir##*/}" "$i"

Create all the directory with one mkdir invocation (mkdir -p -- "$@"; don't forget the --) instead of running one mkdir per file.

Also no need for a temp file:

data=$(< /dev/urandom tr -dc "\t\n [:alnum:]" | head -c32768; echo .)
data=${data%.}

The adding of . is necessary if you want $data to be guaranteed to contain 32768 bytes as command substitution removes all trailing newline characters. Also note that echo without -n adds one back. printf should be used instead of echo anyway for arbitrary data:

Also beware the head -c 32768 gives you 32768 bytes, not characters so could cut characters in the middle.

printf %s "$data" > "$file"

Answer 2

Taking on some of what Stéphane Chazelas said in their fine answer, with some tweaks.

#!/usr/bin/env bash

set -e
set -u

main() {
  < /dev/urandom tr -dc "\t\n [:alnum:]" | dd iflag=fullblock of=./temp.txt bs=32K count=1
  mkdir -p -- "${@:?}"
  for testdir in "$@"; do
    for i in {1..3}; do
      printf "%s/%s_%02d.txt\n" "$testdir" "${testdir##*/}" "$i"
    done
  done | xargs -n1 -P${proc:-16} cp ./temp.txt
}

time main "${@}"

• dd - alternate way to get the exact number of bytes (though as this only happens once, it's not going to make much difference either way)
• all those echo added about 3 seconds on my end with a 1000 count
• multithreaded, tuneable at runtime (via proc variable) -- play around to find the optimum value for your system

e.g.

proc=32 bash ./foo.sh {1..1000}

NB -- assuming that your original question about filling the variable was an instance of an XY problem ... if that is a hard requirement, my answer is not valid as written.

This change should do it though:

data="$(< /dev/urandom tr -dc "\t\n [:alnum:]" | dd iflag=fullblock bs=32K count=1)"
dd iflag=fullblock bs=32K count=1 of=./temp.txt <<<"${data}"

The second dd is to make sure we get only the generated data, without it, somewhere between $( ) and <<< we seem to find an extra byte from somewhere (implied newline?). Happy to take corrections on this. Looks a bit cumbersome I admit, if the size of the random data is arbitrary or otherwise unimportant, you can simplify this I'm sure