1

In a bash script, some integers in a range are generated, concatenated with fixed strings, and printed on a file. OS is Ubuntu 14.04 and bash version is 4.3.11(1)-release.

The strings (and so the lines) to be printed are millions. Here is the hard disk usage from iotop in several times:

 TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND  
5701 be/4 myuser    408.24 B/s   97.27 K/s  0.00 %  0.23 % bash ./script.sh
5701 be/4 myuser    408.20 B/s   97.27 K/s  0.00 %  0.00 % bash ./script.sh
5701 be/4 myuser    408.41 B/s   95.72 K/s  0.00 %  0.11 % bash ./script.sh

It takes about 16 minutes to print 1 million strings. Bandwidth of the hard disk is far away from being saturated. From uptime after more than 15 minutes of execution:

load average: 0,97, 1,14, 1,19

in a 4 core CPU. So, nor the CPU neither the hard disk seem to be responsible of this extremely slow execution.

  1. How is it possible to speed up this process?

  2. Is it possible still using bash or should I use another language?


Relevant parts of the script are the following lines:

#!/bin/bash

i=0
a='fixedstring1'
b='fixedstring2'

while [[ $i -le 9999999 ]]; do
[...]
    ur="$a""$b""$i"
[...]
    echo $ur >> exp1/file$m
[...]
    (( i++ ))
done

file$m is the m-th generated file (the script generates multiple files with a fixed number of lines instead of a unique file with all the lines).

2

Bash loop is extremly slow in your case:

> cat test-1.sh
#!/bin/sh

i=0
a='fixedstring1'
b='fixedstring2'


while [ $i -lt ${1:-9} ]; do
  ur="$a""$b""$i"
  echo $ur
  (( i++ ))
done

> time sh test-1.sh 999999 | wc -l
...
real    1m11.488s

And using jot (or seq) makes it much faster:

> cat test-3.sh
#!/bin/sh

a='fixedstring1'
b='fixedstring2'

jot -w "$a$b" ${1:-9}
# seq -f "$a$b%g" ${1:-9}
> time sh test-3.sh 999999 | wc -l
...
real    0m0.613s

And what is the logic of splitting numbers to file? If it's simple limiting number of lines, you could use split command (right now I mean GNU split due to using -d option):

sh test-3.sh | split -d -l "how much lines in one file" - expr1/file
  • Thank you! This is so much faster than the other solutions. Yes, the lines are split into files in order to limit the number of lines in each file. Since jot is not included in some systems, this is the seq alternative in this case: seq -f "$a%08.0f$b". – BowPark Nov 30 '16 at 17:27
1

You're repeatedly opening and closing the same file (maybe hundreds of times). Try to open each file at most once.

i=0
a='fixedstring1'
b='fixedstring2'
m=0

exec 3>exp1/file$m

while [[ $i -le 9999999 ]]; do
    ur="$a$b$i"
    if ... # m changes
    then
        exec 3>exp1/file$m
    fi
    echo $ur >&3
    (( i++ ))
done

This is the same advice as redirecting the whole loop instead of redirecting a specific command in the loop.

  • Aren't you forgetting to close the file descriptor (exec 3> &-)at the end of the script? – Willian Paixao Nov 25 '16 at 12:09
  • @WillianPaixao at the end of the script, where it exits? Wouldn't it be automatically closed then? O.o – muru Nov 25 '16 at 12:24
  • It works even without closing the file descriptors. Thank you. It improves performance, but the other solution is a lot faster, maybe due to the absence of the while cycle, as pointed out by Fedor. – BowPark Nov 30 '16 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.