5

I ran a simple script to generate a large (10000000 lines) csv file with 6 fields in which some fields changed in each line/row, using a while loop. The machine had all (32) CPUs free, lot of RAM (~31 Gb) was free too.

I timed the script with the command

/usr/bin/time -v bash script.01.sh

After running for around 2 hours, I got the following stats:

Command being timed: "bash script.01.sh"
User time (seconds): 1195.14
System time (seconds): 819.71
Percent of CPU this job got: 27%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:01:10
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4976
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 3131983488
Voluntary context switches: 22593141
Involuntary context switches: 10923348
Swaps: 0
File system inputs: 0
File system outputs: 2182920
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

I want to know why did my script used only 27% of the CPU? The Disk IO was nothing much at all (saw in vmstat output). So what caused the restriction? The code in the script?

Here is the script:

#!/usr/bin/env bash
number=1

while [[ $number -lt 10000001 ]] ; do  
    fname="FirstName LastName $"
    lname=""  
    email="fname.lname.$number@domain.com"  
    password="1234567890"  
    altemail="lname.fname.$number@domain.com"  
    mobile="9876543210"      

    echo "$fname,$lname,$email,$password,$altemail,$mobile" >> /opt/list.csv
    number=$(expr $number + 1)  
done  
14

By using strace, I saw that the line

number=$(expr $number + 1)

causes a fork, path search, and exec of expr. (I'm using bash 4.2.45 on Ubuntu). That filesystem, disk, and process overhead led to bash only getting around 28% of the CPU.

When I changed that line to use only shell builtin operations

((number = number + 1))

bash used around 98% of the CPU and the script ran in a half hour. This was on a single-CPU 1.5GHz Celeron.

The script as is doesn't do anything that runs in parallel, so having 32 free CPUs won't help much. However, you can certainly parallelize it by, for instance, splitting it into 10 1-million-iteration loops that run in parallel, writing to 10 different files, and then using cat to combine them.

The following sample program was added by @Arthur2e5:

max=1000000 step=40000 tmp="$(mktemp -d)"
# Spawning. For loops make a bit more sense in a init-test-incr pattern.
for ((l = 0; l < max; l += step)); do (
    for ((n = l + 1, end = (step + l > max ? max : step + l);
      n <= end; n++)); do
        # Putting all those things into the `buf` line gives you a 1.8x speedup.
        fname="FirstName LastName \$"
        lname=""  
        email="fname.lname.$n@domain.com"  
        password="1234567890"  
        altemail="lname.fname.$n@domain.com"  
        mobile="9876543210"
        buf+="$fname,$lname,$email,$password,$altemail,$mobile"$'\n'
    done
    printf '%s\n' "$buf" > "$tmp/$l" ) &
done # spawning..
wait
# Merging. The filename order from globbing will be a mess,
# since we didn't format $l to some 0-prefixed numbers.
# Let's just do the loop again.
for ((l = 0; l < max; l += step)); do
    printf '%s\n' "$(<"$tmp/$l")" >> /opt/list.csv
done # merging..
rm -rf -- "$tmp" # cleanup
  • +1 (Nice one, not only explaining it but also supplied a correction) – Bonsi Scott Jan 1 '14 at 15:19
  • @Hennes actually that's not entirely correct. The shell doesn't do the calculation. The expr utility does. It doesn't even start a new shell, it just does a fork/exec. Mark's explanation is more accurate. – Patrick Jan 1 '14 at 22:58
  • You are right. The shell does not start a full new shell. Instead it runs static int execute_in_subshell (command, asynchronous, pipe_in, pipe_out, fds_to_close) which has the clear comment "/* If this is a simple command, tell execute_disk_command that it might be able to get away without forking and simply exec."... I'll delete my erronious comment/ – Hennes Jan 1 '14 at 23:33
  • @Mark Plotnick Awesome! Took less than 15 minutes! So I guess the Moral of the Story is to use shell built-in as much as and where ever possible. Right?! – Gautam Somani Jan 2 '14 at 7:01
  • 2
    If you can use builtins as much as possible without making the script difficult to understand, I'd do that. But if speed is important, and if you don't need the shell's filename expansion or process pipelining or backgrounding capabilities, it might actually be better to use another language such as awk or perl. When I rewrote your script in awk, for instance, it took just 50 seconds to run. – Mark Plotnick Jan 2 '14 at 9:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.