4

I have a bash script that takes as input three arrays with equal length: METHODS, INFILES and OUTFILES.

This script will let METHODS[i] solves problem INFILES[i] and saves the result to OUTFILES[i], for all indices i (0 <= i <= length-1).

Each element in METHODSis a string of the form:

$HOME/program/solver -a <method>

where solver is a program that can be called as follows:

$HOME/program/solver -a <method> -m <input file> -o <output file> --timeout <timeout in seconds>

The script solves all the problems in parallel and set the runtime limit for each instance to 1 hour (some methods can solve some problems very quickly though), as follows:

#!/bin/bash
source METHODS
source INFILES
source OUTFILES

start=`date +%s`

## Solve in PARALLEL
for index in ${!OUTFILES[*]}; do 
    (alg=${METHODS[$index]}
    infile=${INFILES[$index]}
    outfile=${OUTFILES[$index]}

    ${!alg}  -m $infile -o $outfile --timeout 3600) &
done
wait


end=`date +%s`

runtime=$((end-start))
echo "Total runtime = $runtime (s)"
echo "Total number of processes = ${#OUTFILES[@]}"

In the above I have length = 619. I submitted this bash to a cluster with 70 available processors, which should take at maximum 9 hours to finish all the tasks. This is not the case in reality, however. When using the top command to investigate, I found that only two or three processes are running (state = R) while all the others are sleeping (state = D).

What am I doing wrong please?

Furthermore, I have learnt that GNU parallel would be much better for running parallel jobs. How can I use it for the above task?

Thank you very much for your help!

Update: My first try with GNU parallel:

The idea is to write all the commands to a file and then use GNU parallel to execute them:

#!/bin/bash
source METHODS
source INFILES
source OUTFILES

start=`date +%s`    

## Write to file
firstline=true
for index in ${!OUTFILES[*]}; do 
    (alg=${METHODS[$index]}
    infile=${INFILES[$index]}
    outfile=${OUTFILES[$index]}
    if [ "$firstline" = true ] ; then
        echo "${!alg}  -m $infile -o $outfile --timeout 3600" > commands.txt
        firstline=false
    else
        echo "${!alg}  -m $infile -o $outfile --timeout 3600" >> commands.txt
    fi
done

## Solve in PARALLEL
time parallel :::: commands.txt

end=`date +%s`

runtime=$((end-start))
echo "Total runtime = $runtime (s)"
echo "Total number of processes = ${#OUTFILES[@]}"

What do you think?

Update 2: I'm using GNU parallel and having the same problem. Here's the output of top:

top - 02:05:25 up 178 days,  8:16,  2 users,  load average: 62.59, 59.90, 53.29
Tasks: 596 total,   7 running, 589 sleeping,   0 stopped,   0 zombie
Cpu(s): 12.9%us,  0.9%sy,  0.0%ni, 63.3%id, 22.9%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  264139632k total, 260564864k used,  3574768k free,     4564k buffers
Swap: 268420092k total, 80593460k used, 187826632k free,    53392k cached

  PID USER     PR  NI  VIRT  RES  SHR S %CPU %MEM   TIME+   COMMAND
28542 khue     20   0 7012m 5.6g 1816 R  100  2.2  12:50.22 opengm_min_sum
28553 khue     20   0 11.6g  11g 1668 R  100  4.4  17:37.37 opengm_min_sum
28544 khue     20   0 13.6g 8.6g 2004 R  100  3.4  12:41.67 opengm_min_sum
28549 khue     20   0 13.6g 8.7g 2000 R  100  3.5   2:54.36 opengm_min_sum
28551 khue     20   0 11.6g  11g 1668 R  100  4.4  19:48.36 opengm_min_sum
28528 khue     20   0 6934m 4.9g 1732 R   29  1.9   1:01.13 opengm_min_sum
28563 khue     20   0 7722m 6.7g 1680 D    2  2.7   0:56.74 opengm_min_sum
28566 khue     20   0 8764m 7.9g 1680 D    2  3.1   1:00.13 opengm_min_sum
28530 khue     20   0 5686m 4.8g 1732 D    1  1.9   0:56.23 opengm_min_sum
28534 khue     20   0 5776m 4.6g 1744 D    1  1.8   0:53.46 opengm_min_sum
28539 khue     20   0 6742m 5.0g 1732 D    1  2.0   0:58.95 opengm_min_sum
28548 khue     20   0 5776m 4.7g 1744 D    1  1.9   0:55.67 opengm_min_sum
28559 khue     20   0 8258m 7.1g 1680 D    1  2.8   0:57.90 opengm_min_sum
28564 khue     20   0 10.6g  10g 1680 D    1  4.0   1:08.75 opengm_min_sum
28529 khue     20   0 5686m 4.4g 1732 D    1  1.7   1:05.55 opengm_min_sum
28531 khue     20   0 4338m 3.6g 1724 D    1  1.4   0:57.72 opengm_min_sum
28533 khue     20   0 6064m 5.2g 1744 D    1  2.1   1:05.19 opengm_min_sum

(opengm_min_sum is the solver above)

I guess that some processes consume so much resource that the others do not have anything left and enter the D state?

  • Processes in D state are typically waiting for I/O ('disk'). It may be that your cluster is lacking in I/O bandwidth. – icarus Feb 8 '17 at 0:44
  • @icarus: Thanks a lot, that is very useful. I'm using GNU parallel instead of wait and encountering the same problem. How should I investigate further to see what is wrong? – Khue Feb 8 '17 at 1:04
  • It is not clear that there is anything wrong. It depends a lot on what the system is trying to do. A couple of extreme examples, running a password cracker it might only read in a single line and output nothing or a single word, and in the mean time use 100% of your CPU. At the other extreme doing a backup of a directory tree without any compression. This would be almost 100% I/O and almost all the processes would show in D. Both things are working correctly. You probably need to see if you can do more work and less I/O or else get more or faster disks. – icarus Feb 8 '17 at 1:13
  • 1
    The system definitely is I/O starved. The system is only using 14% (12.9+0.9) of the available CPU, it is idle 63% and waiting for 23%. The memory consumption looks weird as well, you are using 256GB of ram and 80GB of swap. Once you start using swap things can get very slow. It might be better to scale back the number of things you are running in parallel (just cutting in half is hopefully easy) so the amount of swap used is almost zero. – icarus Feb 8 '17 at 2:10
  • 1
    Programs like ps and top only show you what is happening at the instant something is being run. If a program is reading a lot of data then there is a good chance that at the instant top is running it will be waiting for the disk. I know nothing about your problem but it looks like you have a machine which is fast enough but doesn't have enough memory for you to run all of your programs at the same time and doesn't have enough I/O bandwidth. If you access the data sequentially maybe compressing it will help? Remember to give me credit in your PhD! – icarus Feb 8 '17 at 23:21
2

Summary of the comments: The machine is fast but doesn't have enough memory to run everything in parallel. In addition the problem needs to read a lot of data and the disk bandwidth is not enough, so the cpus are idle most of the time waiting for data.

Rearranging the tasks helps.

Not yet investigated compressing the data to see if it can improve the effective disk I/O bandwidth.

  • It appears that the main issue comes from the software, but your comments and suggestions were very helpful. Could you be more specific on "compressing the data"? (any insight on how to do that). Thanks. – Khue Feb 9 '17 at 21:57
  • @Khue It depends on how you use the data. If you essentially just read it as one long stream then compressing it will probably help. If you can compress your file to a third of the original size (quite reasonable for text file with gzip) then you only need to read a third of the number of bytes from disk. You do need to process the compressed stream, but as we have seen you are not lacking in cpu. Compressing the stream in effect would make your disks 3 times faster. There is no one size fits all approach here. – icarus Feb 10 '17 at 0:17
3

From version 20160422 you can do:

## Solve in PARALLEL
parallel {1} -m {2} -o {3} --timeout 3600 ::: "${METHODS[@]}" :::+ "${INFILES[@]}" :::+ "${OUTFILES[@]}"

If you have an older version:

## Solve in PARALLEL
parallel --xapply {1} -m {2} -o {3} --timeout 3600 ::: "${METHODS[@]}" ::: "${INFILES[@]}" ::: "${OUTFILES[@]}"

Spend an hour walking through man parallel_tutorial. Your command line will love you for it.

  • Very nice! I was updating my question with my first try with GNU parallel. Well, it's not very elegant compared to your solution, but do you have an idea how well the two solutions will perform? Thanks a lot! – Khue Feb 8 '17 at 0:19
  • My solution will use around 250 ms + 10 ms per job. – Ole Tange Feb 8 '17 at 0:22
  • I have just tested your solution and it does not do anything :\ – Khue Feb 8 '17 at 0:52
  • I sincerely apologize for rushing to accept your answer. Your solution does not solve my main problem that is: many processes have the D state. I'm using GNU parallel instead of wait and encountering the same problem. – Khue Feb 8 '17 at 1:07
  • Can you try instead of running your solver, run something that takes < 1 GB RAM per job (like burnP6)? If that makes all processes run, then the problem is due to your system being too small or disk too slow to run as many solvers you want in parallel. Solution: Buy a bigger/faster machine or run fewer solvers in parallel. – Ole Tange Feb 8 '17 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.