2

How can I do parallel processing in bash?

My approach, in my bash script, added so many background jobs with &, and then I am adding each process id using an array, then I am fetching that array using a loop and inside that I am using wait command, but somehow wait command is not working.

my code is like this

for i in some path 
do 
    ls -d $i | xargs du -kh --max-depth=0 |sed 's/\t/,/g' >> $TEMP/Outfile/disk_usage_Session_wise.csv & 
    pid=$!
    ls -d $i/session | xargs du -kh --max-depth=1 | sed 's/\t/,/g' >> $TEMP/Outfile/disk_usage_Session_wise.csv & 
    pid_1=$! 
    process_list=($pid $pid_1) 
done 

for job in echo ${process_list[@]} 
    do 
    echo "$job is running" 
    wait $job # this command is not working 
done –

closed as unclear what you're asking by user181255, Anthon, HalosGhost, countermode, GAD3R Dec 6 '16 at 18:54

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    When you say that wait is not working, what exactly do you mean? Can you show some code that's not behaving as you expect and describe what you want it to do as opposed to what it's actually doing? – DopeGhoti Dec 6 '16 at 16:12
  • Have you considered a language with built-in parallel processing capabilities such as python or Perl ? – steve Dec 6 '16 at 16:16
  • 1
    It would look better in your question. – Tomasz Dec 6 '16 at 16:29
  • @Tomas - thanks ..updated my question session . .i think now it looks good .. – Ashish Sahoo Dec 6 '16 at 16:37
  • 1
    you don't need the PIDs. wait without argument will wait for all background jobs to finish. – lesmana Dec 6 '16 at 17:02
2

Can you use additional software or only bash? Looks like GNU Parallel is what you need for your task. Or at least -n and -P arguments for xargs

1

This improved line should fix the first part of your script:

process_list+=($pid $pid_1) 

And this should be the second part. Notice it's not been tested.

for job in ${process_list[@]} 
do 
    ps -ho pid | >/dev/null grep $job && {
        echo "$job is running" 
        wait $job # this command should be ok now         
    }
done
  • ps -ho pid produces a list of running processes (-o pid) without the heading line (-h).

  • It forwards (|) this list to grep

  • grep looks for the $job and if it finds it (&&), the block of commands ({echo ...}) is run

  • Can you explain what this line is doing ? ps -ho pid | grep $job && – Ashish Sahoo Dec 6 '16 at 17:27
  • @AshishSahoo see update – Tomasz Dec 6 '16 at 17:32
  • is it expected ? i am getting these error when i tried to run your comamnd $ ps -ho pid | head Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ 998 – Ashish Sahoo Dec 6 '16 at 17:35
  • @AshishSahoo I added >/dev/null to keep it clean with regards to grep's output. – Tomasz Dec 6 '16 at 17:35
  • @AshishSahoo No, that's unexpected. Please check for mistypes. – Tomasz Dec 6 '16 at 17:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.