3
votes
3answers
74 views

Bash, how to let some background processes run but wait for others?

I have (yet) another wait, &, && control flow question.. Say I have a script something like this where I want to do as much work at the same time as possible: # may take some hours ...
6
votes
4answers
378 views

How to run parallel processes and combine outputs when both finished

I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file. I then do the same again for a separate similar dataset ...
1
vote
4answers
306 views

Ctrl-C with two simultaneous commands in bash

I want to run two commands simultaneously in bash on a Linux machine. Therefore in my ./execute.sh bash script I put: command 1 & command 2 echo "done" However when I want to stop the bash ...
2
votes
3answers
174 views

Parallelize a Bash FOR Loop

I have been trying to parallelize the following script, specifically each of the three FOR loop instances, using GNU Parallel but haven't been able to. The 4 commands contained within the FOR loop run ...
1
vote
1answer
116 views

Is there a simple way of limiting the number of child processes that execute in parallel?

I have a bash script that kicks off multiple rsync processes in parallel. I'd like to limit the number of simultaneous child processes but not serialize the whole script. Here's a simplified version ...
3
votes
1answer
97 views

Variable declaration in parallel sh -c …

I've been trying to process the output of find with parallel, which in turn invoked a shell (some textual substitutions were needed). I observed some strange behaviour, which I cannot really explain ...
3
votes
2answers
134 views

Running up to X commands in parallel

I'm running something like this: find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do echo "Encrypting $file..." gpg --trust-model always --recipient "me@myself.com" ...
7
votes
2answers
918 views

Executing piped commands in parallel

Consider the following scenario. I have two programs A and B. Program A outputs to stdout lines of strings while program B process lines from stdin. The way to use these two programs is of course ...
2
votes
1answer
79 views

replacement inside parallel command string

I want to download multiple files and save them to the md5 sum of its name: cat list | parallel -j4 "md5=$(wget -O - {} | tee tmpfile_{#} | md5sum | cut -d ' ' -f 1); mv tmpfile_{#} $md5" but the ...
4
votes
4answers
432 views

Using GNU Parallel With Split

I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU ...
5
votes
1answer
392 views

Running parallel bash jobs on a HPC cluster using GNU parallel

On an HPC cluster I am trying to run multiple bash scripts (permute2.sh) from 1 bash script using GNU parallel, however it doesn't complete every job. It randomly completes one job, while it is stuck ...
2
votes
1answer
180 views

Is there a way to run process parallelly in the loop of a bash script [duplicate]

Possible Duplicate: Parallelizing a for loop The original code might look like this: for i in *; do something.py $i; done I was wondering whether I can run these jobs parallelly in the ...
3
votes
3answers
443 views

Copy multiple files to one dir with parallel

I'm using the following script to copy multiple files into one folder: { echo $BASE1; echo $BASE2; echo $BASE3; } | parallel cp -a {} $DEST Is there any way to use only one echo $BASE with brace ...
2
votes
3answers
244 views

control number of started programs in bash

As part of my research project I'm processing huge amount of data splitted up into many files. All files in folder foo have to be processed by the script myScript involving all elements of folder ...
6
votes
4answers
2k views

using parallel to process unique input files to unique output files

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
9
votes
2answers
2k views

Parallelizing a for loop

I want to parallelize the for loops of the following code. How to do this? #!/bin/bash N=$1 n=$2 for (( i=1; i<=$N; i++ )); do min=100000000000000 //set min to some garbage value for (( ...