2
votes
2answers
45 views

split a file, pass each piece as a param to a script, run each script in parallel

I have a words.txt with 10000 words (one to a line). I have 5,000 documents. I want to see which documents contain which of those words (with a regex pattern around the word). I have a script.sh that ...
3
votes
3answers
115 views

Is the output mixed like this because of xargs and how can I fix it?

I'm learning a bit of Bash by using xargs to list whois records for a bunch of IP addresses. The command used is: echo "$1" | tr "\n" "\0" | xargs -0 -n 1 -t -P 3 -I % sh -c 'echo "\n 44rBegin whois ...
6
votes
3answers
257 views

Is it safe to pipe the stdout of A to B in parallel

I'm running A file | B --params > file.txt. Since I want to accelerate the processing speed, I used the parallel -j+0< a.txt to run 20 jobs concurrently. a.txt contains all the commands: A ...
1
vote
0answers
314 views

How to use rsync with GNU parallel?

I am using scp to copy the files in parallel using GNU parallel with my below shell script and it is working fine. I am not sure how can I use rsync in place of scp in my below shell script. I am ...
1
vote
1answer
75 views

How to copy in two folders simultaneously using GNU parallel by spawning multiple threads?

I am trying to copy files from machineB and machineC into machineA as I am running my below shell script on machineA. If the files is not there in machineB then it should be there in machineC for ...
1
vote
2answers
122 views

Variable value is not recognized after using gnu parallel?

I have a below shell script from which I am trying to copy 5 files in parallel. I am running my below shell script on machineA which tries to copy the file from machineB and machineC. If the file is ...
2
votes
3answers
88 views

How to split the array in set of five files and download them in parallel?

I am trying to copy files from testMachineB and testMachineC into testMachineA as I am running my shell script on testMachineA. If the file is not there in testMachineB, then it should be there in ...
3
votes
3answers
253 views

Bash, how to let some background processes run but wait for others?

I have (yet) another wait, &, && control flow question.. Say I have a script something like this where I want to do as much work at the same time as possible: # may take some hours ...
6
votes
4answers
589 views

How to run parallel processes and combine outputs when both finished

I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file. I then do the same again for a separate similar dataset ...
3
votes
4answers
2k views

Ctrl-C with two simultaneous commands in bash

I want to run two commands simultaneously in bash on a Linux machine. Therefore in my ./execute.sh bash script I put: command 1 & command 2 echo "done" However when I want to stop the bash ...
2
votes
3answers
803 views

Parallelize a Bash FOR Loop

I have been trying to parallelize the following script, specifically each of the three FOR loop instances, using GNU Parallel but haven't been able to. The 4 commands contained within the FOR loop run ...
1
vote
1answer
143 views

Is there a simple way of limiting the number of child processes that execute in parallel?

I have a bash script that kicks off multiple rsync processes in parallel. I'd like to limit the number of simultaneous child processes but not serialize the whole script. Here's a simplified version ...
3
votes
1answer
130 views

Variable declaration in parallel sh -c …

I've been trying to process the output of find with parallel, which in turn invoked a shell (some textual substitutions were needed). I observed some strange behaviour, which I cannot really explain ...
3
votes
2answers
174 views

Running up to X commands in parallel

I'm running something like this: find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do echo "Encrypting $file..." gpg --trust-model always --recipient "me@myself.com" ...
8
votes
2answers
2k views

Executing piped commands in parallel

Consider the following scenario. I have two programs A and B. Program A outputs to stdout lines of strings while program B process lines from stdin. The way to use these two programs is of course ...
2
votes
1answer
89 views

replacement inside parallel command string

I want to download multiple files and save them to the md5 sum of its name: cat list | parallel -j4 "md5=$(wget -O - {} | tee tmpfile_{#} | md5sum | cut -d ' ' -f 1); mv tmpfile_{#} $md5" but the ...
4
votes
4answers
553 views

Using GNU Parallel With Split

I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU ...
5
votes
1answer
499 views

Running parallel bash jobs on a HPC cluster using GNU parallel

On an HPC cluster I am trying to run multiple bash scripts (permute2.sh) from 1 bash script using GNU parallel, however it doesn't complete every job. It randomly completes one job, while it is stuck ...
2
votes
1answer
218 views

Is there a way to run process parallelly in the loop of a bash script [duplicate]

Possible Duplicate: Parallelizing a for loop The original code might look like this: for i in *; do something.py $i; done I was wondering whether I can run these jobs parallelly in the ...
3
votes
3answers
542 views

Copy multiple files to one dir with parallel

I'm using the following script to copy multiple files into one folder: { echo $BASE1; echo $BASE2; echo $BASE3; } | parallel cp -a {} $DEST Is there any way to use only one echo $BASE with brace ...
2
votes
3answers
291 views

control number of started programs in bash

As part of my research project I'm processing huge amount of data splitted up into many files. All files in folder foo have to be processed by the script myScript involving all elements of folder ...
7
votes
4answers
3k views

using parallel to process unique input files to unique output files

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
9
votes
2answers
3k views

Parallelizing a for loop

I want to parallelize the for loops of the following code. How to do this? #!/bin/bash N=$1 n=$2 for (( i=1; i<=$N; i++ )); do min=100000000000000 //set min to some garbage value for (( ...