Questions tagged [gnu-parallel]
GNU parallel is a command line utility to run programs in parallel
245
questions
243
votes
10
answers
285k
views
Parallelize a Bash FOR Loop
I have been trying to parallelize the following script, specifically each of the three FOR loop instances, using GNU Parallel but haven't been able to. The 4 commands contained within the FOR loop run ...
- 2,533
50
votes
1
answer
23k
views
GNU parallel vs & (I mean background) vs xargs -P
I'm confused about the difference or advantage (if any) of running a set of tasks in a .sh script using GNU parallel
E.g. Ole Tange's answer:
parallel ./pngout -s0 {} R{} ::: *.png
rather than say ...
35
votes
7
answers
117k
views
Parallelise rsync using GNU Parallel
I have been using a rsync script to synchronize data at one host with the data at another host. The data has numerous small-sized files that contribute to almost 1.2TB.
In order to sync those files, ...
- 3,098
20
votes
3
answers
10k
views
Can GNU parallel output stdout before the program has exited?
echo 'echo "hello, world!";sleep 3;' | parallel
This command does not output anything until it has completed. Parallel's man page claims:
GNU parallel makes sure output from the commands ...
- 3,213
18
votes
6
answers
27k
views
using parallel to process unique input files to unique output files
I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
- 325
16
votes
2
answers
7k
views
Why does (GNU?) parallel fail silently, and how do I fix it?
In a larger script to post-process some simulation data I had the following line:
parallel bnzip2 -- *.bz2
Which, if I understand parallel correctly (and I may not), should run n-core threads of the ...
- 1,303
13
votes
3
answers
7k
views
How would I use GNU Parallel for this while loop?
So I have a while loop:
cat live_hosts | while read host; do \
sortstuff.sh -a "$host" > sortedstuff-"$host"; done
But this can take a long time. How would I use GNU Parallel for this while ...
- 669
13
votes
1
answer
2k
views
Why doesn't GNU parallel work with "bash -c"?
% echo -e '1\n2' | parallel "bash -c 'echo :\$1' '' {}"
:1
:2
% echo -e '1\n2' | parallel bash -c 'echo :\$1' '' {}
%
I'd expect the second line to act the same.
- 333
10
votes
2
answers
14k
views
GNU Parallel: immediately display job stderr/stdout one-at-a-time by jobs order
I know that GNU Parallel buffers std/stderr because it doesn't want jobs output to be mangled, but if I run my jobs with parallel do_something ::: task_1 task_2 task_3, is there anyway for task_1's ...
- 536
9
votes
4
answers
5k
views
Using GNU Parallel With Split
I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU ...
- 285
9
votes
1
answer
8k
views
Does "parallel --jobs 10" mean that exactly 10 jobs will run?
When specifying the option --jobs to GNU parallel, what exactly does it mean?
I execute:
parallel --jobs 10 ./program ::: {1..100}
where program is an intensive task, and the jobs are completely ...
- 1,597
9
votes
2
answers
3k
views
GNU Parallel Limit Memory Usage
Is it possible to limit the memory usage of all processes started by GNU parallel? I realize there are ways to limit the number of jobs, but in cases where it isn't easy to predict the memory usage ...
- 341
8
votes
2
answers
11k
views
Can GNU Parallel execute more parallel processes?
Can I for example execute:
parallel -j 200 < list0
Where "list" has:
nice -n -20 parallel -j 100 < list2
nice -n -20 parallel -j 100 < list1
Would this be feasible/possible?
- 4,975
8
votes
2
answers
2k
views
How do I use GNU parallel to calculate sha256 hash?
Based on this:
Simultaneously calculate multiple digests (md5, sha256)?
I have a folder that has a large number of files that I want to compute the SHA256 hash for.
I used to code segment:
#!/bin/...
- 91
8
votes
2
answers
6k
views
GNU parallel - two parameters from array as parameter
I have a script that uses gnu parallel. I want to pass two parameters for each "iteration"
in serial run I have something like:
for (( i=0; i<=10; i++ ))
do
a = tmp1[$i]
b = tmp2[$i]
done
...
- 265
8
votes
3
answers
4k
views
How to use GNU parallel effectively
Suppose I want to find all the matches in compressed text file:
$ gzcat file.txt.gz | pv --rate -i 5 | grep some-pattern
pv --rate used here for measuring pipe throughput. On my machine it's about ...
- 181
8
votes
2
answers
1k
views
Poor Man's GNU Parallel implemented in ksh?
I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across ...
- 725
8
votes
3
answers
11k
views
How to get GNU parallel on Amazon Linux?
Preferably without having to compile it from source. I tried adding repositories I found on Google: CentOS 6 and CentOS 5, but both give me:
[ec2-user@ip-10-0-1-202 yum.repos.d]$ sudo yum install ...
- 231
7
votes
2
answers
5k
views
How to run x instances of a script parallel?
I have script I'd always like to run 'x' instances in parallel.
The code looks a like that:
for A in
do
for B in
do
(script1.sh $A $B;script2.sh $A $B) &
done #B
done #A
The scripts ...
- 85
7
votes
2
answers
4k
views
gnu parallel: quoted arguments and spaces in file names. How to solve?
This is driving me crazy.
I have a bunch of mp3 that I want to transcode using ffmpeg.
I'm trying to use this one-liner (the script is bigger, this is the problematic section):
find . -type f \( -...
7
votes
3
answers
4k
views
Using parallel on Ubuntu
I am having an issue trying to use parallel command on Ubuntu 10.04. I looked up the parallel documentation and few of the commands seem to run. In all cases I just get the command prompt back without ...
- 277
7
votes
1
answer
2k
views
Running parallel bash jobs on a HPC cluster using GNU parallel [closed]
On an HPC cluster I am trying to run multiple bash scripts (permute2.sh) from 1 bash script using GNU parallel, however it doesn't complete every job. It randomly completes one job, while it is stuck ...
- 283
6
votes
3
answers
502
views
Is it safe to pipe the stdout of A to B in parallel
I'm running A file | B --params > file.txt. Since I want to accelerate the processing speed, I used the parallel -j+0< a.txt to run 20 jobs concurrently. a.txt contains all the commands:
A ...
- 69
6
votes
2
answers
5k
views
split a file, pass each piece as a param to a script, run each script in parallel
I have a words.txt with 10000 words (one to a line). I have 5,000 documents. I want to see which documents contain which of those words (with a regex pattern around the word). I have a script.sh that ...
- 6,285
6
votes
1
answer
1k
views
Multi-threading diff
I'm comparing the content of two directories using the diff command in bash.
It uses only one thread on a multi-thread system.
Does somebody knows a way to have diff to use more than one cpu/thread? ...
- 81
5
votes
2
answers
3k
views
How can I run GNU parallel in record per job, with 1 process per core
What I'm really trying to do is run X number of jobs, with X amount in parallel for testing an API race condition.
I've come up with this
echo {1..10} | xargs -n1 | parallel -m 'echo "{}"';
which ...
- 56.9k
5
votes
1
answer
3k
views
gnu parallel how to use with ffmpeg
I have folder that contains some mp4 files I want to extract images from each of them in parallel and save it as:
filename_number.bmp
where filename is the input filename
how to do that using ...
- 183
5
votes
3
answers
4k
views
Compute md5sum for each line in a file
Given a file like:
a
b
c
How do I get an output like:
a 0cc175b9c0f1b6a831c399e269772661
b 92eb5ffee6ae2fec3ad71c777531578f
c 4a8a08f09d37b73795649038408b5f33
in an efficient way?...
- 32.7k
5
votes
2
answers
696
views
GNU parallel ssh jobs: What happens to an incomplete job if the server dies?
Suppose we want to dispatch jobs to a collection of servers using GNU parallel. What would happen if one of the servers die(power failure, thermal shutdown...) while busy executing a job? Will GNU ...
- 205
5
votes
2
answers
1k
views
Standard Out Append to File Size Limitations
I'm pulling VIN specifications from the National Highway Traffic Safety Administration API for approximately 25,000,000 VIN numbers. This is a great deal of data, and as I'm not transforming the data ...
- 53
5
votes
1
answer
3k
views
GNU parallel excessively slow
I need to run grep on a couple of million files. Therefore I tried to speed it up, following the two approaches mentioned here: xargs -P -n and GNU parallel. I tried this on a subset of my files (9026 ...
- 63
5
votes
2
answers
1k
views
BASH: parallel run
I have a bash script that takes as input three arrays with equal length: METHODS, INFILES and OUTFILES.
This script will let METHODS[i] solves problem INFILES[i] and saves the result to OUTFILES[i], ...
- 209
5
votes
2
answers
6k
views
scp files from particular folder in parallel
I want to scp files from machineA into my machineB and this is how I am doing it. I am copying bunch of files one by one from primary folder of machineA to primary folder of machineB and secondary ...
- 2,077
5
votes
2
answers
553
views
xargs, records, and standard input
I have a 0-byte-delimited file of records.
Record 1, Line 1
Record 1, Line 2
[zero byte]
Record 2, Line 1
Record 2, Line 2
I'd like to run the "process.sh" command once for each record, with the ...
- 327
5
votes
1
answer
1k
views
Running GNU Parallel on 2 or more nodes with Slurm scheduler
I am trying to distribute independent runs of a process using GNU Parallel on a HPC that uses Slurm workload manager. Briefly, here is the data analysis set up:
Script#1: myCommands
./myscript --...
- 191
4
votes
3
answers
421
views
Rename all hidden directories created by GNU parallel
GNU Parallel is outputting hidden directories as follows using the --results parameter.
What command do I use on Ubuntu to change the all to directories so that they are no longer hidden. The ...
- 325
4
votes
1
answer
2k
views
Running a program with multiple arguments from list using parallel
I have a bash program that I run via command line (Ubuntu) like this:
./extract_field.sh ABC001
where ABC001 is the field ID that I want to extract from a given shapefile.
To run this script with ...
- 187
4
votes
2
answers
38k
views
MaxStartups and MaxSessions configurations parameter for ssh connections?
I am copying the files from machineB and machineC into machineA as I am running my below shell script on machineA.
If the files is not there in machineB then it should be there in machineC for sure ...
- 2,983
4
votes
3
answers
9k
views
process files in a directory as they appear [duplicate]
Possible Duplicate:
How to run a command when a directory's contents are updated?
I'm trying to write a simple etl process that would look for files in a directory each minute, and if so, ...
- 325
4
votes
3
answers
4k
views
Running local shell script on multiple remote server in parallel and get the status
I need to run a local script on remote servers. Running script to run in parallel is important.
[ec2-user@ip-172-31-43-140 ~]$ cat hosts.txt
ec2-user@18.218.191.143
ec2-user@...
- 41
4
votes
1
answer
504
views
Multi-machine tool in the spirit of moreutils' `parallel`?
parallel from moreutils is a great tool for, among other things, distributing m independent tasks evenly over n CPUs. Does anybody know of a tool that accomplishes the same thing for multiple machines?...
- 199
4
votes
2
answers
380
views
GNU parallel combination of array arguments?
I have a bash function:
println(){echo "$1" "$2" "$3"}
and
export -f println
I want to use GNU parallel to run println. The command should take inputs of the following ...
- 41
4
votes
2
answers
347
views
How to copy files from local datacenter to local datacenter and have some failover cases?
I have a script which knows about on which HOSTNAME it will be running as it has the HOSTNAME variable which tells about this and it is being passed from outside. I want to check which datacenter that ...
- 2,983
4
votes
1
answer
787
views
Why is parallel --nice not setting the niceness?
If I run a command with nice, then I can see its process having the expected niceness value:
In one terminal:
nice sleep 17
and in another one:
$ ps -aoni,comm | grep sleep
10 sleep
But trying ...
- 143
4
votes
1
answer
1k
views
Why does my parallel command print “Starting” and ”Finished“ at the same time?
ls *.txt | parallel 'echo Starting on file {}; mkdir {.}; cd {.}; longCMD3 ../{} > /dev/null; echo Finished file {}'
This one liner partially works, except longCMD3 takes about 3 minutes, but the ...
- 143
4
votes
1
answer
2k
views
Prevent GNU parallel from splitting quoted arguments
I'm trying to use GNU Parallel to run a comman mutliple times with a combination of constant and varying arguments. But for some reason the constant arguments are split on white-space even though I've ...
- 607
4
votes
1
answer
1k
views
Variable declaration in parallel sh -c …
I've been trying to process the output of find with parallel, which in turn invoked a shell (some textual substitutions were needed). I observed some strange behaviour, which I cannot really explain ...
- 2,270
4
votes
2
answers
3k
views
Copy multiple files to one dir with parallel
I'm using the following script to copy multiple files into one folder:
{ echo $BASE1; echo $BASE2; echo $BASE3; } | parallel cp -a {} $DEST
Is there any way to use only one echo $BASE with brace ...
- 65
4
votes
1
answer
980
views
Make GNU Parallel not delay before executing arguments from STDIN
GNU Parallel, without any command line options, allows you to easily parallelize a command whose last argument is determined by a line of STDIN:
$ seq 3 | parallel echo
2
1
3
Note that parallel does ...
- 145
4
votes
1
answer
3k
views
GNU Parallel - grepping n lines for m regular expressions
The GNU parallel grepping n lines for m regular expressions example states the following:
If the CPU is the limiting factor parallelization should be done on
the regexps:
cat regexp.txt | ...
- 16.4k