Questions tagged [gnu-parallel]
GNU parallel is a command line utility to run programs in parallel
30
questions with no upvoted or accepted answers
5
votes
1
answer
1k
views
Running GNU Parallel on 2 or more nodes with Slurm scheduler
I am trying to distribute independent runs of a process using GNU Parallel on a HPC that uses Slurm workload manager. Briefly, here is the data analysis set up:
Script#1: myCommands
./myscript --...
- 191
4
votes
1
answer
2k
views
Parallel Copy from local folders to remote servers at the same time
I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the ...
- 41
3
votes
0
answers
72
views
2>&1 causes process to block
I am trying to debug what I have done wrong in GNU Parallel (>=20190122).
This runs as expected:
(
echo 1 | parallel-20190122 --pipe -j0 --tee dummy {} ::: {1..1000};
echo Parallel is done here:...
- 32.7k
2
votes
2
answers
102
views
List a file from all directories but skip the ones whose folder contains a specific file
I have a perl script that processes .xml files which reside in multiple folders under a directory. The script takes the absolute file path using the output of ls using parallel.
parallel -j 1 /home/...
- 21
2
votes
1
answer
606
views
GNU Parallel only using one CPU Core
I'm running some training using GNU parallel. The training takes about 30 secs to run one iteration, and I need to run about 3000. The training itself can't be parallelized (at least not without ...
- 21
2
votes
1
answer
305
views
bash array printf in gnu parallel
I am trying pass bash array variable, that contain two elements to GNU parallel script:
echo ${opts[@]} ...
- 141
2
votes
0
answers
281
views
Change output width (columns) of processes created by gnu parallel
I'm trying to save the output of a command into files and it seems to work fine. However, when opening one of the files, I can see that output was limited to 80 characters width. How can I ensure that ...
- 121
2
votes
1
answer
370
views
Can GNU Parallel Alter the Output of a Bash Script
I have a directory which is filled with subdirectories and in each subdirectory 74 files. I have bash script which analyzes the 74 files and wish to run it in parallel on all directories.
When I go ...
- 67
2
votes
1
answer
1k
views
Running GNU parallel --sem hangs on SGE Cluster
I'm trying to run 3 commands in parallel on an SGE cluster, e.g.
cmds=("prog 1 2 3" "prog 4 5 6" "prog 7 8 9")
for cmd in "${cmds[@]}"
do
sem -j 3 $cmd
done
sem --wait
So, my commands execute ...
- 21
1
vote
1
answer
25
views
Why the cpu usage of one process decreased when multiple processes are running?
I'm using gnu parallel like below:
find conf/20230214/ -name "conf.yaml" | parallel run_backtest
the run_backtest is a function which will read the conf.yaml and run some code.
When there ...
- 11
1
vote
1
answer
73
views
parallel compare two arguments and send to command
I am trying to run recon-all command with GNU parallel. It should take one argument at {1} like this:
parallel --progress subj_id='echo {1} | cut -d / -f 3' && recon-all -s $subj_id -i {1} -sd ...
1
vote
0
answers
187
views
Combine gnu-parallel and curl in Linux
I'm testing my webserver and sending API Thousand of request to optimize my program with high traffic. Indeed I write two bash-shells but they not fast as I want any one can help me on this and let me ...
- 49
1
vote
1
answer
249
views
Combining gnu parallel and mpirun: how to take care of the max number of cores?
I have a machine with 32 cores where a I want to run some mpi jobs with a variable number of cores (this is because the problem requires that). Is there a way to make gnu parallel aware of the number ...
- 111
1
vote
1
answer
245
views
running a program in parallel with multiple threads
Say I want to run a parallelize a program to process a bunch of files, and this program accepts a -t option to increase the number of threads used. The machine I am using has the following specs from ...
- 11
1
vote
1
answer
538
views
What are the requirements of gnu-parallel to/for a remote device that is contacted via ssh (and public-key authentication)
I am using gnu-parallel.
I assume there is a problem with my environment.
What I want to achieve:
Trying to execute a command on multiple machines.
For that I've created a sshloginfile (aptly named ~/....
- 11
1
vote
1
answer
1k
views
GNU Parallel to process multiple files in parallel in Unix with bash
I have a bash script I am using to iterate over a list of files and search for regex matches. Here is a snippet from my current code:
for file in $Files_To_Parse; do
(cat $file) | grep -ioE "($...
- 11
1
vote
0
answers
949
views
Why can't I pipe to parallel?
I have been trying to pipe this command
tail -n +$line $FILENAME
to this command
parallel -a $FILENAME -k --block $sub --pipe-part "cut -d ' ' -f1" | sponge $FILENAME
And the resulting command is ...
- 347
1
vote
1
answer
197
views
Networking error with multiple docker containers running in parallel
I have a number of python files containing unit tests in the directory /code/test In an effort to parallelize my test running without any code editing, I figured I could use GNU Parallel to run N ...
- 111
1
vote
1
answer
339
views
Two GNU Parallel commands executed on the same machine
I'm using GNU Parallel to automatically start a large number of jobs and distribute them on the cores of a machine. One job per core.
parallel python3 program.py ::: inputs1*
When one set of jobs is ...
- 141
1
vote
1
answer
164
views
Parallel for long running filter
I am scraping phone numbers from a certain list of web pages and have implemented map-reduce as:
PARALLEL_OPTS="..."
SCRIPT="curl --silent --cookie '$COOKIES' {} | egrep -o '[2-9][0-9][0-9][0-9][0-9][...
- 153
0
votes
1
answer
60
views
Read file and run parallel operations
what I'm doing here is a bit complex, so hopefully I can explain it well enough so that it makes sense. The tldr version is that I want to read 10 lines of a file and then run a script based on the ...
- 201
0
votes
0
answers
34
views
How to use ${LSB_JOB_INDEX} to run a for loop in LSF platform
I have a for loop that looks like this, and I want to convert it to an array job.
I want to run 100 jobs in parallel, but I am not entirely sure where I can add
the ${LSB_JOB_INDEX} parameter.
Any ...
- 101
0
votes
0
answers
82
views
How do I make this parallel bash script more efficient?
I have a simple script involving afni command line tools that I am running in parallel from bash, but it is very disk I/O intensive and slows things down on our server (using only 2-3 out of the total ...
0
votes
1
answer
231
views
GNU parallel export function output to variables failed
This script is to determine when destination file already exist, source file will update the destination one or be removed according to flag "$dup_act".
#!/bin/bash
dup_chk()
{
# $1: f_src,...
0
votes
1
answer
27
views
Is it possible to continuously feed parallel with more input?
What I want to do is something like this:
inotifywait -m ./input -e close_write -e moved_to |
parallel --max-args 1 "someSlowSingleThreadedTaskThatActsOnThisFile"
I can't imagine that ...
- 142
0
votes
1
answer
988
views
Gnu parallel rsync hanging indefinitely
To be clear, I am running this exact command(variables changed) in a gnu parallel setting:
/usr/bin/parallel -q -j20
rsync --recursive -av -e ssh -oCompression=no -x -T
-oUserKnownHostsFile=...
- 131
0
votes
1
answer
66
views
concurrent max process limit when gnuparallel used in unix
I am using gnu parallel to fire some concurrent process. Below is the command am using.
parallel 'node /home/parunkarthick/NodeProject/nodescript_pd.js F_CHARGES' ::: 1 2 3 4 5 6 7 8 9 10 11 12 13 ...
- 101
0
votes
1
answer
862
views
Grep: repetition-operator operand invalid in GNU Parallel call
I am creating a function to be called in parallel using GNU's parallel package in OSX. The function is as such:
parallel_func() {
(cat $1) | grep -ioE "($first$second$third+$fourth)"
}
export -f ...
- 11
0
votes
1
answer
378
views
Uncompressed .lzo files in parallel in both the folders simultaneously and then delete the original .lzo files
So I have .lzo files in /test01/primary folder which I need to uncompress and then delete all the .lzo files. Same thing I need to do in /test02/secondary folder as well. I will have around 150 .lzo ...
- 751
-1
votes
3
answers
295
views
How to concatenate several file with * in the path using xargs or parallel
I have a command that fetch a path to a file for each line in a CSV file :
awk -F, 'NR>0 {print "/mnt/datagenetique/ANALYSIS/Infectiologie/COVID-WGS/Analyse/" $2 "_*/dragen-covidseq/&...
- 509