Questions tagged [gnu-parallel]
GNU parallel is a command line utility to run programs in parallel
245
questions
1
vote
1
answer
25
views
Why the cpu usage of one process decreased when multiple processes are running?
I'm using gnu parallel like below:
find conf/20230214/ -name "conf.yaml" | parallel run_backtest
the run_backtest is a function which will read the conf.yaml and run some code.
When there ...
- 11
0
votes
1
answer
60
views
Read file and run parallel operations
what I'm doing here is a bit complex, so hopefully I can explain it well enough so that it makes sense. The tldr version is that I want to read 10 lines of a file and then run a script based on the ...
- 201
1
vote
1
answer
111
views
Understanding timestamps in a GNU Parallel --joblog output
Does GNU Parallel start a batch of as many jobs as possible (the number of jobs started being governed by GNU Parallel internals or/and the -j option along with given parameters), and once complete, ...
- 1,440
1
vote
1
answer
89
views
GNU Parallel --joblog logs only first line of commands inside a for loop
I use GNU Parallel along a for loop like this:
for BAND in $(seq 1 "$BANDS") ;do
# Do not extract, unscale and merge if the scaled map exists already!
SCALED_MAP="era5_and_land_${...
- 1,440
2
votes
1
answer
151
views
GNU Parallel with -j -N still uses one CPU
How can I get reasonable parallelisation on multi-core nodes without saturating resources? As in many other similar questions, the question is really how to learn to tweak GNU Parallel to get ...
- 1,440
2
votes
2
answers
148
views
How to compress all files in the current directory using lz4 using GNU parallel?
How to compress all files in the current directory using lz4 using GNU parallel?
I try ls | parallel lz4 but it outputs to stdout. Why? How to fix?
- 457
0
votes
0
answers
34
views
How to use ${LSB_JOB_INDEX} to run a for loop in LSF platform
I have a for loop that looks like this, and I want to convert it to an array job.
I want to run 100 jobs in parallel, but I am not entirely sure where I can add
the ${LSB_JOB_INDEX} parameter.
Any ...
- 101
1
vote
1
answer
82
views
How do we write only fail jobs in a log when we use GNU parallel
parallel --joblog /tmp/log exit ::: 1 2 3 0
cat /tmp/log
Ηow can Ι use a filter to write only failed job in my load when using GNU parallel or is there a way to get only failed jobs from the above ...
3
votes
2
answers
292
views
How to print GNU Parallel command & output together?
I'd like to just check the status of a bunch of Git repos with a quick command like parallel git -C {} status --short ::: ~/*/.git/... But the Git status doesn't include the repo name or path, so I'd ...
- 49.4k
0
votes
3
answers
120
views
How safe is it to use EXPORT in bash scripts when using GNU Parallel?
How safe is it to use export in bash scripts when using GNU Parallel?
I have a parent script.
parent.sh
(echo child.sh & echo child_two.sh) || parallel bash
wait
if [[ "$STATUS1" == &...
2
votes
2
answers
242
views
Difference between xargs and GNU parallel
I need to run an executable a large number of times, each time with two command line arguments. I've used to use xargs for this purpose, but lately I've been made aware of the existence of GNU ...
- 145
1
vote
2
answers
125
views
GNU Parallel with named arguments
I have the following command in my Makefile
parallel \
--eta \
--bar \
--joblog mnist/embedder.joblog \
pipenv run python3 \
-m mnist.train_embedder \
--...
- 369
1
vote
2
answers
116
views
Limit total number of GNU-parallel jobs between mutliple calls
I'm working on a computer with 10k cores, but only allowed to access 1,000 at a time.
I have a script that could benefit from GNU parallel in multiple places. Processing level A in parallel, and ...
- 275
1
vote
1
answer
58
views
Strip leading and trailing whitespace when piping to GNU parallel
When I try to write a pipeline like this:
git branch | rg '^\*' | parallel git pull {}
I run into a problem with whitespace. Because the branch names have leading whitespace, parallel ends up ...
- 23
2
votes
1
answer
242
views
A GNU Parallel Job Queue Script
I found a script on GitHub which I've slightly modified to fit the needs of the program I'm trying to run in a queue.
It is not working however and I'm not sure why. It never actually echos the jobs ...
- 41
0
votes
0
answers
82
views
How do I make this parallel bash script more efficient?
I have a simple script involving afni command line tools that I am running in parallel from bash, but it is very disk I/O intensive and slows things down on our server (using only 2-3 out of the total ...
1
vote
2
answers
400
views
How might I execute this nested for loop in parallel?
#!/usr/bin/bash
TARGETS=(
"81.176.235.2"
"81.176.70.2"
"78.41.109.7"
)
myIPs=(
"185.164.100.1"
"185.164.100.2"
"185.164.100.3"
"185....
- 461
1
vote
1
answer
170
views
GNU parset is not working in a script but works in terminal
I ran into a strange problem.I am running this example from this https://www.gnu.org/software/parallel/parset.html. But it is not working inside the script file.
parset myarray seq 3 ::: 4 5 6
echo &...
1
vote
2
answers
300
views
Parallel md5sum on all *.txt files in directory
Trying to calculate md5 hash of all *.txt files in directory and store each in same filename but with extension replaced from .txt to .md5.
For file: foo.txt
i want to create file: foo.md5
with ...
- 131
0
votes
1
answer
231
views
GNU parallel export function output to variables failed
This script is to determine when destination file already exist, source file will update the destination one or be removed according to flag "$dup_act".
#!/bin/bash
dup_chk()
{
# $1: f_src,...
2
votes
1
answer
93
views
Adding to while executing a list of commands
I'm compiling a huge list of commands (all doing the same thing) I want executed, but as it takes a long time to compile that list, I would like execution to begin before I'm done (execution of each ...
1
vote
1
answer
66
views
Bash script hangs when filtering stderr through sed
Introduction
I have a bash script to execute a command in multiple servers through ssh. It uses GNU parallel in the parallel version, a for loop in the sequential one.
The script is used like this:
...
1
vote
2
answers
73
views
Passing second argument to GNU parallel while feeding ls
I process files found by ls as
ls /folder/ | parallel -j20 ./command {}
but I need to pass the job number as well. I tried
ls /folder/ | parallel -j20 ./command {1} {2} ::: {1..20}
ls /folder/ | ...
- 1,869
2
votes
1
answer
805
views
Use GNU parallel to run script with different input arguments
I have a script called step1.sh, and I pass one input parameter to the script ($1).
I can run the script like this if I want the input to be 300.1
./step1.sh 300.1
I want to run this script 200 times,...
- 21
0
votes
1
answer
324
views
pass variable from shell script to GNU parallel [duplicate]
I am using two shell scripts to set up a parallelization. The first script uses a for loop to pass an environment variable to the second script, which makes the call to parallel, which runs a julia ...
- 101
-1
votes
3
answers
295
views
How to concatenate several file with * in the path using xargs or parallel
I have a command that fetch a path to a file for each line in a CSV file :
awk -F, 'NR>0 {print "/mnt/datagenetique/ANALYSIS/Infectiologie/COVID-WGS/Analyse/" $2 "_*/dragen-covidseq/&...
- 509
0
votes
2
answers
416
views
How to find the number of jobs (created by gnu parallel) running in each CPU core at any given time?
I'm using gnu parallel that reads a text file containing curl commands.
If I do ps -ef | grep -cw [c]url it shows total number of curl process at a moment.
But I want to know number of gnu parallel ...
- 119
1
vote
3
answers
762
views
GNU Parallel vs. xargs vs. Python tools - Sending 10000 reqs/second [closed]
I have 1000000 different requests (different request I mean it has different query param. It is just a GET request, and no payload. Size of request and response are in order of KBs only. no images or ...
- 119
2
votes
1
answer
468
views
Make parallel http requests using raw data files
I've several files containg POST body requests.
I'd like to send those requests in parallel.
Related curl command is like:
curl -s -X POST $FHIR_SERVER/ -H "Content-Type: application/fhir+json&...
- 315
1
vote
2
answers
148
views
Why my gnu parallel with ffmpeg execute ffmpeg only on remote host?
I want to use ffmpeg using two pc, i know parallel can do it
I use this cli
parallel --trc {.}.mkv -S virtual,: 'ffmpeg -y -i {} -vf yadif,crop=720:550:0:12,scale=640:352 -c:v libx264 -c:a aac -b:v ...
- 11.7k
1
vote
2
answers
626
views
How to use GNU Parallel for executing a program concurrently?
I need to run a python script several times in parallel but I have done executing it in the background like this
ipython program.py & ipython program.py & ...
I want to know if this way uses ...
- 13
2
votes
4
answers
380
views
Parallel and ls with white spaces
This command works fine for dir with "regular" names as dir1 mydir my-dir, etc
ls | parallel 'echo -n {}" "; ls {}|wc -l'
give me the number of files for each dir
but for dir with ...
- 11.7k
1
vote
2
answers
671
views
How can I use sem to run a shell command in parallel in a for loop?
This isn't speeding anything up for me:
domains=()
for i in `seq 1 100`; do
echo $i
word=`sem "xidel -s 'https://www.thisworddoesnotexist.com/' -e '//div[@id="definition-word"]/...
- 1,961
0
votes
2
answers
103
views
Do any shells generalize the notion of pipes to multiple parallel pipelines? Would supporting it require changes to the OS kernel?
Quite often, when working on the command line, I find myself specifying the same operation on a bunch of different instances specified by some input stream, and then wanting to recombine their outputs ...
- 420
2
votes
2
answers
103
views
--skip-first-line in GNU parallel not working with --pipepart?
Suppose I have a file like this:
COLUMN
1
2
3
4
If I wanna run and process it with GNU parallel but skipping first line aka header, I tried this:
parallel -a test.txt -k --pipepart --will-cite --skip-...
- 181
3
votes
3
answers
725
views
Can GNU Parallel be made to output the command line executed when run in linewise mode?
Suppose I have a list of commands in file cmd_file.
I run these commands via:
cat cmd_file | parallel -k -I {} "{}"
One of the commands fails. All of the commands use the exact same CLI ...
- 791
1
vote
1
answer
137
views
How to parallelize a recursive bash function?
I have a bash function that mainly curls an endpoint for a set of links, and again curls each of those links (for another set of links) recursively.
task() {
link="$1"
response=$(curl &...
1
vote
1
answer
106
views
Is it with GNU parallel possible to split on character count, but provide full lines only?
I would like to split an input file on character count (ASCII is fine), combined with new lines as well. That is, every group of 10000 character should be seen as one record to be piped into the child ...
4
votes
2
answers
380
views
GNU parallel combination of array arguments?
I have a bash function:
println(){echo "$1" "$2" "$3"}
and
export -f println
I want to use GNU parallel to run println. The command should take inputs of the following ...
- 41
4
votes
1
answer
2k
views
Running a program with multiple arguments from list using parallel
I have a bash program that I run via command line (Ubuntu) like this:
./extract_field.sh ABC001
where ABC001 is the field ID that I want to extract from a given shapefile.
To run this script with ...
- 187
2
votes
3
answers
461
views
A universal `while read something` to `parallel` conversion/replacement
I have a ton of scripts which do this:
command | while read something; do
a
long
list
of
commands
done
Has anyone ever thought of how to run all the commands fed through the pipe ...
3
votes
2
answers
388
views
How can I use basename with parallel?
I have files like this on a Linux system:
10S1_S5_L002_chrm.fasta SRR3184711_chrm.fasta SRR3987378_chrm.fasta SRR4029368_chrm.fasta SRR5204465_chrm.fasta SRR5997546_chrm.fasta
...
- 55
0
votes
1
answer
543
views
How to run an expect script in parallel rather than sequentially?
#!/bin/expect --
set vm_num [lindex $argv 0]
puts "``````````````````````````````````````````````````````````````"
puts "``````````````````````````````````````````````````````````````&...
1
vote
1
answer
147
views
Running several Gnuplot commands in parallel with Gnu Parallel
I would like to run a list of Gnuplot commands in parallel.
I'm getting an "Unrecognized option" error:
$ ./parallel-plot-sine.sh | parallel -q gnuplot
unrecognized option -e "set ...
- 225
3
votes
0
answers
72
views
2>&1 causes process to block
I am trying to debug what I have done wrong in GNU Parallel (>=20190122).
This runs as expected:
(
echo 1 | parallel-20190122 --pipe -j0 --tee dummy {} ::: {1..1000};
echo Parallel is done here:...
- 32.7k
1
vote
1
answer
217
views
gnu parallel - speed up command agains remote servers that waits
I'm using gnu parallel to generate backups for about ~1500 web sites on pantheon.io, using their terminus CLI. The terminus backup:create command does not finish until a response is received that it ...
- 63
1
vote
1
answer
143
views
Use GNU Parallel when file has a single (long) line
I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv.
This file has 544609657 characters, is about 545 megabytes, all in a single line.
...
1
vote
1
answer
120
views
How to limit the number of tasks to the first arguments in GNU Parallel?
I want to run a series of parallel jobs based on a set of arguments while assigning a second argument. I use the --link option in GNU Parallel as
parallel --jobs 3 --link echo ::: A B C ::: D E F G
A ...
- 1,869
2
votes
2
answers
1k
views
How to run a bash script using threads/parallel
I am running the following bash script as is. I.e. sequentially. Note that the prefetch and fastq-dump are calls for downloading files from a database, and _shrink is a C++ program that I have made to ...
0
votes
1
answer
27
views
Is it possible to continuously feed parallel with more input?
What I want to do is something like this:
inotifywait -m ./input -e close_write -e moved_to |
parallel --max-args 1 "someSlowSingleThreadedTaskThatActsOnThisFile"
I can't imagine that ...
- 142