Questions tagged [gnu-parallel]
GNU parallel is a command line utility to run programs in parallel
268
questions
1
vote
1
answer
33
views
How do I ask gnu parallel to preserve stdin?
I'm using GNU parallel like this (:::: is a form of --arg-file):
parallel -0Xuj1 my-command -- :::: <(find … -print0)
But it seems like the command's standard input is managed by GNU parallel, ...
- 6,670
2
votes
1
answer
54
views
Parallel for-loop in bash with simultaneous sequential execution of another task with dependencies on the parallelized loop
In my bash script, I need to execute two different functions, taskA and taskB, which take an integer ($i) as an argument. Since taskB $i depends on the completion of taskA $i, the following ...
- 121
1
vote
1
answer
54
views
GNU parallel: how to call exported bash function with empty string argument?
Scenario:
$ process(){ echo "[$1] [$2] [$3]" ; } ; export -f process
$ process "x" "" "a.txt"
[x] [] [a.txt]
Here we see that the 2nd argument is empty string ...
- 629
2
votes
1
answer
63
views
Use GNU parallel with very long lines
I have a very large SQL dumpfile (30GB) that I need to edit (do some find/replace) before loading back into the database.
Besides having a large size, the file also contains very long lines. Except ...
- 23
0
votes
1
answer
25
views
gnu parallel: how to control output of program?
Fast and simple. This command works
locate -i mymovieormysong|parallel mplayer
the song (or movie) play, but i cannot control mplayer with keyboard.
How to do (if possible) this?
Actually when i use ...
- 12.9k
1
vote
1
answer
48
views
Why does /usr/bin/time coupled with GNU parallel output results before command's output rather than after command's output?
Scenario:
$ cat libs.txt
lib.a
lib1.a
$ cat t1a.sh
f1()
{
local lib=$1
stdbuf -o0 printf "job for $lib started\n"
sleep 2
stdbuf -o0 printf "job for $...
- 629
2
votes
1
answer
46
views
GNU parallel: why does diagnostic output look like sequential execution rather than parallel execution?
Scenario:
$ cat libs.txt
lib.a
lib1.a
$ cat t1a.sh
f1()
{
local lib=$1
stdbuf -o0 printf "job for $lib started\n"
sleep 2
stdbuf -o0 printf "job for $...
- 629
4
votes
1
answer
436
views
Download files in parallel with wget over a single connection
This question is a follow-up to How to download multiple files simultaniously with wget?
Similar to that question, I need to download many files. With the accepted answer above recommends the use of ...
- 41
1
vote
2
answers
317
views
How can I multithreadedly extract several 7z archives into their own folder, using the archive name as the folder name?
To extract all 7z archives into their own folder, using the archive name as the folder name, I run the command:
7z x "*.7z" -o*
E.g., if one has the two files a.7z and b.7z, then folders a ...
- 5,205
0
votes
3
answers
105
views
Problem in writing a parallel version of my Bash code
I am trying to parallelise my sample Bash script and have tried commands like & and wait. Please let me know what is an effective way to make it parallel
My current code is working fine for ...
- 11
2
votes
1
answer
97
views
Can I use GNU parallel to group command arguments by a column value?
Consider the data from the GNU parallel manual's example for --group-by:
cat > table.csv <<"EOF"
UserID, Consumption
123, 1
123, 2
12-3, 1
221, 3
221, 1
2/21, 5
EOF
...
1
vote
2
answers
147
views
Use each input argument in GNU Parallel as a working directory
I'm trying to use GNU Parallel to run a command for each input argument, using that argument as the command's working directory (without appending it to the command line).
Basically, what I need to do ...
- 5,506
4
votes
2
answers
321
views
Using GNU parallel on line based output without temporary files
The default output mode of GNU parallel is --group:
The output of each job is
written to a temporary file and passed to the output of parallel only after
the job has finished.
When using this default ...
- 744
5
votes
1
answer
806
views
Using parallel to sg_format 300+ drives
Some time ago I learned about the software parallel: https://forums.servethehome.com/index.php?threads/mdadm-create-raid-0-quick-format.41161/#post-389210
Now I have almost the same goal as previous, ...
- 53
0
votes
3
answers
203
views
RHEL7; split a file into equal parts in parallel
Is it possible to run the linux split command with GNU parallel to speed up the splitting of the file?
Read a compressed file, and split it into some equal parts either by line count or filesize.
I'm ...
- 113
1
vote
2
answers
77
views
GNU Parallel: Run while partial files available and wait for the rest
I have a process like this that generates a predefined number of files but at random intervals:
#!/bin/bash
for i in {1..10}
do
sleep $(shuf -i 20-60 -n 1)
echo $i > file_$i.txt
done
I have ...
- 9,306
2
votes
2
answers
91
views
Arithmetic with GNU parallel
I want to run some script over powers of two in parallel. Doing so by giving GNU Parallel a list of the powers of two I want works well:
%>parallel echo {} ::: 32, 64, 128, 256, 512, 1024
32
64
128
...
- 23
2
votes
1
answer
232
views
Piping xargs into GNU parallel
I have the following script which gets the absolute paths of some directories and pipes them into GNU parallel for zipping. I keep getting a signal 13 error and I'm not sure why.
find $directory -...
- 33
0
votes
3
answers
144
views
Unable to append to array using parallel
I can't append to an array when I use parallel, no issues using a for loop.
Parallel example:
append() { arr+=("$1"); }
export -f append
parallel -j 0 append ::: {1..4}
declare -p arr
...
- 489
1
vote
1
answer
59
views
Parallel in Parallel
I'm trying to wrap my head around running parallel in parallel, and it seems like I have a situation where that would be an ideal solution.
I want to run a set of jobs in series - call them A-1, A-2, ...
- 140
2
votes
0
answers
101
views
Processing large blocklists with GNU Parallel
This is an ongoing project that has undergone a major update to speed up list processing. Here is the homepage for anyone interested!
It processes lists defined in lists.json by transforming the list ...
- 213
1
vote
2
answers
67
views
Passing parameters to GNU Parallel manually and by piping
To set the stage, here is an example file structure:
test/1
test/2
test/3
I want to do this:
find -P -O3 "test/" -type f |
parallel --use-cpus-instead-of-cores -j+0 --tmpdir "test/&...
- 213
3
votes
2
answers
409
views
Iterating over array elements with gnu parallel
I have an input file, names.txt, with the 1 word per line:
apple
abble
aplle
With my bash script I am trying to achieve the following output:
apple and apple
apple and abble
apple and aplle ...
- 33
1
vote
1
answer
213
views
Run one-line programs in parallel on separate lines
So, let's say I have an array with a list of URLs, and I want to use something such as GNU parallel to download the URLs in parallel. A command like this would do the trick.
parallel -u wget -qc --...
- 151
1
vote
1
answer
78
views
Why the cpu usage of one process decreased when multiple processes are running?
I'm using gnu parallel like below:
find conf/20230214/ -name "conf.yaml" | parallel run_backtest
the run_backtest is a function which will read the conf.yaml and run some code.
When there ...
- 11
0
votes
1
answer
661
views
Read file and run parallel operations
what I'm doing here is a bit complex, so hopefully I can explain it well enough so that it makes sense. The tldr version is that I want to read 10 lines of a file and then run a script based on the ...
- 201
1
vote
1
answer
745
views
Understanding timestamps in a GNU Parallel --joblog output
Does GNU Parallel start a batch of as many jobs as possible (the number of jobs started being governed by GNU Parallel internals or/and the -j option along with given parameters), and once complete, ...
- 1,490
1
vote
1
answer
298
views
GNU Parallel --joblog logs only first line of commands inside a for loop
I use GNU Parallel along a for loop like this:
for BAND in $(seq 1 "$BANDS") ;do
# Do not extract, unscale and merge if the scaled map exists already!
SCALED_MAP="era5_and_land_${...
- 1,490
2
votes
1
answer
1k
views
GNU Parallel with -j -N still uses one CPU
How can I get reasonable parallelisation on multi-core nodes without saturating resources? As in many other similar questions, the question is really how to learn to tweak GNU Parallel to get ...
- 1,490
2
votes
2
answers
669
views
How to compress all files in the current directory using lz4 using GNU parallel?
How to compress all files in the current directory using lz4 using GNU parallel?
I try ls | parallel lz4 but it outputs to stdout. Why? How to fix?
- 629
1
vote
1
answer
427
views
How do we write only fail jobs in a log when we use GNU parallel
parallel --joblog /tmp/log exit ::: 1 2 3 0
cat /tmp/log
Ηow can Ι use a filter to write only failed job in my load when using GNU parallel or is there a way to get only failed jobs from the above ...
3
votes
2
answers
1k
views
How to print GNU Parallel command & output together?
I'd like to just check the status of a bunch of Git repos with a quick command like parallel git -C {} status --short ::: ~/*/.git/... But the Git status doesn't include the repo name or path, so I'd ...
- 52.1k
0
votes
3
answers
231
views
How safe is it to use EXPORT in bash scripts when using GNU Parallel?
How safe is it to use export in bash scripts when using GNU Parallel?
I have a parent script.
parent.sh
(echo child.sh & echo child_two.sh) || parallel bash
wait
if [[ "$STATUS1" == &...
2
votes
2
answers
1k
views
Difference between xargs and GNU parallel
I need to run an executable a large number of times, each time with two command line arguments. I've used to use xargs for this purpose, but lately I've been made aware of the existence of GNU ...
- 145
1
vote
2
answers
446
views
GNU Parallel with named arguments
I have the following command in my Makefile
parallel \
--eta \
--bar \
--joblog mnist/embedder.joblog \
pipenv run python3 \
-m mnist.train_embedder \
--...
- 421
1
vote
2
answers
323
views
Limit total number of GNU-parallel jobs between mutliple calls
I'm working on a computer with 10k cores, but only allowed to access 1,000 at a time.
I have a script that could benefit from GNU parallel in multiple places. Processing level A in parallel, and ...
- 285
1
vote
1
answer
163
views
Strip leading and trailing whitespace when piping to GNU parallel
When I try to write a pipeline like this:
git branch | rg '^\*' | parallel git pull {}
I run into a problem with whitespace. Because the branch names have leading whitespace, parallel ends up ...
- 133
3
votes
1
answer
853
views
A GNU Parallel Job Queue Script
I found a script on GitHub which I've slightly modified to fit the needs of the program I'm trying to run in a queue.
It is not working however and I'm not sure why. It never actually echos the jobs ...
user474102
0
votes
0
answers
119
views
How do I make this parallel bash script more efficient?
I have a simple script involving afni command line tools that I am running in parallel from bash, but it is very disk I/O intensive and slows things down on our server (using only 2-3 out of the total ...
1
vote
2
answers
1k
views
How might I execute this nested for loop in parallel?
#!/usr/bin/bash
TARGETS=(
"81.176.235.2"
"81.176.70.2"
"78.41.109.7"
)
myIPs=(
"185.164.100.1"
"185.164.100.2"
"185.164.100.3"
"185....
- 785
1
vote
1
answer
317
views
GNU parset is not working in a script but works in terminal
I ran into a strange problem.I am running this example from this https://www.gnu.org/software/parallel/parset.html. But it is not working inside the script file.
parset myarray seq 3 ::: 4 5 6
echo &...
1
vote
2
answers
596
views
Parallel md5sum on all *.txt files in directory
Trying to calculate md5 hash of all *.txt files in directory and store each in same filename but with extension replaced from .txt to .md5.
For file: foo.txt
i want to create file: foo.md5
with ...
- 131
0
votes
1
answer
393
views
GNU parallel export function output to variables failed
This script is to determine when destination file already exist, source file will update the destination one or be removed according to flag "$dup_act".
#!/bin/bash
dup_chk()
{
# $1: f_src,...
3
votes
1
answer
220
views
Adding to while executing a list of commands
I'm compiling a huge list of commands (all doing the same thing) I want executed, but as it takes a long time to compile that list, I would like execution to begin before I'm done (execution of each ...
1
vote
1
answer
132
views
Bash script hangs when filtering stderr through sed
Introduction
I have a bash script to execute a command in multiple servers through ssh. It uses GNU parallel in the parallel version, a for loop in the sequential one.
The script is used like this:
...
1
vote
2
answers
132
views
Passing second argument to GNU parallel while feeding ls
I process files found by ls as
ls /folder/ | parallel -j20 ./command {}
but I need to pass the job number as well. I tried
ls /folder/ | parallel -j20 ./command {1} {2} ::: {1..20}
ls /folder/ | ...
- 1,959
3
votes
1
answer
2k
views
Use GNU parallel to run script with different input arguments
I have a script called step1.sh, and I pass one input parameter to the script ($1).
I can run the script like this if I want the input to be 300.1
./step1.sh 300.1
I want to run this script 200 times,...
- 31
0
votes
1
answer
798
views
pass variable from shell script to GNU parallel [duplicate]
I am using two shell scripts to set up a parallelization. The first script uses a for loop to pass an environment variable to the second script, which makes the call to parallel, which runs a julia ...
- 101
-1
votes
3
answers
451
views
How to concatenate several file with * in the path using xargs or parallel
I have a command that fetch a path to a file for each line in a CSV file :
awk -F, 'NR>0 {print "/mnt/datagenetique/ANALYSIS/Infectiologie/COVID-WGS/Analyse/" $2 "_*/dragen-covidseq/&...
- 529
0
votes
2
answers
804
views
How to find the number of jobs (created by gnu parallel) running in each CPU core at any given time?
I'm using gnu parallel that reads a text file containing curl commands.
If I do ps -ef | grep -cw [c]url it shows total number of curl process at a moment.
But I want to know number of gnu parallel ...
- 119