Questions tagged [gnu-parallel]

GNU parallel is a command line utility to run programs in parallel

Filter by
Sorted by
Tagged with
1 vote
1 answer
35 views

GNU parallel: how to call exported bash function with empty string argument?

Scenario: $ process(){ echo "[$1] [$2] [$3]" ; } ; export -f process $ process "x" "" "a.txt" [x] [] [a.txt] Here we see that the 2nd argument is empty string ...
pmor's user avatar
  • 609
2 votes
1 answer
37 views

Use GNU parallel with very long lines

I have a very large SQL dumpfile (30GB) that I need to edit (do some find/replace) before loading back into the database. Besides having a large size, the file also contains very long lines. Except ...
fernan's user avatar
  • 23
0 votes
1 answer
22 views

gnu parallel: how to control output of program?

Fast and simple. This command works locate -i mymovieormysong|parallel mplayer the song (or movie) play, but i cannot control mplayer with keyboard. How to do (if possible) this? Actually when i use ...
elbarna's user avatar
  • 12.7k
1 vote
1 answer
42 views

Why does /usr/bin/time coupled with GNU parallel output results before command's output rather than after command's output?

Scenario: $ cat libs.txt lib.a lib1.a $ cat t1a.sh f1() { local lib=$1 stdbuf -o0 printf "job for $lib started\n" sleep 2 stdbuf -o0 printf "job for $...
pmor's user avatar
  • 609
2 votes
1 answer
46 views

GNU parallel: why does diagnostic output look like sequential execution rather than parallel execution?

Scenario: $ cat libs.txt lib.a lib1.a $ cat t1a.sh f1() { local lib=$1 stdbuf -o0 printf "job for $lib started\n" sleep 2 stdbuf -o0 printf "job for $...
pmor's user avatar
  • 609
4 votes
1 answer
144 views

Download files in parallel with wget over a single connection

This question is a follow-up to How to download multiple files simultaniously with wget? Similar to that question, I need to download many files. With the accepted answer above recommends the use of ...
sneumann's user avatar
1 vote
2 answers
108 views

How can I multithreadedly extract several 7z archives into their own folder, using the archive name as the folder name?

To extract all 7z archives into their own folder, using the archive name as the folder name, I run the command: 7z x "*.7z" -o* E.g., if one has the two files a.7z and b.7z, then folders a ...
Franck Dernoncourt's user avatar
0 votes
3 answers
100 views

Problem in writing a parallel version of my Bash code

I am trying to parallelise my sample Bash script and have tried commands like & and wait. Please let me know what is an effective way to make it parallel My current code is working fine for ...
user96368's user avatar
2 votes
1 answer
71 views

Can I use GNU parallel to group command arguments by a column value?

Consider the data from the GNU parallel manual's example for --group-by: cat > table.csv <<"EOF" UserID, Consumption 123, 1 123, 2 12-3, 1 221, 3 221, 1 2/21, 5 EOF ...
Iain Samuel McLean Elder's user avatar
1 vote
2 answers
92 views

Use each input argument in GNU Parallel as a working directory

I'm trying to use GNU Parallel to run a command for each input argument, using that argument as the command's working directory (without appending it to the command line). Basically, what I need to do ...
intelfx's user avatar
  • 5,375
4 votes
2 answers
212 views

Using GNU parallel on line based output without temporary files

The default output mode of GNU parallel is --group: The output of each job is written to a temporary file and passed to the output of parallel only after the job has finished. When using this default ...
Juergen's user avatar
  • 674
5 votes
1 answer
626 views

Using parallel to sg_format 300+ drives

Some time ago I learned about the software parallel: https://forums.servethehome.com/index.php?threads/mdadm-create-raid-0-quick-format.41161/#post-389210 Now I have almost the same goal as previous, ...
Jelle's user avatar
  • 53
0 votes
3 answers
141 views

RHEL7; split a file into equal parts in parallel

Is it possible to run the linux split command with GNU parallel to speed up the splitting of the file? Read a compressed file, and split it into some equal parts either by line count or filesize. I'm ...
Abhishek Jain's user avatar
1 vote
2 answers
67 views

GNU Parallel: Run while partial files available and wait for the rest

I have a process like this that generates a predefined number of files but at random intervals: #!/bin/bash for i in {1..10} do sleep $(shuf -i 20-60 -n 1) echo $i > file_$i.txt done I have ...
Ketan's user avatar
  • 9,226
2 votes
2 answers
75 views

Arithmetic with GNU parallel

I want to run some script over powers of two in parallel. Doing so by giving GNU Parallel a list of the powers of two I want works well: %>parallel echo {} ::: 32, 64, 128, 256, 512, 1024 32 64 128 ...
Will's user avatar
  • 23
2 votes
1 answer
194 views

Piping xargs into GNU parallel

I have the following script which gets the absolute paths of some directories and pipes them into GNU parallel for zipping. I keep getting a signal 13 error and I'm not sure why. find $directory -...
testing09's user avatar
0 votes
3 answers
123 views

Unable to append to array using parallel

I can't append to an array when I use parallel, no issues using a for loop. Parallel example: append() { arr+=("$1"); } export -f append parallel -j 0 append ::: {1..4} declare -p arr ...
Nickotine's user avatar
  • 467
1 vote
1 answer
56 views

Parallel in Parallel

I'm trying to wrap my head around running parallel in parallel, and it seems like I have a situation where that would be an ideal solution. I want to run a set of jobs in series - call them A-1, A-2, ...
Larry's user avatar
  • 141
2 votes
0 answers
99 views

Processing large blocklists with GNU Parallel

This is an ongoing project that has undergone a major update to speed up list processing. Here is the homepage for anyone interested! It processes lists defined in lists.json by transforming the list ...
T145's user avatar
  • 131
1 vote
2 answers
49 views

Passing parameters to GNU Parallel manually and by piping

To set the stage, here is an example file structure: test/1 test/2 test/3 I want to do this: find -P -O3 "test/" -type f | parallel --use-cpus-instead-of-cores -j+0 --tmpdir "test/&...
T145's user avatar
  • 131
3 votes
2 answers
259 views

Iterating over array elements with gnu parallel

I have an input file, names.txt, with the 1 word per line: apple abble aplle With my bash script I am trying to achieve the following output: apple and apple apple and abble apple and aplle ...
duda13's user avatar
  • 33
1 vote
1 answer
157 views

Run one-line programs in parallel on separate lines

So, let's say I have an array with a list of URLs, and I want to use something such as GNU parallel to download the URLs in parallel. A command like this would do the trick. parallel -u wget -qc --...
Eduardo Perez's user avatar
1 vote
1 answer
75 views

Why the cpu usage of one process decreased when multiple processes are running?

I'm using gnu parallel like below: find conf/20230214/ -name "conf.yaml" | parallel run_backtest the run_backtest is a function which will read the conf.yaml and run some code. When there ...
user560657's user avatar
0 votes
1 answer
465 views

Read file and run parallel operations

what I'm doing here is a bit complex, so hopefully I can explain it well enough so that it makes sense. The tldr version is that I want to read 10 lines of a file and then run a script based on the ...
elmerjfudd's user avatar
1 vote
1 answer
590 views

Understanding timestamps in a GNU Parallel --joblog output

Does GNU Parallel start a batch of as many jobs as possible (the number of jobs started being governed by GNU Parallel internals or/and the -j option along with given parameters), and once complete, ...
Nikos Alexandris's user avatar
1 vote
1 answer
253 views

GNU Parallel --joblog logs only first line of commands inside a for loop

I use GNU Parallel along a for loop like this: for BAND in $(seq 1 "$BANDS") ;do # Do not extract, unscale and merge if the scaled map exists already! SCALED_MAP="era5_and_land_${...
Nikos Alexandris's user avatar
2 votes
1 answer
973 views

GNU Parallel with -j -N still uses one CPU

How can I get reasonable parallelisation on multi-core nodes without saturating resources? As in many other similar questions, the question is really how to learn to tweak GNU Parallel to get ...
Nikos Alexandris's user avatar
2 votes
2 answers
571 views

How to compress all files in the current directory using lz4 using GNU parallel?

How to compress all files in the current directory using lz4 using GNU parallel? I try ls | parallel lz4 but it outputs to stdout. Why? How to fix?
pmor's user avatar
  • 609
1 vote
1 answer
337 views

How do we write only fail jobs in a log when we use GNU parallel

parallel --joblog /tmp/log exit ::: 1 2 3 0 cat /tmp/log Ηow can Ι use a filter to write only failed job in my load when using GNU parallel or is there a way to get only failed jobs from the above ...
lc wanigasinghe's user avatar
3 votes
2 answers
966 views

How to print GNU Parallel command & output together?

I'd like to just check the status of a bunch of Git repos with a quick command like parallel git -C {} status --short ::: ~/*/.git/... But the Git status doesn't include the repo name or path, so I'd ...
l0b0's user avatar
  • 51.5k
0 votes
3 answers
192 views

How safe is it to use EXPORT in bash scripts when using GNU Parallel?

How safe is it to use export in bash scripts when using GNU Parallel? I have a parent script. parent.sh (echo child.sh & echo child_two.sh) || parallel bash wait if [[ "$STATUS1" == &...
lc wanigasinghe's user avatar
2 votes
2 answers
1k views

Difference between xargs and GNU parallel

I need to run an executable a large number of times, each time with two command line arguments. I've used to use xargs for this purpose, but lately I've been made aware of the existence of GNU ...
Tor's user avatar
  • 145
1 vote
2 answers
408 views

GNU Parallel with named arguments

I have the following command in my Makefile parallel \ --eta \ --bar \ --joblog mnist/embedder.joblog \ pipenv run python3 \ -m mnist.train_embedder \ --...
nalzok's user avatar
  • 369
1 vote
2 answers
235 views

Limit total number of GNU-parallel jobs between mutliple calls

I'm working on a computer with 10k cores, but only allowed to access 1,000 at a time. I have a script that could benefit from GNU parallel in multiple places. Processing level A in parallel, and ...
mankoff's user avatar
  • 285
1 vote
1 answer
127 views

Strip leading and trailing whitespace when piping to GNU parallel

When I try to write a pipeline like this: git branch | rg '^\*' | parallel git pull {} I run into a problem with whitespace. Because the branch names have leading whitespace, parallel ends up ...
Dommondke's user avatar
  • 133
3 votes
1 answer
752 views

A GNU Parallel Job Queue Script

I found a script on GitHub which I've slightly modified to fit the needs of the program I'm trying to run in a queue. It is not working however and I'm not sure why. It never actually echos the jobs ...
user avatar
0 votes
0 answers
109 views

How do I make this parallel bash script more efficient?

I have a simple script involving afni command line tools that I am running in parallel from bash, but it is very disk I/O intensive and slows things down on our server (using only 2-3 out of the total ...
mackenzie taylor's user avatar
1 vote
2 answers
1k views

How might I execute this nested for loop in parallel?

#!/usr/bin/bash TARGETS=( "81.176.235.2" "81.176.70.2" "78.41.109.7" ) myIPs=( "185.164.100.1" "185.164.100.2" "185.164.100.3" "185....
John Smith's user avatar
1 vote
1 answer
294 views

GNU parset is not working in a script but works in terminal

I ran into a strange problem.I am running this example from this https://www.gnu.org/software/parallel/parset.html. But it is not working inside the script file. parset myarray seq 3 ::: 4 5 6 echo &...
Sanchu Varkey's user avatar
1 vote
2 answers
532 views

Parallel md5sum on all *.txt files in directory

Trying to calculate md5 hash of all *.txt files in directory and store each in same filename but with extension replaced from .txt to .md5. For file: foo.txt i want to create file: foo.md5 with ...
watbywbarif's user avatar
0 votes
1 answer
380 views

GNU parallel export function output to variables failed

This script is to determine when destination file already exist, source file will update the destination one or be removed according to flag "$dup_act". #!/bin/bash dup_chk() { # $1: f_src,...
chuck lee's user avatar
3 votes
1 answer
199 views

Adding to while executing a list of commands

I'm compiling a huge list of commands (all doing the same thing) I want executed, but as it takes a long time to compile that list, I would like execution to begin before I'm done (execution of each ...
Henrik supports the community's user avatar
1 vote
1 answer
115 views

Bash script hangs when filtering stderr through sed

Introduction I have a bash script to execute a command in multiple servers through ssh. It uses GNU parallel in the parallel version, a for loop in the sequential one. The script is used like this: ...
IvanProsperi94's user avatar
1 vote
2 answers
126 views

Passing second argument to GNU parallel while feeding ls

I process files found by ls as ls /folder/ | parallel -j20 ./command {} but I need to pass the job number as well. I tried ls /folder/ | parallel -j20 ./command {1} {2} ::: {1..20} ls /folder/ | ...
Googlebot's user avatar
  • 1,959
3 votes
1 answer
2k views

Use GNU parallel to run script with different input arguments

I have a script called step1.sh, and I pass one input parameter to the script ($1). I can run the script like this if I want the input to be 300.1 ./step1.sh 300.1 I want to run this script 200 times,...
adann's user avatar
  • 31
0 votes
1 answer
718 views

pass variable from shell script to GNU parallel [duplicate]

I am using two shell scripts to set up a parallelization. The first script uses a for loop to pass an environment variable to the second script, which makes the call to parallel, which runs a julia ...
colebrookson's user avatar
-1 votes
3 answers
432 views

How to concatenate several file with * in the path using xargs or parallel

I have a command that fetch a path to a file for each line in a CSV file : awk -F, 'NR>0 {print "/mnt/datagenetique/ANALYSIS/Infectiologie/COVID-WGS/Analyse/" $2 "_*/dragen-covidseq/&...
nstatam's user avatar
  • 519
0 votes
2 answers
703 views

How to find the number of jobs (created by gnu parallel) running in each CPU core at any given time?

I'm using gnu parallel that reads a text file containing curl commands. If I do ps -ef | grep -cw [c]url it shows total number of curl process at a moment. But I want to know number of gnu parallel ...
sofs1's user avatar
  • 119
1 vote
3 answers
1k views

GNU Parallel vs. xargs vs. Python tools - Sending 10000 reqs/second [closed]

I have 1000000 different requests (different request I mean it has different query param. It is just a GET request, and no payload. Size of request and response are in order of KBs only. no images or ...
sofs1's user avatar
  • 119
2 votes
1 answer
717 views

Make parallel http requests using raw data files

I've several files containg POST body requests. I'd like to send those requests in parallel. Related curl command is like: curl -s -X POST $FHIR_SERVER/ -H "Content-Type: application/fhir+json&...
Jordi's user avatar
  • 325

1
2 3 4 5 6