Questions tagged [gnu-parallel]

GNU parallel is a command line utility to run programs in parallel

Filter by
Sorted by
Tagged with
0 votes
1 answer
41 views

Read file and run parallel operations

what I'm doing here is a bit complex, so hopefully I can explain it well enough so that it makes sense. The tldr version is that I want to read 10 lines of a file and then run a script based on the ...
1 vote
1 answer
61 views

Understanding timestamps in a GNU Parallel --joblog output

Does GNU Parallel start a batch of as many jobs as possible (the number of jobs started being governed by GNU Parallel internals or/and the -j option along with given parameters), and once complete, ...
1 vote
1 answer
71 views

GNU Parallel --joblog logs only first line of commands inside a for loop

I use GNU Parallel along a for loop like this: for BAND in $(seq 1 "$BANDS") ;do # Do not extract, unscale and merge if the scaled map exists already! SCALED_MAP="era5_and_land_${...
2 votes
1 answer
76 views

GNU Parallel with -j -N still uses one CPU

How can I get reasonable parallelisation on multi-core nodes without saturating resources? As in many other similar questions, the question is really how to learn to tweak GNU Parallel to get ...
2 votes
2 answers
86 views

How to compress all files in the current directory using lz4 using GNU parallel?

How to compress all files in the current directory using lz4 using GNU parallel? I try ls | parallel lz4 but it outputs to stdout. Why? How to fix?
  • 447
0 votes
0 answers
23 views

How to use ${LSB_JOB_INDEX} to run a for loop in LSF platform

I have a for loop that looks like this, and I want to convert it to an array job. I want to run 100 jobs in parallel, but I am not entirely sure where I can add the ${LSB_JOB_INDEX} parameter. Any ...
  • 101
1 vote
1 answer
60 views

How do we write only fail jobs in a log when we use GNU parallel

parallel --joblog /tmp/log exit ::: 1 2 3 0 cat /tmp/log Ηow can Ι use a filter to write only failed job in my load when using GNU parallel or is there a way to get only failed jobs from the above ...
3 votes
2 answers
205 views

How to print GNU Parallel command & output together?

I'd like to just check the status of a bunch of Git repos with a quick command like parallel git -C {} status --short ::: ~/*/.git/... But the Git status doesn't include the repo name or path, so I'd ...
  • 48.7k
0 votes
3 answers
97 views

How safe is it to use EXPORT in bash scripts when using GNU Parallel?

How safe is it to use export in bash scripts when using GNU Parallel? I have a parent script. parent.sh (echo child.sh & echo child_two.sh) || parallel bash wait if [[ "$STATUS1" == &...
2 votes
2 answers
178 views

Difference between xargs and GNU parallel

I need to run an executable a large number of times, each time with two command line arguments. I've used to use xargs for this purpose, but lately I've been made aware of the existence of GNU ...
  • 145
1 vote
2 answers
97 views

GNU Parallel with named arguments

I have the following command in my Makefile parallel \ --eta \ --bar \ --joblog mnist/embedder.joblog \ pipenv run python3 \ -m mnist.train_embedder \ --...
  • 369
1 vote
2 answers
91 views

Limit total number of GNU-parallel jobs between mutliple calls

I'm working on a computer with 10k cores, but only allowed to access 1,000 at a time. I have a script that could benefit from GNU parallel in multiple places. Processing level A in parallel, and ...
  • 275
1 vote
1 answer
47 views

Strip leading and trailing whitespace when piping to GNU parallel

When I try to write a pipeline like this: git branch | rg '^\*' | parallel git pull {} I run into a problem with whitespace. Because the branch names have leading whitespace, parallel ends up ...
2 votes
1 answer
186 views

A GNU Parallel Job Queue Script

I found a script on GitHub which I've slightly modified to fit the needs of the program I'm trying to run in a queue. It is not working however and I'm not sure why. It never actually echos the jobs ...
0 votes
0 answers
76 views

How do I make this parallel bash script more efficient?

I have a simple script involving afni command line tools that I am running in parallel from bash, but it is very disk I/O intensive and slows things down on our server (using only 2-3 out of the total ...
1 vote
2 answers
273 views

How might I execute this nested for loop in parallel?

#!/usr/bin/bash TARGETS=( "81.176.235.2" "81.176.70.2" "78.41.109.7" ) myIPs=( "185.164.100.1" "185.164.100.2" "185.164.100.3" "185....
1 vote
1 answer
142 views

GNU parset is not working in a script but works in terminal

I ran into a strange problem.I am running this example from this https://www.gnu.org/software/parallel/parset.html. But it is not working inside the script file. parset myarray seq 3 ::: 4 5 6 echo &...
1 vote
2 answers
259 views

Parallel md5sum on all *.txt files in directory

Trying to calculate md5 hash of all *.txt files in directory and store each in same filename but with extension replaced from .txt to .md5. For file: foo.txt i want to create file: foo.md5 with ...
0 votes
1 answer
189 views

GNU parallel export function output to variables failed

This script is to determine when destination file already exist, source file will update the destination one or be removed according to flag "$dup_act". #!/bin/bash dup_chk() { # $1: f_src,...
2 votes
1 answer
90 views

Adding to while executing a list of commands

I'm compiling a huge list of commands (all doing the same thing) I want executed, but as it takes a long time to compile that list, I would like execution to begin before I'm done (execution of each ...
1 vote
1 answer
55 views

Bash script hangs when filtering stderr through sed

Introduction I have a bash script to execute a command in multiple servers through ssh. It uses GNU parallel in the parallel version, a for loop in the sequential one. The script is used like this: ...
1 vote
2 answers
59 views

Passing second argument to GNU parallel while feeding ls

I process files found by ls as ls /folder/ | parallel -j20 ./command {} but I need to pass the job number as well. I tried ls /folder/ | parallel -j20 ./command {1} {2} ::: {1..20} ls /folder/ | ...
  • 1,839
2 votes
1 answer
668 views

Use GNU parallel to run script with different input arguments

I have a script called step1.sh, and I pass one input parameter to the script ($1). I can run the script like this if I want the input to be 300.1 ./step1.sh 300.1 I want to run this script 200 times,...
  • 21
0 votes
1 answer
258 views

pass variable from shell script to GNU parallel [duplicate]

I am using two shell scripts to set up a parallelization. The first script uses a for loop to pass an environment variable to the second script, which makes the call to parallel, which runs a julia ...
-1 votes
3 answers
270 views

How to concatenate several file with * in the path using xargs or parallel

I have a command that fetch a path to a file for each line in a CSV file : awk -F, 'NR>0 {print "/mnt/datagenetique/ANALYSIS/Infectiologie/COVID-WGS/Analyse/" $2 "_*/dragen-covidseq/&...
  • 509
0 votes
2 answers
373 views

How to find the number of jobs (created by gnu parallel) running in each CPU core at any given time?

I'm using gnu parallel that reads a text file containing curl commands. If I do ps -ef | grep -cw [c]url it shows total number of curl process at a moment. But I want to know number of gnu parallel ...
  • 119
1 vote
3 answers
667 views

GNU Parallel vs. xargs vs. Python tools - Sending 10000 reqs/second [closed]

I have 1000000 different requests (different request I mean it has different query param. It is just a GET request, and no payload. Size of request and response are in order of KBs only. no images or ...
  • 119
2 votes
1 answer
435 views

Make parallel http requests using raw data files

I've several files containg POST body requests. I'd like to send those requests in parallel. Related curl command is like: curl -s -X POST $FHIR_SERVER/ -H "Content-Type: application/fhir+json&...
  • 315
1 vote
2 answers
133 views

Why my gnu parallel with ffmpeg execute ffmpeg only on remote host?

I want to use ffmpeg using two pc, i know parallel can do it I use this cli parallel --trc {.}.mkv -S virtual,: 'ffmpeg -y -i {} -vf yadif,crop=720:550:0:12,scale=640:352 -c:v libx264 -c:a aac -b:v ...
  • 11.5k
1 vote
2 answers
557 views

How to use GNU Parallel for executing a program concurrently?

I need to run a python script several times in parallel but I have done executing it in the background like this ipython program.py & ipython program.py & ... I want to know if this way uses ...
  • 13
2 votes
4 answers
345 views

Parallel and ls with white spaces

This command works fine for dir with "regular" names as dir1 mydir my-dir, etc ls | parallel 'echo -n {}" "; ls {}|wc -l' give me the number of files for each dir but for dir with ...
  • 11.5k
1 vote
2 answers
573 views

How can I use sem to run a shell command in parallel in a for loop?

This isn't speeding anything up for me: domains=() for i in `seq 1 100`; do echo $i word=`sem "xidel -s 'https://www.thisworddoesnotexist.com/' -e '//div[@id="definition-word"]/...
  • 1,941
0 votes
2 answers
94 views

Do any shells generalize the notion of pipes to multiple parallel pipelines? Would supporting it require changes to the OS kernel?

Quite often, when working on the command line, I find myself specifying the same operation on a bunch of different instances specified by some input stream, and then wanting to recombine their outputs ...
2 votes
2 answers
82 views

--skip-first-line in GNU parallel not working with --pipepart?

Suppose I have a file like this: COLUMN 1 2 3 4 If I wanna run and process it with GNU parallel but skipping first line aka header, I tried this: parallel -a test.txt -k --pipepart --will-cite --skip-...
3 votes
3 answers
611 views

Can GNU Parallel be made to output the command line executed when run in linewise mode?

Suppose I have a list of commands in file cmd_file. I run these commands via: cat cmd_file | parallel -k -I {} "{}" One of the commands fails. All of the commands use the exact same CLI ...
  • 759
1 vote
1 answer
121 views

How to parallelize a recursive bash function?

I have a bash function that mainly curls an endpoint for a set of links, and again curls each of those links (for another set of links) recursively. task() { link="$1" response=$(curl &...
1 vote
1 answer
103 views

Is it with GNU parallel possible to split on character count, but provide full lines only?

I would like to split an input file on character count (ASCII is fine), combined with new lines as well. That is, every group of 10000 character should be seen as one record to be piped into the child ...
4 votes
2 answers
347 views

GNU parallel combination of array arguments?

I have a bash function: println(){echo "$1" "$2" "$3"} and export -f println I want to use GNU parallel to run println. The command should take inputs of the following ...
  • 41
4 votes
1 answer
2k views

Running a program with multiple arguments from list using parallel

I have a bash program that I run via command line (Ubuntu) like this: ./extract_field.sh ABC001 where ABC001 is the field ID that I want to extract from a given shapefile. To run this script with ...
2 votes
3 answers
402 views

A universal `while read something` to `parallel` conversion/replacement

I have a ton of scripts which do this: command | while read something; do a long list of commands done Has anyone ever thought of how to run all the commands fed through the pipe ...
3 votes
2 answers
358 views

How can I use basename with parallel?

I have files like this on a Linux system: 10S1_S5_L002_chrm.fasta SRR3184711_chrm.fasta SRR3987378_chrm.fasta SRR4029368_chrm.fasta SRR5204465_chrm.fasta SRR5997546_chrm.fasta ...
  • 55
0 votes
1 answer
486 views

How to run an expect script in parallel rather than sequentially?

#!/bin/expect -- set vm_num [lindex $argv 0] puts "``````````````````````````````````````````````````````````````" puts "``````````````````````````````````````````````````````````````&...
1 vote
1 answer
127 views

Running several Gnuplot commands in parallel with Gnu Parallel

I would like to run a list of Gnuplot commands in parallel. I'm getting an "Unrecognized option" error: $ ./parallel-plot-sine.sh | parallel -q gnuplot unrecognized option -e "set ...
3 votes
0 answers
72 views

2>&1 causes process to block

I am trying to debug what I have done wrong in GNU Parallel (>=20190122). This runs as expected: ( echo 1 | parallel-20190122 --pipe -j0 --tee dummy {} ::: {1..1000}; echo Parallel is done here:...
  • 31.9k
1 vote
1 answer
209 views

gnu parallel - speed up command agains remote servers that waits

I'm using gnu parallel to generate backups for about ~1500 web sites on pantheon.io, using their terminus CLI. The terminus backup:create command does not finish until a response is received that it ...
  • 63
1 vote
1 answer
135 views

Use GNU Parallel when file has a single (long) line

I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv. This file has 544609657 characters, is about 545 megabytes, all in a single line. ...
1 vote
1 answer
108 views

How to limit the number of tasks to the first arguments in GNU Parallel?

I want to run a series of parallel jobs based on a set of arguments while assigning a second argument. I use the --link option in GNU Parallel as parallel --jobs 3 --link echo ::: A B C ::: D E F G A ...
  • 1,839
2 votes
2 answers
1k views

How to run a bash script using threads/parallel

I am running the following bash script as is. I.e. sequentially. Note that the prefetch and fastq-dump are calls for downloading files from a database, and _shrink is a C++ program that I have made to ...
0 votes
1 answer
25 views

Is it possible to continuously feed parallel with more input?

What I want to do is something like this: inotifywait -m ./input -e close_write -e moved_to | parallel --max-args 1 "someSlowSingleThreadedTaskThatActsOnThisFile" I can't imagine that ...
  • 142
2 votes
2 answers
287 views

How to use parallel with inotifywait?

I currently have this: inotifywait -m ./input -e close_write | parallel --max-args 1 "echo {1}" For some reason this doesn't do anything at all. The output of inotifywait is the same as ...
  • 142

1
2 3 4 5