Questions tagged [gnu-parallel]

GNU parallel is a command line utility to run programs in parallel

Filter by
Sorted by
Tagged with
1 vote
1 answer
63 views

GNU parset is not working in a script but works in terminal

I ran into a strange problem.I am running this example from this https://www.gnu.org/software/parallel/parset.html. But it is not working inside the script file. parset myarray seq 3 ::: 4 5 6 echo &...
user avatar
1 vote
2 answers
39 views

Parallel md5sum on all *.txt files in directory

Trying to calculate md5 hash of all *.txt files in directory and store each in same filename but with extension replaced from .txt to .md5. For file: foo.txt i want to create file: foo.md5 with ...
user avatar
0 votes
1 answer
28 views

GNU parallel export function output to variables failed

This script is to determine when destination file already exist, source file will update the destination one or be removed according to flag "$dup_act". #!/bin/bash dup_chk() { # $1: f_src,...
user avatar
2 votes
1 answer
48 views

Adding to while executing a list of commands

I'm compiling a huge list of commands (all doing the same thing) I want executed, but as it takes a long time to compile that list, I would like execution to begin before I'm done (execution of each ...
user avatar
1 vote
1 answer
34 views

Bash script hangs when filtering stderr through sed

Introduction I have a bash script to execute a command in multiple servers through ssh. It uses GNU parallel in the parallel version, a for loop in the sequential one. The script is used like this: ...
user avatar
1 vote
2 answers
38 views

Passing second argument to GNU parallel while feeding ls

I process files found by ls as ls /folder/ | parallel -j20 ./command {} but I need to pass the job number as well. I tried ls /folder/ | parallel -j20 ./command {1} {2} ::: {1..20} ls /folder/ | ...
user avatar
  • 1,663
1 vote
1 answer
77 views

Use GNU parallel to run script with different input arguments

I have a script called step1.sh, and I pass one input parameter to the script ($1). I can run the script like this if I want the input to be 300.1 ./step1.sh 300.1 I want to run this script 200 times,...
user avatar
  • 11
0 votes
1 answer
20 views

pass variable from shell script to GNU parallel [duplicate]

I am using two shell scripts to set up a parallelization. The first script uses a for loop to pass an environment variable to the second script, which makes the call to parallel, which runs a julia ...
user avatar
-1 votes
3 answers
90 views

How to concatenate several file with * in the path using xargs or parallel

I have a command that fetch a path to a file for each line in a CSV file : awk -F, 'NR>0 {print "/mnt/datagenetique/ANALYSIS/Infectiologie/COVID-WGS/Analyse/" $2 "_*/dragen-covidseq/&...
user avatar
  • 499
0 votes
2 answers
86 views

How to find the number of jobs (created by gnu parallel) running in each CPU core at any given time?

I'm using gnu parallel that reads a text file containing curl commands. If I do ps -ef | grep -cw [c]url it shows total number of curl process at a moment. But I want to know number of gnu parallel ...
user avatar
  • 119
1 vote
3 answers
224 views

GNU Parallel vs. xargs vs. Python tools - Sending 10000 reqs/second [closed]

I have 1000000 different requests (different request I mean it has different query param. It is just a GET request, and no payload. Size of request and response are in order of KBs only. no images or ...
user avatar
  • 119
1 vote
1 answer
87 views

Make parallel http requests using raw data files

I've several files containg POST body requests. I'd like to send those requests in parallel. Related curl command is like: curl -s -X POST $FHIR_SERVER/ -H "Content-Type: application/fhir+json&...
user avatar
  • 275
1 vote
2 answers
65 views

Why my gnu parallel with ffmpeg execute ffmpeg only on remote host?

I want to use ffmpeg using two pc, i know parallel can do it I use this cli parallel --trc {.}.mkv -S virtual,: 'ffmpeg -y -i {} -vf yadif,crop=720:550:0:12,scale=640:352 -c:v libx264 -c:a aac -b:v ...
user avatar
  • 10.9k
1 vote
2 answers
94 views

How to use GNU Parallel for executing a program concurrently?

I need to run a python script several times in parallel but I have done executing it in the background like this ipython program.py & ipython program.py & ... I want to know if this way uses ...
user avatar
  • 13
1 vote
4 answers
133 views

Parallel and ls with white spaces

This command works fine for dir with "regular" names as dir1 mydir my-dir, etc ls | parallel 'echo -n {}" "; ls {}|wc -l' give me the number of files for each dir but for dir with ...
user avatar
  • 10.9k
1 vote
2 answers
246 views

How can I use sem to run a shell command in parallel in a for loop?

This isn't speeding anything up for me: domains=() for i in `seq 1 100`; do echo $i word=`sem "xidel -s 'https://www.thisworddoesnotexist.com/' -e '//div[@id="definition-word"]/...
user avatar
  • 1,841
-1 votes
2 answers
78 views

Do any shells generalize the notion of pipes to multiple parallel pipelines? Would supporting it require changes to the OS kernel?

Quite often, when working on the command line, I find myself specifying the same operation on a bunch of different instances specified by some input stream, and then wanting to recombine their outputs ...
user avatar
2 votes
2 answers
40 views

--skip-first-line in GNU parallel not working with --pipepart?

Suppose I have a file like this: COLUMN 1 2 3 4 If I wanna run and process it with GNU parallel but skipping first line aka header, I tried this: parallel -a test.txt -k --pipepart --will-cite --skip-...
user avatar
2 votes
2 answers
137 views

Can GNU Parallel be made to output the command line executed when run in linewise mode?

Suppose I have a list of commands in file cmd_file. I run these commands via: cat cmd_file | parallel -k -I {} "{}" One of the commands fails. All of the commands use the exact same CLI ...
user avatar
  • 653
1 vote
1 answer
84 views

How to parallelize a recursive bash function?

I have a bash function that mainly curls an endpoint for a set of links, and again curls each of those links (for another set of links) recursively. task() { link="$1" response=$(curl &...
user avatar
1 vote
1 answer
82 views

Is it with GNU parallel possible to split on character count, but provide full lines only?

I would like to split an input file on character count (ASCII is fine), combined with new lines as well. That is, every group of 10000 character should be seen as one record to be piped into the child ...
user avatar
4 votes
2 answers
240 views

GNU parallel combination of array arguments?

I have a bash function: println(){echo "$1" "$2" "$3"} and export -f println I want to use GNU parallel to run println. The command should take inputs of the following ...
user avatar
3 votes
1 answer
440 views

Running a program with multiple arguments from list using parallel

I have a bash program that I run via command line (Ubuntu) like this: ./extract_field.sh ABC001 where ABC001 is the field ID that I want to extract from a given shapefile. To run this script with ...
user avatar
2 votes
3 answers
160 views

A universal `while read something` to `parallel` conversion/replacement

I have a ton of scripts which do this: command | while read something; do a long list of commands done Has anyone ever thought of how to run all the commands fed through the pipe ...
user avatar
3 votes
2 answers
255 views

How can I use basename with parallel?

I have files like this on a Linux system: 10S1_S5_L002_chrm.fasta SRR3184711_chrm.fasta SRR3987378_chrm.fasta SRR4029368_chrm.fasta SRR5204465_chrm.fasta SRR5997546_chrm.fasta ...
user avatar
  • 55
0 votes
1 answer
217 views

How to run an expect script in parallel rather than sequentially?

#!/bin/expect -- set vm_num [lindex $argv 0] puts "``````````````````````````````````````````````````````````````" puts "``````````````````````````````````````````````````````````````&...
user avatar
1 vote
1 answer
75 views

Running several Gnuplot commands in parallel with Gnu Parallel

I would like to run a list of Gnuplot commands in parallel. I'm getting an "Unrecognized option" error: $ ./parallel-plot-sine.sh | parallel -q gnuplot unrecognized option -e "set ...
user avatar
3 votes
0 answers
71 views

2>&1 causes process to block

I am trying to debug what I have done wrong in GNU Parallel (>=20190122). This runs as expected: ( echo 1 | parallel-20190122 --pipe -j0 --tee dummy {} ::: {1..1000}; echo Parallel is done here:...
user avatar
  • 29.5k
1 vote
1 answer
119 views

gnu parallel - speed up command agains remote servers that waits

I'm using gnu parallel to generate backups for about ~1500 web sites on pantheon.io, using their terminus CLI. The terminus backup:create command does not finish until a response is received that it ...
user avatar
  • 63
1 vote
1 answer
84 views

Use GNU Parallel when file has a single (long) line

I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv. This file has 544609657 characters, is about 545 megabytes, all in a single line. ...
user avatar
1 vote
1 answer
49 views

How to limit the number of tasks to the first arguments in GNU Parallel?

I want to run a series of parallel jobs based on a set of arguments while assigning a second argument. I use the --link option in GNU Parallel as parallel --jobs 3 --link echo ::: A B C ::: D E F G A ...
user avatar
  • 1,663
2 votes
2 answers
570 views

How to run a bash script using threads/parallel

I am running the following bash script as is. I.e. sequentially. Note that the prefetch and fastq-dump are calls for downloading files from a database, and _shrink is a C++ program that I have made to ...
user avatar
0 votes
1 answer
20 views

Is it possible to continuously feed parallel with more input?

What I want to do is something like this: inotifywait -m ./input -e close_write -e moved_to | parallel --max-args 1 "someSlowSingleThreadedTaskThatActsOnThisFile" I can't imagine that ...
user avatar
  • 142
2 votes
2 answers
176 views

How to use parallel with inotifywait?

I currently have this: inotifywait -m ./input -e close_write | parallel --max-args 1 "echo {1}" For some reason this doesn't do anything at all. The output of inotifywait is the same as ...
user avatar
  • 142
0 votes
1 answer
114 views

Use parallel to speed up a recursive shell max function?

I needed a max function, to find the highest value in an array (of 350 int values), and being a friend of recursion, I tried a recursive approach: max () { len=$# a=$1 b=$2 case $len ...
user avatar
  • 9,752
1 vote
1 answer
107 views

convert each line in a file into an input file

I am trying to convert/use each line from a file as an input file. For example file.txt contains the following lines: cat dog lion tiger rabbit Now, using the following command: cat file.txt | ...
user avatar
1 vote
1 answer
33 views

parallel compare two arguments and send to command

I am trying to run recon-all command with GNU parallel. It should take one argument at {1} like this: parallel --progress subj_id='echo {1} | cut -d / -f 3' && recon-all -s $subj_id -i {1} -sd ...
user avatar
3 votes
3 answers
267 views

Boosting the grep search using GNU parallel

I am using the following grep script to output all the unmatched patterns: grep -oFf patterns.txt large_strings.txt | grep -vFf - patterns.txt > unmatched_patterns.txt patterns file contains the ...
user avatar
1 vote
1 answer
154 views

gnu parallel: Warning: This job was killed because it timed out

I am running multiple commands on remote hosts. if I pass my command directly in my ssh script it's working, once I pass it from another script as an argument it gives me a result of my first host and ...
user avatar
  • 105
2 votes
1 answer
198 views

GNU Parallel stopped working

GNU Parallel worked fine and suddenly whenever I try to use it I am getting this error message I am getting while running any parallel command: parallel: This should not happen. You have found a bug. ...
user avatar
  • 105
0 votes
1 answer
606 views

parallel: Warning: No more file handles

I am running GNU parallel and after some time I am getting: parallel: Warning: no more file handles Raising ulimit -n or etc/security/limits.conf may help. What argument would be appropriate to add ...
user avatar
  • 105
0 votes
1 answer
123 views

Passwordless SSH, list of possible passwords to try against any given host

So I know that to enable Passwordless SSH I need to generate a public authentication key and append it to the remote hosts ~/.ssh/authorized_keys file, generating a new SSH key pair. The question is ...
user avatar
  • 105
1 vote
1 answer
111 views

GNU parallel bash

I am learning GNU parallel and I wonder it this makes sense: bash: IFS=" " while read field || [ -n "$field" ]; do targets_array+=("$field") done </...
user avatar
  • 105
1 vote
1 answer
169 views

GNU parallel is executing all commands at once within my function

Ok so i have a bash function that I apply on several folders: function task(){ do_thing1 do_thing2 do_thing3 ... } I want to run that function in parallel. So far I was using a little fork trick: N=4 ...
user avatar
  • 11
2 votes
1 answer
108 views

gnu parallel: how to set the limit per second

I need to use parallel and set the rate limit per second, because I need to query an API that has a "5 per second" rate limit. Do I must combine -n5 and --timeout 1? Thank you
user avatar
  • 2,039
0 votes
3 answers
612 views

get minimum and maximum line count from files within a directory

I have some 5 million text files under a directory - all of the same format (nothing special, just plain text files with some integers in each line). Id like to compute the maximum and minimum line ...
user avatar
  • 3
2 votes
2 answers
152 views

Process multiple inputs with multiple equivalent commands (multiple thread pools) in GNU parallel

I have a large dataset (>200k files) that I would like to process (convert files into another format). The algorithm is mostly single-threaded, so it would be natural to use parallel processing. ...
user avatar
  • 4,707
1 vote
1 answer
63 views

Bash parallel command is executing commands sequentially

I am trying to download multiple files parallelly in bash and I came across GNU parallel. It looks very simple and straight forward. But I am having a hard time getting GNU parallel working. What am I ...
user avatar
1 vote
2 answers
166 views

running tasks in parallel as well writing to single file

#!/bin/bash range=$(seq -f "ma%04g" 5001 5505) for n in $range;do ping -q -c 1 -w 3 -s 10 $n >/dev/null if [ $? -eq 0 ];then awk -F, -v key="${n}" -v ...
user avatar
  • 37
3 votes
2 answers
435 views

Zenity Cancel button for GNU parallel progress bar

As GNU parallel's manual shows, you can use a zenity progress bar with parallel: seq 1000 | parallel -j30 --bar '(echo {};sleep 0.1)' \ 2> >(zenity --progress --auto-kill) | wc However, in ...
user avatar

1
2 3 4 5