Questions tagged [gnu-parallel]

GNU parallel is a command line utility to run programs in parallel

Filter by
Sorted by
Tagged with
1
vote
1answer
22 views

gnu parallel: how to set the limit per second

I need to use parallel and set the rate limit per second, because I need to query an API that has a "5 per second" rate limit. Do I must combine -n5 and --timeout 1? Thank you
0
votes
0answers
20 views

Start 2 commands, the second after the first printed a given line [duplicate]

I am developing on 2 different stacks: quarkus and angular. In order to develop in dev mode (watch mode), I need to start both applications on different terminals: quarkus is started from maven (mvn ...
0
votes
3answers
67 views

get minimum and maximum line count from files within a directory

I have some 5 million text files under a directory - all of the same format (nothing special, just plain text files with some integers in each line). Id like to compute the maximum and minimum line ...
0
votes
0answers
35 views

Problem with the dynamic job limit in GNU/parallel

I have a list of jobs, each of which requires different execution times and different RAM usage. It is impossible to predict beforehand if a job is heavy or not. I run this list on a system with 88 ...
2
votes
2answers
43 views

Process multiple inputs with multiple equivalent commands (multiple thread pools) in GNU parallel

I have a large dataset (>200k files) that I would like to process (convert files into another format). The algorithm is mostly single-threaded, so it would be natural to use parallel processing. ...
1
vote
1answer
40 views

Bash parallel command is executing commands sequentially

I am trying to download multiple files parallelly in bash and I came across GNU parallel. It looks very simple and straight forward. But I am having a hard time getting GNU parallel working. What am I ...
1
vote
2answers
47 views

running tasks in parallel as well writing to single file

#!/bin/bash range=$(seq -f "ma%04g" 5001 5505) for n in $range;do ping -q -c 1 -w 3 -s 10 $n >/dev/null if [ $? -eq 0 ];then awk -F, -v key="${n}" -v ...
3
votes
2answers
186 views

Zenity Cancel button for GNU parallel progress bar

As GNU parallel's manual shows, you can use a zenity progress bar with parallel: seq 1000 | parallel -j30 --bar '(echo {};sleep 0.1)' \ 2> >(zenity --progress --auto-kill) | wc However, in ...
1
vote
1answer
31 views

parallel: Pausing (swapping out) long-running progress when above limit threshold

Let's say that I have 10 GBs of RAM and unlimited swap. I want to run 10 jobs in parallel (gnu parallel is an option but not the only one necessarily). These jobs progressively need more and more ...
1
vote
1answer
60 views

GNU parallel + gunzip (or 7z, or bsdtar, or unzip): extract every “archivename.zip” into (to-be-created) its “archivename” subfolder

As in title. I've got a lot of ZIP archives that I want to extract. All archives have their own unique name. All archives contain files only (inside archives there are NOT folder(s) at all: no ...
2
votes
1answer
56 views

GNU Parallel time stamp output

Not being a perl programmer I'm thinking this would be fairly easy with --tagstring, but basically I'd like to time-stamp each line of output from each job individually from Parallel. Something like, ...
1
vote
2answers
107 views

How to pass job number in GNU Parallel?

I pass the variables from a script to the main script in a parallel command like ./script1 | parallel -u --jobs 3 "./script2 {}" How can I pass the job number as the second argument of ./script2? ...
1
vote
2answers
81 views

GNU Parallel linking arguments with alternating arguments

I made a Python simulator that runs on the basis of user-provided arguments. To use the program, I run multiple random simulations (controlled with a seed value). I use GNU parallel to run the ...
2
votes
1answer
69 views

How can I recursively find directories and parse them into python script call within bash script?

I am attempting to look into multiple subdirectories within a single directory, and plot the files in each subdirectory using a python script that I am calling within this bash script using gnu-...
2
votes
1answer
180 views

Can GNU Parallel execute a bash script 25000 times per seconds and parallel processes?

I am testing my web server response. Well I have a computer with 15 CPUs and 30 GB memory. How can I execute the following bash command on this computer 25000 times per second? curl -Z url="...
1
vote
0answers
67 views

Combine gnu-parallel and curl in Linux

I'm testing my webserver and sending API Thousand of request to optimize my program with high traffic. Indeed I write two bash-shells but they not fast as I want any one can help me on this and let me ...
4
votes
3answers
308 views

Rename all hidden directories created by GNU parallel

GNU Parallel is outputting hidden directories as follows using the --results parameter. What command do I use on Ubuntu to change the all to directories so that they are no longer hidden. The ...
0
votes
2answers
380 views

Archive multiple zip files into one file

How can I combine multiple existing zip files into a single file? I'm asking because I have many PDFs to zip, and I want to leverage parallel to save time in zipping them all up (as in ls ./*.pdf | ...
2
votes
2answers
65 views

List a file from all directories but skip the ones whose folder contains a specific file

I have a perl script that processes .xml files which reside in multiple folders under a directory. The script takes the absolute file path using the output of ls using parallel. parallel -j 1 /home/...
1
vote
1answer
131 views

GNU Parallel only using one CPU Core

I'm running some training using GNU parallel. The training takes about 30 secs to run one iteration, and I need to run about 3000. The training itself can't be parallelized (at least not without ...
2
votes
3answers
136 views

How to optimize GNU parallel for this use?

I created this script out of boredom with the sole purpose of using/testing GNU parallel so I know it's not particularly useful or optimized, but I have a script that will calculate all prime numbers ...
1
vote
1answer
105 views

GNU Parallel Python Semaphore

I have a Python script parallelized through GNU parallel which finds a certain result which I would like to output to a file, which I currently do through standard Python file IO. The issue is that I ...
1
vote
1answer
62 views

GNU parallel parameter behavior seems odd

So I'm trying to make a simple script that takes a source dir, and a destination dir as arguments, and creates an identical directory structure at the destination, then converts every file in the ...
2
votes
1answer
124 views

bash array printf in gnu parallel

I am trying pass bash array variable, that contain two elements to GNU parallel script: echo ${opts[@]} ...
8
votes
2answers
961 views

How do I use GNU parallel to calculate sha256 hash?

Based on this: Simultaneously calculate multiple digests (md5, sha256)? I have a folder that has a large number of files that I want to compute the SHA256 hash for. I used to code segment: #!/bin/...
1
vote
1answer
52 views

Combining gnu parallel and mpirun: how to take care of the max number of cores?

I have a machine with 32 cores where a I want to run some mpi jobs with a variable number of cores (this is because the problem requires that). Is there a way to make gnu parallel aware of the number ...
3
votes
2answers
299 views

ssh_askpass Permission denied, when using GNU parallel, even when using nohup

I am trying to run code across a network of external nodes. I have access to the 'main' node through ssh, and can execute a parallel script that divides the jobs over the cluster of 5 available nodes. ...
2
votes
2answers
402 views

Using variables as inputs in GNU parallel

I want to use variables as input while passing the arguments in GNU parallel. For instance, I have three bash scripts that I want to run in parallel using GNU parallel "par1.sh","par2.sh","par3.sh". ...
1
vote
4answers
123 views

Converting for loops on script that is called by another script into GNU parallel commands

Currently I have the following script for using the HaploTypeCaller program on my Unix system on a repeatable environment I created: #!/bin/bash #parallel call SNPs with chromosomes by GATK for i in ...
1
vote
1answer
160 views

How do I tell GNU Parallel to not quote the replacement string

GNU Parallel quotes replacement strings by default so that they are not expanded by the shell. But in certain cases you really want the replacement string to be interpreted by the shell. E.g. $ cat ...
0
votes
1answer
143 views

Simplify and parallel bash if statement script

I have written a script that finds files in directories, and brings through if statement, here is a code: for dirname in /input/*; do id=${dirname#/input/} # remove "/input/sub-" id=${id%/...
3
votes
1answer
78 views

Is there a way to tell GNU parallel to hold off spawning new jobs until all jobs in a batch has finished?

I want to run four processes in parallel, but not spawn any new jobs until all of these four have finished. EDIT: My command looks like this: find . -name "*.log" | parallel -j 4 './process.sh {}'
0
votes
1answer
178 views

Bash script doesn't executed properly inside docker container [duplicate]

I am trying to run bash script: export LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH export SUBJECTS_DIR=/output timefmt="%C --- CPU:\t%E real,\t%U user,\t%S sys\t%P\tMem:\t%KkiBavg.,\t%MkiB max.\...
1
vote
1answer
66 views

GNU Parallel waits for n jobs before starting

Im trying to use GNU parallel on a script, and i noticed that it only starts to output, after -jX X jobs # Only spawns cat after 100 seconds (echo a; sleep 100) | parallel -j1 --lb cat # Starts ...
3
votes
1answer
763 views

Parallel running with only limited CPU cores

I have a 12 cores server and I want to run a script which take a file as input argument and does processing on it. I want to use 8 core and no further to do parallel processing on 8 files at a time on ...
1
vote
2answers
2k views

How may I run rsync with “parallel” from the local system to fetch files in parallel?

I'm trying to rsync approx 10 TB of data from a remote system to the local machine, and I want to use the parallel utility for multi-thread execution. I want to trigger the rsync from the local server....
1
vote
1answer
52 views

parallel with quote in command

my original input in linux terminal is bcftools filter -e 'TYPE="snp"' input1.vcf -O v -o output1.filter.vcf bcftools filter -e 'TYPE="snp"' input2.vcf -O v -o output2.filter.vcf bcftools filter ...
3
votes
1answer
1k views

gnu parallel how to use with ffmpeg

I have folder that contains some mp4 files I want to extract images from each of them in parallel and save it as: filename_number.bmp where filename is the input filename how to do that using ...
2
votes
1answer
3k views

How to include input file name in output file name in ffmpeg

I want to extract frames as images from video and I want each image to be named as InputFileName_number.bmp.  How can I do this? I tried the following command: ffmpeg -i clip.mp4 fr1/$filename%d.jpg ...
0
votes
1answer
377 views

GNU Parallel facing silent exit and invalid option error

I tried performing the full installation from: http://git.savannah.gnu.org/cgit/parallel.git/tree/README The installation was successful. It's working well when installed on Mac OS but on Amazon ...
1
vote
1answer
97 views

Change max line length of GNU parallel

The following command: parallel --max-line-length-allowed returns 131049. Is this a constant, or can I somehow change this limit?
2
votes
0answers
106 views

Change output width (columns) of processes created by gnu parallel

I'm trying to save the output of a command into files and it seems to work fine. However, when opening one of the files, I can see that output was limited to 80 characters width. How can I ensure that ...
4
votes
1answer
431 views

Make GNU Parallel not delay before executing arguments from STDIN

GNU Parallel, without any command line options, allows you to easily parallelize a command whose last argument is determined by a line of STDIN: $ seq 3 | parallel echo 2 1 3 Note that parallel does ...
1
vote
3answers
104 views

GNU Parallel alternating jobs

I want to run a task where I specify two commands which will be run in an alternating fashion with different parameters. E.g: 1. exec --foo $inputfile1 $inputfile.outfile1 2. exec --bar $inputfile2 $...
0
votes
2answers
199 views

Convert parallel to xargs command

I'm trying to convert this command to something that does not use parallel. find . -type f -iregex '.*.ts\|.*.js\|.*.tsx\|.*.jsx' | parallel 'touch {} -d "$(date -d \@$((0x$(md5sum {} | cut -b 1-7))))...
1
vote
1answer
350 views

gnu parallel exit process with timeout

Is it possible to abort process for gnu parallel process if it exceeds an estimated runtime? For example, I have a handler for recon-all processing: while [ -n "${ids[0]}" ] ; do printf 'Processing ...
1
vote
2answers
859 views

gnu parallel with bash array

I trying to run command recon-all with GNU parallel freesurfer preproc i have a bash array of list of patients to run 8 patents simultaneously: root@4d8896dfec6c:/tmp# echo ${ids[@]} G001 G002 G003 ...
0
votes
1answer
436 views

GNU parallel proper usage in conjunction with tar, xz, gpg

I would like to speed up my archiving operations, I am usually doing 23 GiB (one Blu-Ray) backups. I have found this: How to do large file parallel encryption using GnuPG and GNU parallel? Since I ...
1
vote
1answer
169 views

GNU Parallel: redirect piped stdin as if it were a file

Suppose I have the following command to be parallelized: my_command --file <(my | pipeline) Now, I would like to parallelize in specific chunks: my | pipeline | parallel --spreadstdin ...
1
vote
1answer
362 views

kill gnu parallel inside a bash function

I am using gnu parallel in a subshell function, e.g., func()( parallel --will-cite sleep ::: 60 ) upon sending SIGTERM to the function I want parallel to be killed too. But I get $ func & [...