Questions tagged [gnu-parallel]

GNU parallel is a command line utility to run programs in parallel

Filter by
Sorted by
Tagged with
3
votes
0answers
67 views

2>&1 causes process to block

I am trying to debug what I have done wrong in GNU Parallel (>=20190122). This runs as expected: ( echo 1 | parallel-20190122 --pipe -j0 --tee dummy {} ::: {1..1000}; echo Parallel is done here:...
1
vote
1answer
67 views

gnu parallel - speed up command agains remote servers that waits

I'm using gnu parallel to generate backups for about ~1500 web sites on pantheon.io, using their terminus CLI. The terminus backup:create command does not finish until a response is received that it ...
1
vote
1answer
37 views

Use GNU Parallel when file has a single (long) line

I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv. This file has 544609657 characters, is about 545 megabytes, all in a single line. ...
1
vote
1answer
22 views

How to limit the number of tasks to the first arguments in GNU Parallel?

I want to run a series of parallel jobs based on a set of arguments while assigning a second argument. I use the --link option in GNU Parallel as parallel --jobs 3 --link echo ::: A B C ::: D E F G A ...
2
votes
2answers
79 views

How to run a bash script using threads/parallel

I am running the following bash script as is. I.e. sequentially. Note that the prefetch and fastq-dump are calls for downloading files from a database, and _shrink is a C++ program that I have made to ...
0
votes
1answer
11 views

Is it possible to continuously feed parallel with more input?

What I want to do is something like this: inotifywait -m ./input -e close_write -e moved_to | parallel --max-args 1 "someSlowSingleThreadedTaskThatActsOnThisFile" I can't imagine that ...
1
vote
2answers
30 views

How to use parallel with inotifywait?

I currently have this: inotifywait -m ./input -e close_write | parallel --max-args 1 "echo {1}" For some reason this doesn't do anything at all. The output of inotifywait is the same as ...
0
votes
1answer
45 views

Use parallel to speed up a recursive shell max function?

I needed a max function, to find the highest value in an array (of 350 int values), and being a friend of recursion, I tried a recursive approach: max () { len=$# a=$1 b=$2 case $len ...
1
vote
1answer
39 views

convert each line in a file into an input file

I am trying to convert/use each line from a file as an input file. For example file.txt contains the following lines: cat dog lion tiger rabbit Now, using the following command: cat file.txt | ...
1
vote
1answer
21 views

parallel compare two arguments and send to command

I am trying to run recon-all command with GNU parallel. It should take one argument at {1} like this: parallel --progress subj_id='echo {1} | cut -d / -f 3' && recon-all -s $subj_id -i {1} -sd ...
3
votes
3answers
143 views

Boosting the grep search using GNU parallel

I am using the following grep script to output all the unmatched patterns: grep -oFf patterns.txt large_strings.txt | grep -vFf - patterns.txt > unmatched_patterns.txt patterns file contains the ...
1
vote
1answer
46 views

gnu parallel: Warning: This job was killed because it timed out

I am running multiple commands on remote hosts. if I pass my command directly in my ssh script it's working, once I pass it from another script as an argument it gives me a result of my first host and ...
1
vote
1answer
65 views

GNU Parallel stopped working

GNU Parallel worked fine and suddenly whenever I try to use it I am getting this error message I am getting while running any parallel command: parallel: This should not happen. You have found a bug. ...
1
vote
1answer
162 views

parallel: Warning: No more file handles

I am running GNU parallel and after some time I am getting: parallel: Warning: no more file handles Raising ulimit -n or etc/security/limits.conf may help. What argument would be appropriate to add ...
0
votes
1answer
57 views

Passwordless SSH, list of possible passwords to try against any given host

So I know that to enable Passwordless SSH I need to generate a public authentication key and append it to the remote hosts ~/.ssh/authorized_keys file, generating a new SSH key pair. The question is ...
1
vote
1answer
61 views

GNU parallel bash

I am learning GNU parallel and I wonder it this makes sense: bash: IFS=" " while read field || [ -n "$field" ]; do targets_array+=("$field") done </...
1
vote
1answer
64 views

GNU parallel is executing all commands at once within my function

Ok so i have a bash function that I apply on several folders: function task(){ do_thing1 do_thing2 do_thing3 ... } I want to run that function in parallel. So far I was using a little fork trick: N=4 ...
2
votes
1answer
46 views

gnu parallel: how to set the limit per second

I need to use parallel and set the rate limit per second, because I need to query an API that has a "5 per second" rate limit. Do I must combine -n5 and --timeout 1? Thank you
0
votes
0answers
20 views

Start 2 commands, the second after the first printed a given line [duplicate]

I am developing on 2 different stacks: quarkus and angular. In order to develop in dev mode (watch mode), I need to start both applications on different terminals: quarkus is started from maven (mvn ...
0
votes
3answers
179 views

get minimum and maximum line count from files within a directory

I have some 5 million text files under a directory - all of the same format (nothing special, just plain text files with some integers in each line). Id like to compute the maximum and minimum line ...
0
votes
0answers
56 views

Problem with the dynamic job limit in GNU/parallel

I have a list of jobs, each of which requires different execution times and different RAM usage. It is impossible to predict beforehand if a job is heavy or not. I run this list on a system with 88 ...
2
votes
2answers
83 views

Process multiple inputs with multiple equivalent commands (multiple thread pools) in GNU parallel

I have a large dataset (>200k files) that I would like to process (convert files into another format). The algorithm is mostly single-threaded, so it would be natural to use parallel processing. ...
1
vote
1answer
47 views

Bash parallel command is executing commands sequentially

I am trying to download multiple files parallelly in bash and I came across GNU parallel. It looks very simple and straight forward. But I am having a hard time getting GNU parallel working. What am I ...
1
vote
2answers
56 views

running tasks in parallel as well writing to single file

#!/bin/bash range=$(seq -f "ma%04g" 5001 5505) for n in $range;do ping -q -c 1 -w 3 -s 10 $n >/dev/null if [ $? -eq 0 ];then awk -F, -v key="${n}" -v ...
3
votes
2answers
260 views

Zenity Cancel button for GNU parallel progress bar

As GNU parallel's manual shows, you can use a zenity progress bar with parallel: seq 1000 | parallel -j30 --bar '(echo {};sleep 0.1)' \ 2> >(zenity --progress --auto-kill) | wc However, in ...
2
votes
2answers
53 views

parallel: Pausing (swapping out) long-running progress when above memory limit threshold

Let's say that I have 10 GBs of RAM and unlimited swap. I want to run 10 jobs in parallel (gnu parallel is an option but not the only one necessarily). These jobs progressively need more and more ...
1
vote
1answer
171 views

GNU parallel + gunzip (or 7z, or bsdtar, or unzip): extract every “archivename.zip” into (to-be-created) its “archivename” subfolder

As in title. I've got a lot of ZIP archives that I want to extract. All archives have their own unique name. All archives contain files only (inside archives there are NOT folder(s) at all: no ...
2
votes
1answer
85 views

GNU Parallel time stamp output

Not being a perl programmer I'm thinking this would be fairly easy with --tagstring, but basically I'd like to time-stamp each line of output from each job individually from Parallel. Something like, ...
1
vote
2answers
264 views

How to pass job number in GNU Parallel?

I pass the variables from a script to the main script in a parallel command like ./script1 | parallel -u --jobs 3 "./script2 {}" How can I pass the job number as the second argument of ./script2? ...
1
vote
2answers
124 views

GNU Parallel linking arguments with alternating arguments

I made a Python simulator that runs on the basis of user-provided arguments. To use the program, I run multiple random simulations (controlled with a seed value). I use GNU parallel to run the ...
2
votes
1answer
197 views

How can I recursively find directories and parse them into python script call within bash script?

I am attempting to look into multiple subdirectories within a single directory, and plot the files in each subdirectory using a python script that I am calling within this bash script using gnu-...
2
votes
1answer
332 views

Can GNU Parallel execute a bash script 25000 times per seconds and parallel processes?

I am testing my web server response. Well I have a computer with 15 CPUs and 30 GB memory. How can I execute the following bash command on this computer 25000 times per second? curl -Z url="...
1
vote
0answers
107 views

Combine gnu-parallel and curl in Linux

I'm testing my webserver and sending API Thousand of request to optimize my program with high traffic. Indeed I write two bash-shells but they not fast as I want any one can help me on this and let me ...
4
votes
3answers
341 views

Rename all hidden directories created by GNU parallel

GNU Parallel is outputting hidden directories as follows using the --results parameter. What command do I use on Ubuntu to change the all to directories so that they are no longer hidden. The ...
0
votes
2answers
943 views

Archive multiple zip files into one file

How can I combine multiple existing zip files into a single file? I'm asking because I have many PDFs to zip, and I want to leverage parallel to save time in zipping them all up (as in ls ./*.pdf | ...
2
votes
2answers
68 views

List a file from all directories but skip the ones whose folder contains a specific file

I have a perl script that processes .xml files which reside in multiple folders under a directory. The script takes the absolute file path using the output of ls using parallel. parallel -j 1 /home/...
1
vote
1answer
225 views

GNU Parallel only using one CPU Core

I'm running some training using GNU parallel. The training takes about 30 secs to run one iteration, and I need to run about 3000. The training itself can't be parallelized (at least not without ...
2
votes
3answers
224 views

How to optimize GNU parallel for this use?

I created this script out of boredom with the sole purpose of using/testing GNU parallel so I know it's not particularly useful or optimized, but I have a script that will calculate all prime numbers ...
1
vote
1answer
146 views

GNU Parallel Python Semaphore

I have a Python script parallelized through GNU parallel which finds a certain result which I would like to output to a file, which I currently do through standard Python file IO. The issue is that I ...
1
vote
1answer
76 views

GNU parallel parameter behavior seems odd

So I'm trying to make a simple script that takes a source dir, and a destination dir as arguments, and creates an identical directory structure at the destination, then converts every file in the ...
2
votes
1answer
162 views

bash array printf in gnu parallel

I am trying pass bash array variable, that contain two elements to GNU parallel script: echo ${opts[@]} ...
8
votes
2answers
1k views

How do I use GNU parallel to calculate sha256 hash?

Based on this: Simultaneously calculate multiple digests (md5, sha256)? I have a folder that has a large number of files that I want to compute the SHA256 hash for. I used to code segment: #!/bin/...
1
vote
1answer
98 views

Combining gnu parallel and mpirun: how to take care of the max number of cores?

I have a machine with 32 cores where a I want to run some mpi jobs with a variable number of cores (this is because the problem requires that). Is there a way to make gnu parallel aware of the number ...
3
votes
2answers
427 views

ssh_askpass Permission denied, when using GNU parallel, even when using nohup

I am trying to run code across a network of external nodes. I have access to the 'main' node through ssh, and can execute a parallel script that divides the jobs over the cluster of 5 available nodes. ...
2
votes
2answers
713 views

Using variables as inputs in GNU parallel

I want to use variables as input while passing the arguments in GNU parallel. For instance, I have three bash scripts that I want to run in parallel using GNU parallel "par1.sh","par2.sh","par3.sh". ...
1
vote
4answers
142 views

Converting for loops on script that is called by another script into GNU parallel commands

Currently I have the following script for using the HaploTypeCaller program on my Unix system on a repeatable environment I created: #!/bin/bash #parallel call SNPs with chromosomes by GATK for i in ...
1
vote
1answer
268 views

How do I tell GNU Parallel to not quote the replacement string

GNU Parallel quotes replacement strings by default so that they are not expanded by the shell. But in certain cases you really want the replacement string to be interpreted by the shell. E.g. $ cat ...
0
votes
1answer
193 views

Simplify and parallel bash if statement script

I have written a script that finds files in directories, and brings through if statement, here is a code: for dirname in /input/*; do id=${dirname#/input/} # remove "/input/sub-" id=${id%/...
3
votes
1answer
97 views

Is there a way to tell GNU parallel to hold off spawning new jobs until all jobs in a batch has finished?

I want to run four processes in parallel, but not spawn any new jobs until all of these four have finished. EDIT: My command looks like this: find . -name "*.log" | parallel -j 4 './process.sh {}'
0
votes
1answer
636 views

Bash script doesn't executed properly inside docker container [duplicate]

I am trying to run bash script: export LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH export SUBJECTS_DIR=/output timefmt="%C --- CPU:\t%E real,\t%U user,\t%S sys\t%P\tMem:\t%KkiBavg.,\t%MkiB max.\...

1
2 3 4 5