Questions tagged [gnu-parallel]

GNU parallel is a command line utility to run programs in parallel

Filter by
Sorted by
Tagged with
1
vote
1answer
34 views

How can I recursively find directories and parse them into python script call within bash script?

I am attempting to look into multiple subdirectories within a single directory, and plot the files in each subdirectory using a python script that I am calling within this bash script using gnu-...
2
votes
1answer
68 views

Can GNU Parallel execute a bash script 25000 times per seconds and parallel processes?

I am testing my web server response. Well I have a computer with 15 CPUs and 30 GB memory. How can I execute the following bash command on this computer 25000 times per second? curl -Z url="...
1
vote
0answers
35 views

Combine gnu-parallel and curl in Linux

I'm testing my webserver and sending API Thousand of request to optimize my program with high traffic. Indeed I write two bash-shells but they not fast as I want any one can help me on this and let me ...
4
votes
3answers
294 views

Rename all hidden directories created by GNU parallel

GNU Parallel is outputting hidden directories as follows using the --results parameter. What command do I use on Ubuntu to change the all to directories so that they are no longer hidden. The ...
0
votes
2answers
46 views

Archive multiple zip files into one file

How can I combine multiple existing zip files into a single file? I'm asking because I have many PDFs to zip, and I want to leverage parallel to save time in zipping them all up (as in ls ./*.pdf | ...
2
votes
2answers
58 views

List a file from all directories but skip the ones whose folder contains a specific file

I have a perl script that processes .xml files which reside in multiple folders under a directory. The script takes the absolute file path using the output of ls using parallel. parallel -j 1 /home/...
1
vote
1answer
48 views

GNU Parallel only using one CPU Core

I'm running some training using GNU parallel. The training takes about 30 secs to run one iteration, and I need to run about 3000. The training itself can't be parallelized (at least not without ...
2
votes
3answers
91 views

How to optimize GNU parallel for this use?

I created this script out of boredom with the sole purpose of using/testing GNU parallel so I know it's not particularly useful or optimized, but I have a script that will calculate all prime numbers ...
1
vote
1answer
69 views

GNU Parallel Python Semaphore

I have a Python script parallelized through GNU parallel which finds a certain result which I would like to output to a file, which I currently do through standard Python file IO. The issue is that I ...
1
vote
1answer
55 views

GNU parallel parameter behavior seems odd

So I'm trying to make a simple script that takes a source dir, and a destination dir as arguments, and creates an identical directory structure at the destination, then converts every file in the ...
2
votes
1answer
57 views

bash array printf in gnu parallel

I am trying pass bash array variable, that contain two elements to GNU parallel script: echo ${opts[@]} ...
8
votes
2answers
750 views

How do I use GNU parallel to calculate sha256 hash?

Based on this: Simultaneously calculate multiple digests (md5, sha256)? I have a folder that has a large number of files that I want to compute the SHA256 hash for. I used to code segment: #!/bin/...
1
vote
1answer
23 views

Combining gnu parallel and mpirun: how to take care of the max number of cores?

I have a machine with 32 cores where a I want to run some mpi jobs with a variable number of cores (this is because the problem requires that). Is there a way to make gnu parallel aware of the number ...
3
votes
2answers
157 views

ssh_askpass Permission denied, when using GNU parallel, even when using nohup

I am trying to run code across a network of external nodes. I have access to the 'main' node through ssh, and can execute a parallel script that divides the jobs over the cluster of 5 available nodes. ...
2
votes
2answers
129 views

Using variables as inputs in GNU parallel

I want to use variables as input while passing the arguments in GNU parallel. For instance, I have three bash scripts that I want to run in parallel using GNU parallel "par1.sh","par2.sh","par3.sh". ...
1
vote
4answers
87 views

Converting for loops on script that is called by another script into GNU parallel commands

Currently I have the following script for using the HaploTypeCaller program on my Unix system on a repeatable environment I created: #!/bin/bash #parallel call SNPs with chromosomes by GATK for i in ...
0
votes
1answer
90 views

How do I tell GNU Parallel to not quote the replacement string

GNU Parallel quotes replacement strings by default so that they are not expanded by the shell. But in certain cases you really want the replacement string to be interpreted by the shell. E.g. $ cat ...
0
votes
1answer
106 views

Simplify and parallel bash if statement script

I have written a script that finds files in directories, and brings through if statement, here is a code: for dirname in /input/*; do id=${dirname#/input/} # remove "/input/sub-" id=${id%/...
3
votes
1answer
55 views

Is there a way to tell GNU parallel to hold off spawning new jobs until all jobs in a batch has finished?

I want to run four processes in parallel, but not spawn any new jobs until all of these four have finished. EDIT: My command looks like this: find . -name "*.log" | parallel -j 4 './process.sh {}'
0
votes
1answer
79 views

Bash script doesn't executed properly inside docker container [duplicate]

I am trying to run bash script: export LD_LIBRARY_PATH=/usr/lib/fsl/5.0:$LD_LIBRARY_PATH export SUBJECTS_DIR=/output timefmt="%C --- CPU:\t%E real,\t%U user,\t%S sys\t%P\tMem:\t%KkiBavg.,\t%MkiB max.\...
1
vote
1answer
47 views

GNU Parallel waits for n jobs before starting

Im trying to use GNU parallel on a script, and i noticed that it only starts to output, after -jX X jobs # Only spawns cat after 100 seconds (echo a; sleep 100) | parallel -j1 --lb cat # Starts ...
3
votes
1answer
418 views

Parallel running with only limited CPU cores

I have a 12 cores server and I want to run a script which take a file as input argument and does processing on it. I want to use 8 core and no further to do parallel processing on 8 files at a time on ...
1
vote
2answers
1k views

How may I run rsync with “parallel” from the local system to fetch files in parallel?

I'm trying to rsync approx 10 TB of data from a remote system to the local machine, and I want to use the parallel utility for multi-thread execution. I want to trigger the rsync from the local server....
1
vote
1answer
38 views

parallel with quote in command

my original input in linux terminal is bcftools filter -e 'TYPE="snp"' input1.vcf -O v -o output1.filter.vcf bcftools filter -e 'TYPE="snp"' input2.vcf -O v -o output2.filter.vcf bcftools filter ...
3
votes
1answer
624 views

gnu parallel how to use with ffmpeg

I have folder that contains some mp4 files I want to extract images from each of them in parallel and save it as: filename_number.bmp where filename is the input filename how to do that using ...
1
vote
1answer
2k views

How to include input file name in output file name in ffmpeg

I want to extract frames as images from video and I want each image to be named as InputFileName_number.bmp.  How can I do this? I tried the following command: ffmpeg -i clip.mp4 fr1/$filename%d.jpg ...
0
votes
1answer
204 views

GNU Parallel facing silent exit and invalid option error

I tried performing the full installation from: http://git.savannah.gnu.org/cgit/parallel.git/tree/README The installation was successful. It's working well when installed on Mac OS but on Amazon ...
1
vote
1answer
70 views

Change max line length of GNU parallel

The following command: parallel --max-line-length-allowed returns 131049. Is this a constant, or can I somehow change this limit?
2
votes
0answers
73 views

Change output width (columns) of processes created by gnu parallel

I'm trying to save the output of a command into files and it seems to work fine. However, when opening one of the files, I can see that output was limited to 80 characters width. How can I ensure that ...
3
votes
1answer
296 views

Make GNU Parallel not delay before executing arguments from STDIN

GNU Parallel, without any command line options, allows you to easily parallelize a command whose last argument is determined by a line of STDIN: $ seq 3 | parallel echo 2 1 3 Note that parallel does ...
1
vote
3answers
88 views

GNU Parallel alternating jobs

I want to run a task where I specify two commands which will be run in an alternating fashion with different parameters. E.g: 1. exec --foo $inputfile1 $inputfile.outfile1 2. exec --bar $inputfile2 $...
0
votes
2answers
138 views

Convert parallel to xargs command

I'm trying to convert this command to something that does not use parallel. find . -type f -iregex '.*.ts\|.*.js\|.*.tsx\|.*.jsx' | parallel 'touch {} -d "$(date -d \@$((0x$(md5sum {} | cut -b 1-7))))...
1
vote
1answer
244 views

gnu parallel exit process with timeout

Is it possible to abort process for gnu parallel process if it exceeds an estimated runtime? For example, I have a handler for recon-all processing: while [ -n "${ids[0]}" ] ; do printf 'Processing ...
1
vote
2answers
417 views

gnu parallel with bash array

I trying to run command recon-all with GNU parallel freesurfer preproc i have a bash array of list of patients to run 8 patents simultaneously: root@4d8896dfec6c:/tmp# echo ${ids[@]} G001 G002 G003 ...
0
votes
1answer
262 views

GNU parallel proper usage in conjunction with tar, xz, gpg

I would like to speed up my archiving operations, I am usually doing 23 GiB (one Blu-Ray) backups. I have found this: How to do large file parallel encryption using GnuPG and GNU parallel? Since I ...
1
vote
1answer
107 views

GNU Parallel: redirect piped stdin as if it were a file

Suppose I have the following command to be parallelized: my_command --file <(my | pipeline) Now, I would like to parallelize in specific chunks: my | pipeline | parallel --spreadstdin ...
1
vote
1answer
280 views

kill gnu parallel inside a bash function

I am using gnu parallel in a subshell function, e.g., func()( parallel --will-cite sleep ::: 60 ) upon sending SIGTERM to the function I want parallel to be killed too. But I get $ func & [...
2
votes
1answer
283 views

filtering a large file with a larg filter

I want to extract all lines of $file1 that start with a string stored in $file2. $file1 is 4 GB large with about 20 million lines, $file2 has 2 million lines, is about 140 MB large and contains two ...
1
vote
3answers
251 views

Copying & Renaming Files with GNU Parallel

I have a simple script that I want to copy and rename files, in files.lst based on a list of names in names.lst **name.lst** 100GV200.vcf 150GV200.vcf 14300GV200.vcf **file.lst** file1.txt file2.txt ...
1
vote
1answer
47 views

running a program in parallel with multiple threads

Say I want to run a parallelize a program to process a bunch of files, and this program accepts a -t option to increase the number of threads used. The machine I am using has the following specs from ...
1
vote
1answer
116 views

What are the requirements of gnu-parallel to/for a remote device that is contacted via ssh (and public-key authentication)

I am using gnu-parallel. I assume there is a problem with my environment. What I want to achieve: Trying to execute a command on multiple machines. For that I've created a sshloginfile (aptly named ~/....
1
vote
3answers
1k views

Can't run parallel zgrep with regex

When I run this: find . -name "my.log*.gz" -print | \ parallel zgrep -e "\(text1\|text2\).*Exception" to find all lines that match /(text1|text2).*Exception/ in all files in the folder, I get this ...
1
vote
2answers
1k views

Run parallel command and redirect the output to files with specific name

I can't undestand well how the parallel command works. I need to run this simple command: (100 times) curl https://jsonplaceholder.typicode.com/todos/1 curl https://jsonplaceholder.typicode.com/...
2
votes
1answer
91 views

parallel doesn't show intermediate output to terminal?

parallel -k python a1.py {} ::: {0..50..2} In the a1.py, I have some print statement. Running the parallel command doesn't show the print statement of my python-script. Is there any way for ...
3
votes
1answer
149 views

Why is find piping in non directories when the “-type d” test is used?

I have a directory ~/root/ |-- bar |-- eggs |-- foo |-- hello.txt |-- script.sh `-- spam 4 directories, 2 files Issuing find . -type d while in ~/root/ yields . ./spam ./eggs ./bar ./foo However, ...
2
votes
1answer
425 views

Multi-threading diff

I'm comparing the content of two directories using the diff command in bash. It uses only one thread on a multi-thread system. Does somebody knows a way to have diff to use more than one cpu/thread? ...
0
votes
1answer
68 views

Copy Files between Wondows and Linux parallelly [closed]

I have a problem to transfer files from windows dir to Linux parallelly via internet. For example, using putty, I have 8 files in my windows directory, and should send them to the Linux server ...
4
votes
1answer
311 views

GNU Parallel doesn't run the jobs until the program has exited

When I run (rss-notifier's code is included at the end), rss-notifier.zsh https://www.wuxiaworld.com/feed/chapters ".*"|parallel --null -k --lb echo {} I get, Title: Sovereign of the Three Realms - ...
3
votes
1answer
806 views

Parallel Copy from local folders to remote servers at the same time

I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the ...
2
votes
1answer
173 views

Can GNU Parallel Alter the Output of a Bash Script

I have a directory which is filled with subdirectories and in each subdirectory 74 files. I have bash script which analyzes the 74 files and wish to run it in parallel on all directories. When I go ...