Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gnu-parallel]

GNU parallel is a command line utility to run programs in parallel

1
vote
1answer
19 views

GNU Parallel waits for n jobs before starting

Im trying to use GNU parallel on a script, and i noticed that it only starts to output, after -jX X jobs # Only spawns cat after 100 seconds (echo a; sleep 100) | parallel -j1 --lb cat # Starts ...
3
votes
1answer
45 views

Parallel running with only limited CPU cores

I have a 12 cores server and I want to run a script which take a file as input argument and does processing on it. I want to use 8 core and no further to do parallel processing on 8 files at a time on ...
1
vote
2answers
72 views

How may I run rsync with “parallel” from the local system to fetch files in parallel?

I'm trying to rsync approx 10 TB of data from a remote system to the local machine, and I want to use the parallel utility for multi-thread execution. I want to trigger the rsync from the local server....
1
vote
1answer
23 views

parallel with quote in command

my original input in linux terminal is bcftools filter -e 'TYPE="snp"' input1.vcf -O v -o output1.filter.vcf bcftools filter -e 'TYPE="snp"' input2.vcf -O v -o output2.filter.vcf bcftools filter ...
2
votes
1answer
202 views

gnu parallel how to use with ffmpeg

I have folder that contains some mp4 files I want to extract images from each of them in parallel and save it as: filename_number.bmp where filename is the input filename how to do that using ...
1
vote
1answer
91 views

How to include input file name in output file name in ffmpeg

I want to extract frames as images from video and I want each image to be named as InputFileName_number.bmp.  How can I do this? I tried the following command: ffmpeg -i clip.mp4 fr1/$filename%d.jpg ...
0
votes
1answer
56 views

GNU Parallel facing silent exit and invalid option error

I tried performing the full installation from: http://git.savannah.gnu.org/cgit/parallel.git/tree/README The installation was successful. It's working well when installed on Mac OS but on Amazon ...
1
vote
1answer
52 views

Change max line length of GNU parallel

The following command: parallel --max-line-length-allowed returns 131049. Is this a constant, or can I somehow change this limit?
2
votes
0answers
34 views

Change output width (columns) of processes created by gnu parallel

I'm trying to save the output of a command into files and it seems to work fine. However, when opening one of the files, I can see that output was limited to 80 characters width. How can I ensure that ...
3
votes
1answer
84 views

Make GNU Parallel not delay before executing arguments from STDIN

GNU Parallel, without any command line options, allows you to easily parallelize a command whose last argument is determined by a line of STDIN: $ seq 3 | parallel echo 2 1 3 Note that parallel does ...
1
vote
3answers
35 views

GNU Parallel alternating jobs

I want to run a task where I specify two commands which will be run in an alternating fashion with different parameters. E.g: 1. exec --foo $inputfile1 $inputfile.outfile1 2. exec --bar $inputfile2 $...
0
votes
2answers
51 views

Convert parallel to xargs command

I'm trying to convert this command to something that does not use parallel. find . -type f -iregex '.*.ts\|.*.js\|.*.tsx\|.*.jsx' | parallel 'touch {} -d "$(date -d \@$((0x$(md5sum {} | cut -b 1-7))))...
1
vote
1answer
90 views

gnu parallel exit process with timeout

Is it possible to abort process for gnu parallel process if it exceeds an estimated runtime? For example, I have a handler for recon-all processing: while [ -n "${ids[0]}" ] ; do printf 'Processing ...
1
vote
2answers
74 views

gnu parallel with bash array

I trying to run command recon-all with GNU parallel freesurfer preproc i have a bash array of list of patients to run 8 patents simultaneously: root@4d8896dfec6c:/tmp# echo ${ids[@]} G001 G002 G003 ...
0
votes
1answer
81 views

GNU parallel proper usage in conjunction with tar, xz, gpg

I would like to speed up my archiving operations, I am usually doing 23 GiB (one Blu-Ray) backups. I have found this: How to do large file parallel encryption using GnuPG and GNU parallel? Since I ...
1
vote
1answer
50 views

GNU Parallel: redirect piped stdin as if it were a file

Suppose I have the following command to be parallelized: my_command --file <(my | pipeline) Now, I would like to parallelize in specific chunks: my | pipeline | parallel --spreadstdin ...
0
votes
1answer
70 views

kill gnu parallel inside a bash function

I am using gnu parallel in a subshell function, e.g., func()( parallel --will-cite sleep ::: 60 ) upon sending SIGTERM to the function I want parallel to be killed too. But I get $ func & [...
2
votes
1answer
116 views

filtering a large file with a larg filter

I want to extract all lines of $file1 that start with a string stored in $file2. $file1 is 4 GB large with about 20 million lines, $file2 has 2 million lines, is about 140 MB large and contains two ...
1
vote
3answers
68 views

Copying & Renaming Files with GNU Parallel

I have a simple script that I want to copy and rename files, in files.lst based on a list of names in names.lst **name.lst** 100GV200.vcf 150GV200.vcf 14300GV200.vcf **file.lst** file1.txt file2.txt ...
1
vote
1answer
39 views

running a program in parallel with multiple threads

Say I want to run a parallelize a program to process a bunch of files, and this program accepts a -t option to increase the number of threads used. The machine I am using has the following specs from ...
1
vote
1answer
48 views

What are the requirements of gnu-parallel to/for a remote device that is contacted via ssh (and public-key authentication)

I am using gnu-parallel. I assume there is a problem with my environment. What I want to achieve: Trying to execute a command on multiple machines. For that I've created a sshloginfile (aptly named ~/....
1
vote
3answers
219 views

Can't run parallel zgrep with regex

When I run this: find . -name "my.log*.gz" -print | \ parallel zgrep -e "\(text1\|text2\).*Exception" to find all lines that match /(text1|text2).*Exception/ in all files in the folder, I get this ...
1
vote
2answers
252 views

Run parallel command and redirect the output to files with specific name

I can't undestand well how the parallel command works. I need to run this simple command: (100 times) curl https://jsonplaceholder.typicode.com/todos/1 curl https://jsonplaceholder.typicode.com/...
2
votes
1answer
36 views

parallel doesn't show intermediate output to terminal?

parallel -k python a1.py {} ::: {0..50..2} In the a1.py, I have some print statement. Running the parallel command doesn't show the print statement of my python-script. Is there any way for ...
3
votes
1answer
144 views

Why is find piping in non directories when the “-type d” test is used?

I have a directory ~/root/ |-- bar |-- eggs |-- foo |-- hello.txt |-- script.sh `-- spam 4 directories, 2 files Issuing find . -type d while in ~/root/ yields . ./spam ./eggs ./bar ./foo However, ...
2
votes
1answer
183 views

Multi-threading diff

I'm comparing the content of two directories using the diff command in bash. It uses only one thread on a multi-thread system. Does somebody knows a way to have diff to use more than one cpu/thread? ...
0
votes
1answer
40 views

Copy Files between Wondows and Linux parallelly [closed]

I have a problem to transfer files from windows dir to Linux parallelly via internet. For example, using putty, I have 8 files in my windows directory, and should send them to the Linux server ...
4
votes
1answer
127 views

GNU Parallel doesn't run the jobs until the program has exited

When I run (rss-notifier's code is included at the end), rss-notifier.zsh https://www.wuxiaworld.com/feed/chapters ".*"|parallel --null -k --lb echo {} I get, Title: Sovereign of the Three Realms - ...
2
votes
0answers
370 views

Parallel Copy from local folders to remote servers at the same time

I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the ...
2
votes
1answer
135 views

Can GNU Parallel Alter the Output of a Bash Script

I have a directory which is filled with subdirectories and in each subdirectory 74 files. I have bash script which analyzes the 74 files and wish to run it in parallel on all directories. When I go ...
2
votes
1answer
82 views

Extract the pattern of Directory in GNU parallel

I am running a command line software on multiple folder/samples.Each folder has such files *fastq.gz. Below is an example of a folder. Sample_EC_only/EC_only_S1_L005_I1_001.fastq.gz Sample_EC_only/...
2
votes
2answers
68 views

Parallely running multiple copies of the same file with different inputs using shell script

Suppose I have a file "Analysis.C" which takes a data file as input. The data file is named as "a.00001.txt" through "a.01000.txt". One way to loop over all the files is to write a shell script where ...
1
vote
1answer
426 views

Execute as fast as possible many curl commands

I am gathering some data locally at home on a Raspberry Pi and I want to send the data as fast as possible to a REST API (which I own) on a server on the web. The locally gathered data could flow as ...
0
votes
1answer
75 views

AWK to replace a line starts with in big file [closed]

I would like to replace a line in file which start with the content of variable $cont, using awk. 5473 12G Ju 2 06:32 part3 5423 11G Ju 6 07:32 part4 573 11G Ju 2 08:21 part5 5473 11G Ju 6 09:12 ...
2
votes
1answer
89 views

Replace a line with another content using sed and parallel for a big file

I have a big file (50G) and I would like to replace the lines starting with the content of variable $i with content of variable $newline using parallel command. My script is given below #!/bin/sh ...
3
votes
3answers
1k views

GNU parallel with for loop?

I've found answers close to this but fail to understand how to use them in my case (I'm rather new to Bash)... so, I'm trying to process a folder containing a large image sequence (100k+ files) with ...
1
vote
1answer
29 views

Syntax error when using sed to replace line-specific string in parallel: {= s/_1/_2/ =}?

I have a directory with files that look like this: id1_1.txt id1_2.txt id2_1.txt id2_2.txt I need to pass these files as a couple (e.g id1_1.txt and id1_2.txt) to my_script. Here's what I thought ...
4
votes
3answers
1k views

Running local shell script on multiple remote server in parallel and get the status

I need to run a local script on remote servers. Running script to run in parallel is important. [ec2-user@ip-172-31-43-140 ~]$ cat hosts.txt ec2-user@18.218.191.143 ec2-user@...
3
votes
1answer
51 views

Create separate 7z archives for each directory in the current directory and additionally parallelize through GNU Parallel

I have about two hundred sub-directories located within a directory of interest: $ ls backup 201302 201607 201608 201609 201610 201701 201702 201705 201801 201802 I want to create a 7z archive xyz....
5
votes
2answers
164 views

Standard Out Append to File Size Limitations

I'm pulling VIN specifications from the National Highway Traffic Safety Administration API for approximately 25,000,000 VIN numbers. This is a great deal of data, and as I'm not transforming the data ...
0
votes
0answers
467 views

Gnu parallel rsync hanging indefinitely

To be clear, I am running this exact command(variables changed) in a gnu parallel setting: /usr/bin/parallel -q -j20 rsync --recursive -av -e ssh -oCompression=no -x -T -oUserKnownHostsFile=...
3
votes
2answers
1k views

Can GNU parallel output stdout before the program has exited?

echo bash -c \''echo "hello, world!";sleep 3;'\' | parallel This command does not output anything until it has completed. Parallel's man page claims: GNU parallel makes sure output from the ...
1
vote
2answers
477 views

Running multiple bash scripts with different names in parallel

I need to run a command for each individual instance of a given variable name in parallel. Sometimes, there might be 4 variables, other times there might be 100. For example, say I have this ...
1
vote
1answer
179 views

Counting failed commands when using `sem`

I want to run several shell scripts in parallel using sem and get the number of failed commands. However, their exit codes are always zero: sem bash -c 'exit 0' echo $? # => 0 sem bash -c 'exit 1'...
2
votes
1answer
906 views

gnu parallel pair argument with file input arguments

I have a file with many lines, and on each line I have the arguments I want to pass to parallel with a tab separator. I run this script cat $hdd_file | grep $ssd | parallel -C '\t' clean_and_destroy ...
2
votes
3answers
97 views

On which host does this parallel command hang?

I run this parallel command: parallel -q -j0 ssh {} 'echo {}; tmp/myscript' ::: host1 host2 host3 .... Above line shows a result for most hosts. Unfortunately the parallel command hangs on some ...
3
votes
1answer
684 views

parallelize rsync calls using gnu parallel

I am trying to parallelize my rsync calls using gnu-parallel as shown below. But whenever I run my below script, it looks like it doesn't copy anything at all. But I do see lot of rsync processes (ps ...
3
votes
2answers
891 views

Combine parallel and sequential commands

1. Summary I don't understand, how I can combine parallel and sequential commands in Linux. 2. Expected behavior Pseudocode: pip install pipenv sequential pipenv install --dev parallel task npm ...
1
vote
1answer
57 views

Keeping dirs in order with GNU Parallel

I need to copy a large amount of files into their own directories. The issue I am having is keeping them in order when I copy them with GNU parallel. For example, file_1.output gets placed in dir_19. ...
1
vote
2answers
383 views

How can I use GNU Parallel to assign values to variables?

I am trying to assign values to variables using GNU Parallel: $ parallel --dry-run --xapply a{1}={2} ::: a b c ::: 5 6 7 aa=5 ab=6 ac=7 But the execution seems not to be working OK: $ parallel --...