Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [gnu-parallel]

GNU parallel is a command line utility to run programs in parallel

0
votes
1answer
25 views

GNU Parallel facing silent exit and invalid option error

I tried performing the full installation from: http://git.savannah.gnu.org/cgit/parallel.git/tree/README The installation was successful. It's working well when installed on Mac OS but on Amazon ...
1
vote
1answer
38 views

Change max line length of GNU parallel

The following command: parallel --max-line-length-allowed returns 131049. Is this a constant, or can I somehow change this limit?
2
votes
0answers
26 views

Change output width (columns) of processes created by gnu parallel

I'm trying to save the output of a command into files and it seems to work fine. However, when opening one of the files, I can see that output was limited to 80 characters width. How can I ensure that ...
3
votes
1answer
54 views

Make GNU Parallel not delay before executing arguments from STDIN

GNU Parallel, without any command line options, allows you to easily parallelize a command whose last argument is determined by a line of STDIN: $ seq 3 | parallel echo 2 1 3 Note that parallel does ...
0
votes
2answers
24 views

GNU Parallel alternating jobs

I want to run a task where I specify two commands which will be run in an alternating fashion with different parameters. E.g: 1. exec --foo $inputfile1 $inputfile.outfile1 2. exec --bar $inputfile2 $...
-1
votes
2answers
37 views

Convert parallel to xargs command

I'm trying to convert this command to something that does not use parallel. find . -type f -iregex '.*.ts\|.*.js\|.*.tsx\|.*.jsx' | parallel 'touch {} -d "$(date -d \@$((0x$(md5sum {} | cut -b 1-7))))...
1
vote
1answer
60 views

gnu parallel exit process with timeout

Is it possible to abort process for gnu parallel process if it exceeds an estimated runtime? For example, I have a handler for recon-all processing: while [ -n "${ids[0]}" ] ; do printf 'Processing ...
1
vote
2answers
38 views

gnu parallel with bash array

I trying to run command recon-all with GNU parallel freesurfer preproc i have a bash array of list of patients to run 8 patents simultaneously: root@4d8896dfec6c:/tmp# echo ${ids[@]} G001 G002 G003 ...
0
votes
1answer
60 views

GNU parallel proper usage in conjunction with tar, xz, gpg

I would like to speed up my archiving operations, I am usually doing 23 GiB (one Blu-Ray) backups. I have found this: How to do large file parallel encryption using GnuPG and GNU parallel? Since I ...
1
vote
1answer
26 views

GNU Parallel: redirect piped stdin as if it were a file

Suppose I have the following command to be parallelized: my_command --file <(my | pipeline) Now, I would like to parallelize in specific chunks: my | pipeline | parallel --spreadstdin ...
0
votes
0answers
37 views

kill gnu parallel inside a bash function

I am using gnu parallel in a subshell function, e.g., func()( parallel --will-cite sleep ::: 60 ) upon sending SIGTERM to the function I want parallel to be killed too. But I get $ func & [...
2
votes
1answer
89 views

filtering a large file with a larg filter

I want to extract all lines of $file1 that start with a string stored in $file2. $file1 is 4 GB large with about 20 million lines, $file2 has 2 million lines, is about 140 MB large and contains two ...
1
vote
3answers
41 views

Copying & Renaming Files with GNU Parallel

I have a simple script that I want to copy and rename files, in files.lst based on a list of names in names.lst **name.lst** 100GV200.vcf 150GV200.vcf 14300GV200.vcf **file.lst** file1.txt file2.txt ...
1
vote
1answer
29 views

running a program in parallel with multiple threads

Say I want to run a parallelize a program to process a bunch of files, and this program accepts a -t option to increase the number of threads used. The machine I am using has the following specs from ...
1
vote
1answer
38 views

What are the requirements of gnu-parallel to/for a remote device that is contacted via ssh (and public-key authentication)

I am using gnu-parallel. I assume there is a problem with my environment. What I want to achieve: Trying to execute a command on multiple machines. For that I've created a sshloginfile (aptly named ~/....
1
vote
3answers
127 views

Can't run parallel zgrep with regex

When I run this: find . -name "my.log*.gz" -print | \ parallel zgrep -e "\(text1\|text2\).*Exception" to find all lines that match /(text1|text2).*Exception/ in all files in the folder, I get this ...
1
vote
2answers
126 views

Run parallel command and redirect the output to files with specific name

I can't undestand well how the parallel command works. I need to run this simple command: (100 times) curl https://jsonplaceholder.typicode.com/todos/1 curl https://jsonplaceholder.typicode.com/...
2
votes
1answer
31 views

parallel doesn't show intermediate output to terminal?

parallel -k python a1.py {} ::: {0..50..2} In the a1.py, I have some print statement. Running the parallel command doesn't show the print statement of my python-script. Is there any way for ...
3
votes
1answer
140 views

Why is find piping in non directories when the “-type d” test is used?

I have a directory ~/root/ |-- bar |-- eggs |-- foo |-- hello.txt |-- script.sh `-- spam 4 directories, 2 files Issuing find . -type d while in ~/root/ yields . ./spam ./eggs ./bar ./foo However, ...
2
votes
1answer
124 views

Multi-threading diff

I'm comparing the content of two directories using the diff command in bash. It uses only one thread on a multi-thread system. Does somebody knows a way to have diff to use more than one cpu/thread? ...
0
votes
1answer
35 views

Copy Files between Wondows and Linux parallelly [closed]

I have a problem to transfer files from windows dir to Linux parallelly via internet. For example, using putty, I have 8 files in my windows directory, and should send them to the Linux server ...
4
votes
1answer
87 views

GNU Parallel doesn't run the jobs until the program has exited

When I run (rss-notifier's code is included at the end), rss-notifier.zsh https://www.wuxiaworld.com/feed/chapters ".*"|parallel --null -k --lb echo {} I get, Title: Sovereign of the Three Realms - ...
2
votes
0answers
304 views

Parallel Copy from local folders to remote servers at the same time

I have a multiple folders, and each folder has about 1500 files. I have a kind of for loop going over each folder and then sending the files to either one or 4 remote hosts depending upon the ...
2
votes
1answer
125 views

Can GNU Parallel Alter the Output of a Bash Script

I have a directory which is filled with subdirectories and in each subdirectory 74 files. I have bash script which analyzes the 74 files and wish to run it in parallel on all directories. When I go ...
2
votes
1answer
67 views

Extract the pattern of Directory in GNU parallel

I am running a command line software on multiple folder/samples.Each folder has such files *fastq.gz. Below is an example of a folder. Sample_EC_only/EC_only_S1_L005_I1_001.fastq.gz Sample_EC_only/...
2
votes
2answers
54 views

Parallely running multiple copies of the same file with different inputs using shell script

Suppose I have a file "Analysis.C" which takes a data file as input. The data file is named as "a.00001.txt" through "a.01000.txt". One way to loop over all the files is to write a shell script where ...
1
vote
1answer
345 views

Execute as fast as possible many curl commands

I am gathering some data locally at home on a Raspberry Pi and I want to send the data as fast as possible to a REST API (which I own) on a server on the web. The locally gathered data could flow as ...
0
votes
1answer
70 views

AWK to replace a line starts with in big file [closed]

I would like to replace a line in file which start with the content of variable $cont, using awk. 5473 12G Ju 2 06:32 part3 5423 11G Ju 6 07:32 part4 573 11G Ju 2 08:21 part5 5473 11G Ju 6 09:12 ...
2
votes
1answer
74 views

Replace a line with another content using sed and parallel for a big file

I have a big file (50G) and I would like to replace the lines starting with the content of variable $i with content of variable $newline using parallel command. My script is given below #!/bin/sh ...
3
votes
3answers
994 views

GNU parallel with for loop?

I've found answers close to this but fail to understand how to use them in my case (I'm rather new to Bash)... so, I'm trying to process a folder containing a large image sequence (100k+ files) with ...
1
vote
1answer
28 views

Syntax error when using sed to replace line-specific string in parallel: {= s/_1/_2/ =}?

I have a directory with files that look like this: id1_1.txt id1_2.txt id2_1.txt id2_2.txt I need to pass these files as a couple (e.g id1_1.txt and id1_2.txt) to my_script. Here's what I thought ...
4
votes
3answers
848 views

Running local shell script on multiple remote server in parallel and get the status

I need to run a local script on remote servers. Running script to run in parallel is important. [ec2-user@ip-172-31-43-140 ~]$ cat hosts.txt ec2-user@18.218.191.143 ec2-user@...
3
votes
1answer
39 views

Create separate 7z archives for each directory in the current directory and additionally parallelize through GNU Parallel

I have about two hundred sub-directories located within a directory of interest: $ ls backup 201302 201607 201608 201609 201610 201701 201702 201705 201801 201802 I want to create a 7z archive xyz....
5
votes
2answers
155 views

Standard Out Append to File Size Limitations

I'm pulling VIN specifications from the National Highway Traffic Safety Administration API for approximately 25,000,000 VIN numbers. This is a great deal of data, and as I'm not transforming the data ...
0
votes
0answers
434 views

Gnu parallel rsync hanging indefinitely

To be clear, I am running this exact command(variables changed) in a gnu parallel setting: /usr/bin/parallel -q -j20 rsync --recursive -av -e ssh -oCompression=no -x -T -oUserKnownHostsFile=...
3
votes
2answers
967 views

Can GNU parallel output stdout before the program has exited?

echo bash -c \''echo "hello, world!";sleep 3;'\' | parallel This command does not output anything until it has completed. Parallel's man page claims: GNU parallel makes sure output from the ...
1
vote
2answers
442 views

Running multiple bash scripts with different names in parallel

I need to run a command for each individual instance of a given variable name in parallel. Sometimes, there might be 4 variables, other times there might be 100. For example, say I have this ...
1
vote
1answer
160 views

Counting failed commands when using `sem`

I want to run several shell scripts in parallel using sem and get the number of failed commands. However, their exit codes are always zero: sem bash -c 'exit 0' echo $? # => 0 sem bash -c 'exit 1'...
2
votes
1answer
727 views

gnu parallel pair argument with file input arguments

I have a file with many lines, and on each line I have the arguments I want to pass to parallel with a tab separator. I run this script cat $hdd_file | grep $ssd | parallel -C '\t' clean_and_destroy ...
2
votes
3answers
85 views

On which host does this parallel command hang?

I run this parallel command: parallel -q -j0 ssh {} 'echo {}; tmp/myscript' ::: host1 host2 host3 .... Above line shows a result for most hosts. Unfortunately the parallel command hangs on some ...
3
votes
1answer
636 views

parallelize rsync calls using gnu parallel

I am trying to parallelize my rsync calls using gnu-parallel as shown below. But whenever I run my below script, it looks like it doesn't copy anything at all. But I do see lot of rsync processes (ps ...
3
votes
2answers
737 views

Combine parallel and sequential commands

1. Summary I don't understand, how I can combine parallel and sequential commands in Linux. 2. Expected behavior Pseudocode: pip install pipenv sequential pipenv install --dev parallel task npm ...
1
vote
1answer
52 views

Keeping dirs in order with GNU Parallel

I need to copy a large amount of files into their own directories. The issue I am having is keeping them in order when I copy them with GNU parallel. For example, file_1.output gets placed in dir_19. ...
1
vote
2answers
340 views

How can I use GNU Parallel to assign values to variables?

I am trying to assign values to variables using GNU Parallel: $ parallel --dry-run --xapply a{1}={2} ::: a b c ::: 5 6 7 aa=5 ab=6 ac=7 But the execution seems not to be working OK: $ parallel --...
2
votes
1answer
174 views

Using 'parallel' to execute command on remote hosts - nothing is returned, failed logins

I am trying to execute simple 'parallel' command parallel -S server1,server2,server3 echo "Number {}: Running on \`hostname\`" ::: 1 2 3 It asks me for passwords to the three servers, but then ...
1
vote
1answer
67 views

How to print only 1 filename together with the matching pattern?

I want to print the filename/s together with the matching pattern but only once even if the pattern match has multiple occurrence in the file. E.g. I have a list of patterns; list_of_patterns.txt and ...
3
votes
1answer
209 views

Why is parallel --nice not setting the niceness?

If I run a command with nice, then I can see its process having the expected niceness value: In one terminal: nice sleep 17 and in another one: $ ps -aoni,comm | grep sleep 10 sleep But trying ...
2
votes
1answer
369 views

wget using -i with -O

I have a db of files to download containing fields for filename and download_url with the format: "foo-1.23.4.jar", "http://example.com/files/12345678/download" "bar-5.67.8.jar", "http://example.com/...
4
votes
2answers
904 views

Compute md5sum for each line in a file

Given a file like: a b c How do I get an output like: a 0cc175b9c0f1b6a831c399e269772661 b 92eb5ffee6ae2fec3ad71c777531578f c 4a8a08f09d37b73795649038408b5f33 in an efficient way?...
2
votes
1answer
354 views

Convert Webp to PDF

I want to convert a bunch of Webp images to individual PDFs. I'm able to do it with this command: parallel convert '{} {.}.pdf' ::: *.webp or I can use this FFMPEG command: find ./ -name "*.webp" -...