AMBIGUOUS TAG — DO NOT USE. For the Mac virtualization software, use [parallels]. For the `parallel` utilities, use [gnu-parallel] or [moreutils-parallel]. For parallel computing in general, use [parallelism]. For parallel ports once common on PCs, use [parallel-port].

learn more… | top users | synonyms

20
votes
2answers
20k views

Limit process to one cpu core

How to limit process to one cpu core ? Something similar to ulimit or cpulimit would be nice. (Just to ensure: I do NOT want to limit percentage usage or time of execution. I want to force app (with ...
18
votes
4answers
5k views

Four tasks in parallel… how do I do that?

I have a bunch of PNG images on a directory. I have an application called pngout that I run to compress these images. This application is called by a script I did. The problem is that this script does ...
11
votes
1answer
1k views

GNU parallel vs & (I mean background) vs xargs -P

I'm confused about the difference or advantage (if any) of running a set of tasks in a .sh script using GNU parallel E.g. Ole Tange's answer: parallel ./pngout -s0 {} R{} ::: *.png rather than say ...
11
votes
8answers
943 views

Spreading stdin to parallel processes

I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of ...
10
votes
2answers
3k views

Executing piped commands in parallel

Consider the following scenario. I have two programs A and B. Program A outputs to stdout lines of strings while program B process lines from stdin. The way to use these two programs is of course ...
9
votes
2answers
3k views

Parallelizing a for loop

I want to parallelize the for loops of the following code. How to do this? #!/bin/bash N=$1 n=$2 for (( i=1; i<=$N; i++ )); do min=100000000000000 //set min to some garbage value for (( ...
8
votes
6answers
5k views

Is there parallel wget? Something like fping but only for downloading?

I've found only puf (Parallel URL fetcher) but could to get it work with reading urls from file and something like puf < urls.txt does not work either. The operating system installed on the ...
8
votes
4answers
4k views

using parallel to process unique input files to unique output files

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
8
votes
4answers
167 views

Virtual write-only file system for storing files in archive

I have an embarrassingly parallel process that creates a huge amount of nearly (but not completely) identical files. Is there a way to archive the files "on the fly", so that the data does not consume ...
7
votes
4answers
2k views

How to run parallel processes and combine outputs when both finished

I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file. I then do the same again for a separate similar dataset ...
7
votes
2answers
2k views

How to stop xargs from badly merging output from multiple processes?

I'm using xargs with the option --max-args=0 (alternatively -P 0). However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end ...
7
votes
4answers
785 views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
7
votes
4answers
4k views

Parallel execution of a program on multiple files

I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's for file in ./folder/*; do ./bin/myProgram $file > ./done/$file ...
7
votes
1answer
570 views

Poor Man's GNU Parallel implemented in ksh?

I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across ...
6
votes
3answers
275 views

Is it safe to pipe the stdout of A to B in parallel

I'm running A file | B --params > file.txt. Since I want to accelerate the processing speed, I used the parallel -j+0< a.txt to run 20 jobs concurrently. a.txt contains all the commands: A ...
6
votes
3answers
406 views

Multiple tar processes writing to the same archive file at once

I am running many tasks on a Linux cluster. Each task creates many output files. When all tasks are finished, I run something like tar cf foo.tar output_files/to create a tar archive. This is a very ...
6
votes
3answers
2k views

Using parallel on Ubuntu

I am having an issue trying to use parallel command on Ubuntu 10.04. I looked up the parallel documentation and few of the commands seem to run. In all cases I just get the command prompt back without ...
6
votes
1answer
595 views

Running parallel bash jobs on a HPC cluster using GNU parallel

On an HPC cluster I am trying to run multiple bash scripts (permute2.sh) from 1 bash script using GNU parallel, however it doesn't complete every job. It randomly completes one job, while it is stuck ...
5
votes
3answers
867 views

How to speed up my build

I am doing a build on a Linux machine with ubuntu 10.04 on it. I want to know how can I really speed up my build. I have 4 CPUs and lots of RAM. I already reniced the process group to -20. Is there ...
5
votes
3answers
3k views

Parallelize a Bash FOR Loop

I have been trying to parallelize the following script, specifically each of the three FOR loop instances, using GNU Parallel but haven't been able to. The 4 commands contained within the FOR loop run ...
5
votes
10answers
3k views

How do I run the same linux command in more than one tab/shell simultaneously?

Is there any tool/command in Linux that I can use to run a command in more than one tab simultaneously? I want to run the same command: ./myprog argument1 argument2 simultaneously in more than one ...
5
votes
2answers
539 views

How can I parallelise the upload of a directory by FTP?

I need to upload a directory with a rather complicated tree (lots of subdirectories, etc.) by FTP. I am unable to compress this directory, since I do not have any access to the destination apart from ...
5
votes
2answers
396 views

`mpirun -np N`: what if `N` is larger than my physical cores?

Say I have a 4-core workstation, what would Linux (Ubuntu) do if I execute mpirun -np 9 XXX Will 9 run immediately together, or they will run 4 after 4? I suppose that using 9 is not good, because ...
4
votes
4answers
2k views

Remove numbers from filenames

I've a problem modifying the files' names in my Music/ directory. I have a list of names like these: $ ls 01 American Idiot.mp3 01 Articolo 31 - Domani Smetto.mp3 01 Bohemian rapsody.mp3 01 Eye of ...
4
votes
2answers
786 views

Forcing GNU make to run commands in order

With the following Makefile, GNU make runs the two commands in parallel. Since the first one takes time to finish, rm *.log is run before the log file is created, and fails. dummy.pdf: dummy.tex ...
4
votes
3answers
1k views

shell script to read from multiple files in parallel

I need to write a script that runs parallel and looks for a string in multiple files. I tried a lot of options but they slow down the speed of my processor.
4
votes
3answers
599 views

How to create a bounded queue for shell tasks?

I have 1000 gzipped files which I want to sort. Doing this sequentially, the procedure looks pretty straightforward: find . -name *.gz -exec zcat {} | sort > {}.txt \; Not sure that the code ...
4
votes
3answers
722 views

Using GNU Parallel With Split

I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU ...
4
votes
3answers
2k views

Bash, how to let some background processes run but wait for others?

I have (yet) another wait, &, && control flow question.. Say I have a script something like this where I want to do as much work at the same time as possible: # may take some hours ...
4
votes
1answer
3k views

Should I disable hyperthreading when concerned about performance of single-threaded applications?

I use a i5-2410M processor, which is setup to do hyperthreading by default on my laptop. Considering that this is a 2-core processor, this means it can do 4 threads at a time. This also means that ...
3
votes
3answers
160 views

Is the output mixed like this because of xargs and how can I fix it?

I'm learning a bit of Bash by using xargs to list whois records for a bunch of IP addresses. The command used is: echo "$1" | tr "\n" "\0" | xargs -0 -n 1 -t -P 3 -I % sh -c 'echo "\n 44rBegin whois ...
3
votes
2answers
226 views

Running up to X commands in parallel

I'm running something like this: find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do echo "Encrypting $file..." gpg --trust-model always --recipient "me@myself.com" ...
3
votes
3answers
3k views

process files in a directory as they appear [duplicate]

Possible Duplicate: How to run a command when a directory's contents are updated? I'm trying to write a simple etl process that would look for files in a directory each minute, and if ...
3
votes
3answers
492 views

How to parallelize the for loop while scp the files?

I am running my below shell script from machineA which is copying the files machineB and machineC into machineA. If the files are not there in machineB, then it should be there in machineC. The below ...
3
votes
1answer
199 views

Variable declaration in parallel sh -c …

I've been trying to process the output of find with parallel, which in turn invoked a shell (some textual substitutions were needed). I observed some strange behaviour, which I cannot really explain ...
3
votes
1answer
365 views

How can I issue parallel commands to remote nodes with different arguments?

I need to execute an application in parallel on multiple Ubuntu servers while supplying different arguments for different servers. I tried to google it, but could not get to the possible solution. I ...
3
votes
2answers
685 views

Copy multiple files to one dir with parallel

I'm using the following script to copy multiple files into one folder: { echo $BASE1; echo $BASE2; echo $BASE3; } | parallel cp -a {} $DEST Is there any way to use only one echo $BASE with brace ...
3
votes
2answers
456 views

Running jobs in parallel on Ubuntu - I/O contention differences between Perl and Java

Apologies if this is off topic - it concerns the relative efficiencies of running I/O-heavy Perl/Java scripts in parallel on a Ubuntu system. I have written two simple versions of a file copy script ...
2
votes
4answers
2k views

Parallelizing a for loop with very large number of iterations

I want to parallelize a for loop where the number of iterations in the loop can be very large such as 10^6. So ,it will be better if I can create threads rather than process. How to do it? The code is ...
2
votes
4answers
1k views

Number of processors in /proc/cpuinfo

While i was learning about cpu load i came to know that it depends on numbers of core. if i have 2 cores then load 2 will give 100% cpu utilization. So i tried to find out cores.( i already know that ...
2
votes
4answers
100 views

Parallel pausing and resuming?

I have a pretty strange thing I'm seeing as I run 8 GPG encryption jobs with GNU parallel: The command I've run is this: find . -type f -not -iname "*.gpg" | sort |parallel --gnu -j 8 --workdir ...
2
votes
2answers
79 views

Background execution in parallel

I have a program ./pgm taking some arguments (say -a file1 -b val), which requires 2 seconds to execute. I would like to use all the processors on my machine to run this program on all the input files ...
2
votes
2answers
162 views

How does a parallelized Python program look with top command?

I work on a shared cluster. I've seen people run parallelized c code on this cluster which, when I use top to see what processes are running, are shown to be using (for example) 400% of the CPU, ...
2
votes
3answers
766 views

parallel-ssh with Passphrase Protected SSH Key

I'd like to use my passphrase protected SSH key when performing tasks with parallel-ssh (and related tools). However, I can't get this to work. All the documentation, regarding parallel-ssh, show ...
2
votes
1answer
299 views

Is there a way to run process parallelly in the loop of a bash script [duplicate]

Possible Duplicate: Parallelizing a for loop The original code might look like this: for i in *; do something.py $i; done I was wondering whether I can run these jobs parallelly in the ...
2
votes
3answers
457 views

Are there any disadvantages against using qsub to run tasks all the time?

When I'm running a task on a computer network? I've just started to realize that if I qsub any task, then the task won't hog up my terminal, and I can do other things on the same terminal (which is ...
2
votes
2answers
110 views

How to read commands from file?

I have som php scripts which I run sequential order like: php index.php import file1 --offline && php index.php import file2 --deleteUnused && php index.php import file3 Now I have ...
2
votes
3answers
377 views

control number of started programs in bash

As part of my research project I'm processing huge amount of data splitted up into many files. All files in folder foo have to be processed by the script myScript involving all elements of folder ...
2
votes
2answers
207 views

Can GNU Parallel execute more parallel processes?

Can I for example execute: parallel -j 200 < list0 Where "list" has: nice -n -20 parallel -j 100 < list2 nice -n -20 parallel -j 100 < list1 Would this be feasible/possible?
2
votes
1answer
235 views

No output using parallel in tandem with ag or ack

I have a list of java reserved words, first letter capitalised. $ tail -n5 ~/reservedjava.txt Break While True False Null I'm trying to look through all my java source code to find methods that ...