Running commands in parallel, with or without GNU parallel. For virtualization on Macs, see /parallels. For parallel ports, see /parallel-port.

learn more… | top users | synonyms

3
votes
3answers
2k views

process files in a directory as they appear [duplicate]

Possible Duplicate: How to run a command when a directory's contents are updated? I'm trying to write a simple etl process that would look for files in a directory each minute, and if ...
4
votes
1answer
172 views

Multi-machine tool in the spirit of moreutils' `parallel`?

parallel from moreutils is a great tool for, among other things, distributing m independent tasks evenly over n CPUs. Does anybody know of a tool that accomplishes the same thing for multiple ...
6
votes
4answers
2k views

using parallel to process unique input files to unique output files

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
2
votes
3answers
301 views

Are there any disadvantages against using qsub to run tasks all the time?

When I'm running a task on a computer network? I've just started to realize that if I qsub any task, then the task won't hog up my terminal, and I can do other things on the same terminal (which is ...
4
votes
3answers
417 views

How to create a bounded queue for shell tasks?

I have 1000 gzipped files which I want to sort. Doing this sequentially, the procedure looks pretty straightforward: find . -name *.gz -exec zcat {} | sort > {}.txt \; Not sure that the code ...
2
votes
4answers
1k views

Parallelizing a for loop with very large number of iterations

I want to parallelize a for loop where the number of iterations in the loop can be very large such as 10^6. So ,it will be better if I can create threads rather than process. How to do it? The code is ...
9
votes
2answers
2k views

Parallelizing a for loop

I want to parallelize the for loops of the following code. How to do this? #!/bin/bash N=$1 n=$2 for (( i=1; i<=$N; i++ )); do min=100000000000000 //set min to some garbage value for (( ...
2
votes
1answer
889 views

How to run Abinit via MPI?

I'm trying to run Abinit in parallel, since in sequential mode it takes too much time. I have followed all steps in the abinit tutorials to run it in parallel, and they say to use OpenMPI, but I ...
13
votes
2answers
14k views

Limit process to one cpu core

How to limit process to one cpu core ? Something similar to ulimit or cpulimit would be nice. (Just to ensure: I do NOT want to limit percentage usage or time of execution. I want to force app (with ...
5
votes
3answers
592 views

How to speed up my build

I am doing a build on a Linux machine with ubuntu 10.04 on it. I want to know how can I really speed up my build. I have 4 CPUs and lots of RAM. I already reniced the process group to -20. Is there ...
4
votes
1answer
3k views

Should I disable hyperthreading when concerned about performance of single-threaded applications?

I use a i5-2410M processor, which is setup to do hyperthreading by default on my laptop. Considering that this is a 2-core processor, this means it can do 4 threads at a time. This also means that ...
7
votes
2answers
1k views

How to stop xargs from badly merging output from multiple processes?

I'm using xargs with the option --max-args=0 (alternatively -P 0). However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end ...
7
votes
1answer
469 views

Poor Man's GNU Parallel implemented in ksh?

I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across ...
4
votes
2answers
604 views

Forcing GNU make to run commands in order

With the following Makefile, GNU make runs the two commands in parallel. Since the first one takes time to finish, rm *.log is run before the log file is created, and fails. dummy.pdf: dummy.tex ...
5
votes
3answers
1k views

Using parallel on Ubuntu

I am having an issue trying to use parallel command on Ubuntu 10.04. I looked up the parallel documentation and few of the commands seem to run. In all cases I just get the command prompt back without ...