Running commands in parallel, with or without GNU parallel. For virtualization on Macs, see /parallels. For parallel ports, see /parallel-port.
4
votes
1answer
203 views
Running parallel bash jobs on a HPC cluster using GNU parallel
On an HPC cluster I am trying to run multiple bash scripts (permute2.sh) from 1 bash script using GNU parallel, however it doesn't complete every job. It randomly completes one job, while it is stuck ...
4
votes
3answers
737 views
shell script to read from multiple files in parallel
I need to write a script that runs parallel and looks for a string in multiple files. I tried a lot of options but they slow down the speed of my processor.
1
vote
4answers
52 views
How can i run a piece of code in background?
Can I run a piece of code in background instead of using another script.
[sesiv@itseelm-lx4151 ~]$ cat ./testback2
#!/bin/bash
start_time=$(date +%s)
for i in {1..5}
do
./testscript &
done
wait
...
3
votes
2answers
448 views
Why does (GNU?) parallel fail silently, and how do I fix it?
In a larger script to post-process some simulation data I had the following line:
parallel bnzip2 -- *.bz2
Which, if I understand parallel correctly (and I may not), should run n-core threads of ...
6
votes
4answers
377 views
Spreading stdin to parallel processes
I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of ...
2
votes
1answer
36 views
replacement inside parallel command string
I want to download multiple files and save them to the md5 sum of its name:
cat list | parallel -j4 "md5=$(wget -O - {} | tee tmpfile_{#} | md5sum | cut -d ' ' -f 1); mv tmpfile_{#} $md5"
but the ...
3
votes
4answers
181 views
Using GNU Parallel With Split
I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU ...
2
votes
6answers
2k views
Is there parallel wget? Something like fping but only for downloading?
I've found only puf (Parallel URL fetcher) but could to get it work with reading urls from file and something like
puf < urls.txt
does not work either.
The operating system installed on the ...
1
vote
0answers
122 views
MPI mpdboot error
I have a problem with mpi in a 3-box cluster with machines named head, node1 and node2 respectively. I install a newer version on node2 and the problems arises. When I include only head and node2 in ...
2
votes
3answers
179 views
Are there any disadvantages against using qsub to run tasks all the time?
When I'm running a task on a computer network? I've just started to realize that if I qsub any task, then the task won't hog up my terminal, and I can do other things on the same terminal (which is ...
7
votes
2answers
1k views
Parallelizing a for loop
I want to parallelize the for loops of the following code. How to do this?
#!/bin/bash
N=$1
n=$2
for (( i=1; i<=$N; i++ )); do
min=100000000000000 //set min to some garbage value
for (( ...
1
vote
1answer
105 views
Is there a way to run process parallelly in the loop of a bash script [duplicate]
Possible Duplicate:
Parallelizing a for loop
The original code might look like this:
for i in *; do something.py $i; done
I was wondering whether I can run these jobs parallelly in the ...
7
votes
4answers
318 views
Creating a single output stream out of three other streams produced in parallel
I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format.
This Python script is slow and CPU-bound (to a ...
3
votes
3answers
291 views
Copy multiple files to one dir with parallel
I'm using the following script to copy multiple files into one folder:
{ echo $BASE1; echo $BASE2; echo $BASE3; } | parallel cp -a {} $DEST
Is there any way to use only one echo $BASE with brace ...
3
votes
3answers
1k views
process files in a directory as they appear [duplicate]
Possible Duplicate:
How to run a command when a directory's contents are updated?
I'm trying to write a simple etl process that would look for files in a directory each minute, and if ...
2
votes
3answers
167 views
control number of started programs in bash
As part of my research project I'm processing huge amount of data splitted up into many files.
All files in folder foo have to be processed by the script myScript involving all elements of folder ...
12
votes
4answers
1k views
Four tasks in parallel… how do I do that?
I have a bunch of PNG images on a directory. I have an application called pngout that I run to compress these images. This application is called by a script I did. The problem is that this script does ...
3
votes
2answers
200 views
Running jobs in parallel on Ubuntu - I/O contention differences between Perl and Java
Apologies if this is off topic - it concerns the relative efficiencies of running I/O-heavy Perl/Java scripts in parallel on a Ubuntu system.
I have written two simple versions of a file copy script ...
3
votes
4answers
829 views
Remove numbers from filenames
I've a problem modifying the files' names in my Music/ directory.
I have a list of names like these:
$ ls
01 American Idiot.mp3
01 Articolo 31 - Domani Smetto.mp3
01 Bohemian rapsody.mp3
01 Eye of ...
4
votes
2answers
198 views
`mpirun -np N`: what if `N` is larger than my physical cores?
Say I have a 4-core workstation, what would Linux (Ubuntu) do if I execute
mpirun -np 9 XXX
Will 9 run immediately together, or they will run 4 after 4?
I suppose that using 9 is not good, because ...
6
votes
4answers
2k views
Parallel execution of a program on multiple files
I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's
for file in ./folder/*;
do
./bin/myProgram $file > ./done/$file
...
2
votes
1answer
617 views
How to run Abinit via MPI?
I'm trying to run Abinit in parallel, since in sequential mode it takes too much time. I have followed all steps in the abinit tutorials to run it in parallel, and they say to use OpenMPI, but I ...
2
votes
4answers
736 views
Parallelizing a for loop with very large number of iterations
I want to parallelize a for loop where the number of iterations in the loop can be very large such as 10^6. So ,it will be better if I can create threads rather than process. How to do it? The code is ...
3
votes
1answer
133 views
Multi-machine tool in the spirit of moreutils' `parallel`?
parallel from moreutils is a great tool for, among other things, distributing m independent tasks evenly over n CPUs. Does anybody know of a tool that accomplishes the same thing for multiple ...
6
votes
4answers
1k views
using parallel to process unique input files to unique output files
I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
3
votes
3answers
272 views
How to create a bounded queue for shell tasks?
I have 1000 gzipped files which I want to sort.
Doing this sequentially, the procedure looks pretty straightforward:
find . -name *.gz -exec zcat {} | sort > {}.txt \;
Not sure that the code ...
6
votes
2answers
6k views
Limit process to one cpu core
How to limit process to one cpu core ?
Something similar to ulimit or cpulimit would be nice. (Just to ensure: I do NOT want to limit percentage usage or time of execution. I want to force app (with ...
5
votes
3answers
374 views
How to speed up my build
I am doing a build on a Linux machine with ubuntu 10.04 on it. I want to know how can I really speed up my build. I have 4 CPUs and lots of RAM. I already reniced the process group to -20. Is there ...
2
votes
1answer
2k views
Should I disable hyperthreading when concerned about performance of single-threaded applications?
I use a i5-2410M processor, which is setup to do hyperthreading by default on my laptop. Considering that this is a 2-core processor, this means it can do 4 threads at a time. This also means that ...
5
votes
2answers
796 views
How to stop xargs from badly merging output from multiple processes?
I'm using xargs with the option --max-args=0 (alternatively -P 0).
However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end ...
5
votes
1answer
343 views
Poor Man's GNU Parallel implemented in ksh?
I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across ...
4
votes
2answers
439 views
Forcing GNU make to run commands in order
With the following Makefile, GNU make runs the two commands in parallel. Since the first one takes time to finish, rm *.log is run before the log file is created, and fails.
dummy.pdf: dummy.tex
...
3
votes
2answers
737 views
Using parallel on Ubuntu
I am having an issue trying to use parallel command on Ubuntu 10.04. I looked up the parallel documentation and few of the commands seem to run. In all cases I just get the command prompt back without ...
