AMBIGUOUS TAG — DO NOT USE. For the Mac virtualization software, use [parallels]. For the `parallel` utilities, use [gnu-parallel] or [moreutils-parallel]. For parallel computing in general, use [parallelism]. For parallel ports once common on PCs, use [parallel-port].

learn more… | top users | synonyms

1
vote
4answers
134 views

How can i run a piece of code in background?

Can I run a piece of code in background instead of using another script. [sesiv@itseelm-lx4151 ~]$ cat ./testback2 #!/bin/bash start_time=$(date +%s) for i in {1..5} do ./testscript & done wait ...
2
votes
1answer
113 views

replacement inside parallel command string

I want to download multiple files and save them to the md5 sum of its name: cat list | parallel -j4 "md5=$(wget -O - {} | tee tmpfile_{#} | md5sum | cut -d ' ' -f 1); mv tmpfile_{#} $md5" but the ...
4
votes
3answers
722 views

Using GNU Parallel With Split

I'm loading a pretty gigantic file to a postgresql database. To do this I first use split in the file to get smaller files (30Gb each) and then I load each smaller file to the database using GNU ...
1
vote
0answers
346 views

MPI mpdboot error

I have a problem with mpi in a 3-box cluster with machines named head, node1 and node2 respectively. I install a newer version on node2 and the problems arises. When I include only head and node2 in ...
6
votes
1answer
595 views

Running parallel bash jobs on a HPC cluster using GNU parallel

On an HPC cluster I am trying to run multiple bash scripts (permute2.sh) from 1 bash script using GNU parallel, however it doesn't complete every job. It randomly completes one job, while it is stuck ...
2
votes
1answer
297 views

Is there a way to run process parallelly in the loop of a bash script [duplicate]

Possible Duplicate: Parallelizing a for loop The original code might look like this: for i in *; do something.py $i; done I was wondering whether I can run these jobs parallelly in the ...
3
votes
2answers
685 views

Copy multiple files to one dir with parallel

I'm using the following script to copy multiple files into one folder: { echo $BASE1; echo $BASE2; echo $BASE3; } | parallel cp -a {} $DEST Is there any way to use only one echo $BASE with brace ...
7
votes
4answers
784 views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
11
votes
8answers
942 views

Spreading stdin to parallel processes

I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of ...
2
votes
3answers
377 views

control number of started programs in bash

As part of my research project I'm processing huge amount of data splitted up into many files. All files in folder foo have to be processed by the script myScript involving all elements of folder ...
4
votes
3answers
1k views

shell script to read from multiple files in parallel

I need to write a script that runs parallel and looks for a string in multiple files. I tried a lot of options but they slow down the speed of my processor.
3
votes
2answers
455 views

Running jobs in parallel on Ubuntu - I/O contention differences between Perl and Java

Apologies if this is off topic - it concerns the relative efficiencies of running I/O-heavy Perl/Java scripts in parallel on a Ubuntu system. I have written two simple versions of a file copy script ...
5
votes
2answers
396 views

`mpirun -np N`: what if `N` is larger than my physical cores?

Say I have a 4-core workstation, what would Linux (Ubuntu) do if I execute mpirun -np 9 XXX Will 9 run immediately together, or they will run 4 after 4? I suppose that using 9 is not good, because ...
4
votes
4answers
2k views

Remove numbers from filenames

I've a problem modifying the files' names in my Music/ directory. I have a list of names like these: $ ls 01 American Idiot.mp3 01 Articolo 31 - Domani Smetto.mp3 01 Bohemian rapsody.mp3 01 Eye of ...
7
votes
4answers
4k views

Parallel execution of a program on multiple files

I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's for file in ./folder/*; do ./bin/myProgram $file > ./done/$file ...
8
votes
6answers
5k views

Is there parallel wget? Something like fping but only for downloading?

I've found only puf (Parallel URL fetcher) but could to get it work with reading urls from file and something like puf < urls.txt does not work either. The operating system installed on the ...
18
votes
4answers
5k views

Four tasks in parallel… how do I do that?

I have a bunch of PNG images on a directory. I have an application called pngout that I run to compress these images. This application is called by a script I did. The problem is that this script does ...
3
votes
3answers
3k views

process files in a directory as they appear [duplicate]

Possible Duplicate: How to run a command when a directory's contents are updated? I'm trying to write a simple etl process that would look for files in a directory each minute, and if ...
8
votes
4answers
4k views

using parallel to process unique input files to unique output files

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
2
votes
3answers
457 views

Are there any disadvantages against using qsub to run tasks all the time?

When I'm running a task on a computer network? I've just started to realize that if I qsub any task, then the task won't hog up my terminal, and I can do other things on the same terminal (which is ...
4
votes
3answers
599 views

How to create a bounded queue for shell tasks?

I have 1000 gzipped files which I want to sort. Doing this sequentially, the procedure looks pretty straightforward: find . -name *.gz -exec zcat {} | sort > {}.txt \; Not sure that the code ...
2
votes
4answers
2k views

Parallelizing a for loop with very large number of iterations

I want to parallelize a for loop where the number of iterations in the loop can be very large such as 10^6. So ,it will be better if I can create threads rather than process. How to do it? The code is ...
9
votes
2answers
3k views

Parallelizing a for loop

I want to parallelize the for loops of the following code. How to do this? #!/bin/bash N=$1 n=$2 for (( i=1; i<=$N; i++ )); do min=100000000000000 //set min to some garbage value for (( ...
2
votes
1answer
1k views

How to run Abinit via MPI?

I'm trying to run Abinit in parallel, since in sequential mode it takes too much time. I have followed all steps in the abinit tutorials to run it in parallel, and they say to use OpenMPI, but I ...
20
votes
2answers
20k views

Limit process to one cpu core

How to limit process to one cpu core ? Something similar to ulimit or cpulimit would be nice. (Just to ensure: I do NOT want to limit percentage usage or time of execution. I want to force app (with ...
5
votes
3answers
867 views

How to speed up my build

I am doing a build on a Linux machine with ubuntu 10.04 on it. I want to know how can I really speed up my build. I have 4 CPUs and lots of RAM. I already reniced the process group to -20. Is there ...
4
votes
1answer
3k views

Should I disable hyperthreading when concerned about performance of single-threaded applications?

I use a i5-2410M processor, which is setup to do hyperthreading by default on my laptop. Considering that this is a 2-core processor, this means it can do 4 threads at a time. This also means that ...
7
votes
2answers
2k views

How to stop xargs from badly merging output from multiple processes?

I'm using xargs with the option --max-args=0 (alternatively -P 0). However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end ...
7
votes
1answer
569 views

Poor Man's GNU Parallel implemented in ksh?

I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across ...
4
votes
2answers
786 views

Forcing GNU make to run commands in order

With the following Makefile, GNU make runs the two commands in parallel. Since the first one takes time to finish, rm *.log is run before the log file is created, and fails. dummy.pdf: dummy.tex ...
6
votes
3answers
2k views

Using parallel on Ubuntu

I am having an issue trying to use parallel command on Ubuntu 10.04. I looked up the parallel documentation and few of the commands seem to run. In all cases I just get the command prompt back without ...