5
votes
3answers
169 views

Multiple tar processes writing to the same archive file at once

I am running many tasks on a Linux cluster. Each task creates many output files. When all tasks are finished, I run something like tar cf foo.tar output_files/to create a tar archive. This is a very ...
1
vote
2answers
123 views

MaxStartups and MaxSessions configurations parameter for ssh connections?

I am copying the files from machineB and machineC into machineA as I am running my below shell script on machineA. If the files is not there in machineB then it should be there in machineC for sure ...
1
vote
0answers
258 views

How to use rsync with GNU parallel?

I am using scp to copy the files in parallel using GNU parallel with my below shell script and it is working fine. I am not sure how can I use rsync in place of scp in my below shell script. I am ...
1
vote
1answer
72 views

How to copy in two folders simultaneously using GNU parallel by spawning multiple threads?

I am trying to copy files from machineB and machineC into machineA as I am running my below shell script on machineA. If the files is not there in machineB then it should be there in machineC for ...
2
votes
3answers
87 views

How to split the array in set of five files and download them in parallel?

I am trying to copy files from testMachineB and testMachineC into testMachineA as I am running my shell script on testMachineA. If the file is not there in testMachineB, then it should be there in ...
1
vote
1answer
53 views

Can GNU Parallel execute more parallel processes?

Can I for example execute: parallel -j 200 < list0 Where "list" has: nice -n -20 parallel -j 100 < list2 nice -n -20 parallel -j 100 < list1 Would this be feasible/possible?
0
votes
1answer
35 views

Starting GNU Parallel in background?

Is it possible to start GNU Parallel in the background, or somehow, send it to the background? I want to do this so that I can use two commands that run parallel at the same time. And why dont I ...
1
vote
1answer
40 views

Why does parallel slow down after a while?

I'm executing 60 scripts with GNU parallel(they all have wget commands in there)but I have noticed that after a few hours execution will slow down a bit. What could be causing this? I'm executing ...
3
votes
3answers
204 views

How to parallelize the for loop while scp the files?

I am running my below shell script from machineA which is copying the files machineB and machineC into machineA. If the files are not there in machineB, then it should be there in machineC. The below ...
1
vote
1answer
81 views

Strategies for building distributed computing system with hybrid Mac and Linux systems

I want to build a distributed computing system to run Matlab, C and other programming languages for scientific computing. Now I've several old Mac machines with Lion Mac OS installed acted as web ...
1
vote
4answers
118 views

How can i run a piece of code in background?

Can I run a piece of code in background instead of using another script. [sesiv@itseelm-lx4151 ~]$ cat ./testback2 #!/bin/bash start_time=$(date +%s) for i in {1..5} do ./testscript & done wait ...
5
votes
2answers
351 views

`mpirun -np N`: what if `N` is larger than my physical cores?

Say I have a 4-core workstation, what would Linux (Ubuntu) do if I execute mpirun -np 9 XXX Will 9 run immediately together, or they will run 4 after 4? I suppose that using 9 is not good, because ...
5
votes
3answers
694 views

How to speed up my build

I am doing a build on a Linux machine with ubuntu 10.04 on it. I want to know how can I really speed up my build. I have 4 CPUs and lots of RAM. I already reniced the process group to -20. Is there ...
4
votes
1answer
3k views

Should I disable hyperthreading when concerned about performance of single-threaded applications?

I use a i5-2410M processor, which is setup to do hyperthreading by default on my laptop. Considering that this is a 2-core processor, this means it can do 4 threads at a time. This also means that ...