AMBIGUOUS TAG — DO NOT USE. For the Mac virtualization software, use [parallels]. For the `parallel` utilities, use [gnu-parallel] or [moreutils-parallel]. For parallel computing in general, use [parallelism]. For parallel ports once common on PCs, use [parallel-port].

learn more… | top users | synonyms

2
votes
3answers
642 views

control number of started programs in bash

As part of my research project I'm processing huge amount of data splitted up into many files. All files in folder foo have to be processed by the script myScript involving all elements of folder bar....
2
votes
2answers
860 views

Variable value is not recognized after using gnu parallel?

I have a below shell script from which I am trying to copy 5 files in parallel. I am running my below shell script on machineA which tries to copy the file from machineB and machineC. If the file is ...
2
votes
2answers
1k views

Can GNU Parallel execute more parallel processes?

Can I for example execute: parallel -j 200 < list0 Where "list" has: nice -n -20 parallel -j 100 < list2 nice -n -20 parallel -j 100 < list1 Would this be feasible/possible?
2
votes
1answer
1k views

How to run Abinit via MPI?

I'm trying to run Abinit in parallel, since in sequential mode it takes too much time. I have followed all steps in the abinit tutorials to run it in parallel, and they say to use OpenMPI, but I ...
2
votes
3answers
183 views

How to split the array in set of five files and download them in parallel?

I am trying to copy files from testMachineB and testMachineC into testMachineA as I am running my shell script on testMachineA. If the file is not there in testMachineB, then it should be there in ...
2
votes
1answer
1k views

Can I force Linux to boot its initializations parallel? [closed]

I was wondering if I could manage to initialize drivers, services etc. (all these jobs what Linux does during startup) in parallel instead of sequentially. I want to aggressively lower the boot time. ...
2
votes
1answer
215 views

replacement inside parallel command string

I want to download multiple files and save them to the md5 sum of its name: cat list | parallel -j4 "md5=$(wget -O - {} | tee tmpfile_{#} | md5sum | cut -d ' ' -f 1); mv tmpfile_{#} $md5" but the ...
2
votes
1answer
172 views

Find | parallel executing script with path from find + other arguments

Due to my lack of Bash knowledge I've tried for a few hours now to get something like this to work: find Directories -mindepth 4 -type d -print0 | parallel -0 -j0 ./MyScript -d {Found Directory} {1} :...
2
votes
1answer
612 views

How to get exit code of a parallel process launched by xargs?

I'd made a script for launching parallel rsync process: #! /bin/bash LIST=$1 DEST_DIR=$2 RSYNC_OPTS=$3 #echo "rsyncing From=$SRC_DIR To=$DEST_DIR RSYNC_OPTS=$RSYNC_OPTS" echo $LIST|xargs -n1 -d, echo|...
2
votes
2answers
1k views

scp does not display output when used with gnu parallel

I have a short script that uses scp to copy files to a number of remote hosts (yes, I know about rdist and rsync; they both fail to work for a few of the hosts - that's not the point here; I'm only ...
1
vote
1answer
7k views

Is it possible to run two commands at the same time in a shell script?

I know that on the command line I can use & to run a command in the background. But I'm wondering if I can do it in a script. I have a script like this: date_stamp=$(date +"%Y-%m-%d" --date='...
1
vote
2answers
46 views

GNU Parallel and sshpass with server list in a loop

With this loop we sequential update on all servers (server list = consul members | grep awk {'print $2'} | cut -d ":" -f1) the package consul. for i in $(consul members | grep awk {'print $2'} | cut -...
1
vote
2answers
199 views

How can I count the time that needs systemd to boot a default target and then graph it?

When I wanted to know how long systemd actually needed to boot the default target, how would I do that ? An then, is it possible to create a graph to show which unit takes how much time to initialize ...
1
vote
1answer
28 views

submitting jobs to get 3 nodes running parallel executions

I have a submit script looks like below, it tries to run a large number of instances of csce.py in backgrounds with 3 nodes.... in a laptop, this usually could successfully automatically distribute ...
1
vote
1answer
241 views

How to copy in two folders simultaneously using GNU parallel by spawning multiple threads?

I am trying to copy files from machineB and machineC into machineA as I am running my below shell script on machineA. If the files is not there in machineB then it should be there in machineC for ...
1
vote
3answers
501 views

running an executable in parallel by dividing the input directory tree

Commands like find and grep would benefit greatly if they were to run in parallel. Is there a utility or a program that takes a command as an argument and makes it run in parallel by running multiple ...
1
vote
1answer
492 views

Is there a simple way of limiting the number of child processes that execute in parallel?

I have a bash script that kicks off multiple rsync processes in parallel. I'd like to limit the number of simultaneous child processes but not serialize the whole script. Here's a simplified version ...
1
vote
4answers
189 views

How can i run a piece of code in background?

Can I run a piece of code in background instead of using another script. [sesiv@itseelm-lx4151 ~]$ cat ./testback2 #!/bin/bash start_time=$(date +%s) for i in {1..5} do ./testscript & done wait ...
1
vote
2answers
98 views

Run multiple scripts and continue until every script is done

my question is, can I run multiple scripts at once. I have access to a cluster with 32 cpus but at the same time I have 900 scripts. Now what I would like to do is, put every script in a queue, and ...
1
vote
1answer
62 views

Is it safe to resize a partition while writing to it?

I'm working on a script for automatically setting up Amazon Linux servers. I create them with 100gb virtual disks, but the main partition is always 8gb. No problem, I call sudo resize2fs /dev/sda1 at ...
1
vote
1answer
1k views

Rsync from multiple folders to one

Is it safe to have multiple folders Rsync simultaneously into one? Each of the source folders contains a subset of the destination folder, and everybody just syncs into a master destination folder. I ...
1
vote
2answers
65 views

Need to format 5 disks in parallel

I tried this xargs -p 5 -n1 mke2fs -t ext4 /dev/sd[d-p]1 and got this [/dev/mapper]# seq 1 15 | xargs -p 5 -n 1 mke2fs -t ext4 /dev/sd[b-p]1 5 -n 1 mke2fs -t ext4 /dev/sdb1 /dev/sdc1 /dev/sdd1 /...
1
vote
1answer
452 views

Few questions about system calls and kernel modules (kernel services in parallel)

I have few questions about system calls and kernel modules. Let's assume we have two applications (A and B) and each of them is running on a different core. (A-CPU 0, B-CPU 1) When two applications ...
1
vote
1answer
602 views

How to launch multiple instances of a program in parallel using parallel?

I would like to execute a command: curl example.com multiple times in parallel using GNU parallel. I tried: parallel -j 3 curl example.com but it doesn't work.
1
vote
1answer
86 views

Wine and parallel processing

I'm quite new to linux, right now, running LMDE on my machine. I was wondering if wine can support the parallel processing needs of some software. Anyone knows any performance benchmark of wine ? ...
1
vote
2answers
238 views

Starting GNU Parallel in background?

Is it possible to start GNU Parallel in the background, or somehow, send it to the background? I want to do this so that I can use two commands that run parallel at the same time. And why dont I ...
1
vote
1answer
202 views

Why does parallel slow down after a while?

I'm executing 60 scripts with GNU parallel(they all have wget commands in there)but I have noticed that after a few hours execution will slow down a bit. What could be causing this? I'm executing ...
1
vote
1answer
587 views

Gnu Parallel: more than one per CPU? [closed]

If I only have two cores, does parallel only run two processes at a time? I guess I'm still confused about processes, threads, and cores....
1
vote
1answer
77 views

Memory-efficient utility to return N first sorted values

I would like to implement a very popular MapReduce example using only existing programs operating in a UNIX way. The problem is to find N most frequent values in an enormous amount of data. The ...
1
vote
0answers
87 views

GNU parallel hangs when using --load 100% argument

On a machine with RH5 (Scientific Linux SL release 5.1), I have no issues with using the '--load 100%' argument. On a machine with RH6 (Scientific Linux release 6.2), the parallel hangs after printing ...
1
vote
1answer
343 views

Strategies for building distributed computing system with hybrid Mac and Linux systems

I want to build a distributed computing system to run Matlab, C and other programming languages for scientific computing. Now I've several old Mac machines with Lion Mac OS installed acted as web ...
1
vote
0answers
520 views

MPI mpdboot error

I have a problem with mpi in a 3-box cluster with machines named head, node1 and node2 respectively. I install a newer version on node2 and the problems arises. When I include only head and node2 in ...
0
votes
2answers
317 views

Why does GNU Parallel slow down?

I'm learning GNU parallel and tried the following: $ for i in {1.txt,2.txt}; do time wc -l $i; done 100 1.txt real 0m0.010s user 0m0.000s sys 0m0.010s 10000012 2.txt real 0m0.069s user ...
0
votes
1answer
89 views

Parallelize openssl as input to dd

I am using @tremby's great idea to fill a disk with random data. This involves piping openssl, which is encrypting a lot of zeros, to dd (bs=4M). I'm maxing out the single core on which this is ...
0
votes
1answer
501 views

Running shell script on cluster in parralel

I thought dsh(Distributed Shel) will be a good option but when i ran my shell script across my nodes, I didnt get expected output, dsh -aM -c bash /home/cloudera/bash_script.sh server2@x.x.x.110: ...
0
votes
0answers
12 views

Running a function in parallel (in python) on clusters using different screens [migrated]

I have 2 functions which I need to run them in parallel written in python. The issue is that I'm running the code on the cluster, so first I open a screen and then I execute the python code. if ...
0
votes
1answer
198 views

ERROR: cube viewer is not available!

I have installed and configured the following: cube-4.2.1 opari2-1.1.1 papi-5.2.0 scalasca-2.0 scorep-1.2.1 I am trying to run scalasca -examine, but I get the following error: ERROR: cube viewer ...
0
votes
0answers
170 views

log4cxx/logger.h or libcql/cql.hpp fatal error while doing make -j4

I am running make -j4 and I am getting below exception... userName@phx5qa01c-4e23:~/build$ make -j4 [ 1%] [ 2%] Building CXX object nda/common/loadtest/CMakeFiles/loadtester.dir/LoadTester.cc.o ...