Tagged Questions

AMBIGUOUS TAG — DO NOT USE. For the Mac virtualization software, use [parallels]. For the `parallel` utilities, use [gnu-parallel] or [moreutils-parallel]. For parallel computing in general, use [parallelism]. For parallel ports once common on PCs, use [parallel-port].

learn more… | top users | synonyms

-1
votes
1answer
71 views

Is it possible to run two commands at the same time in a shell script?

I know that on the command line I can use & to run a command in the background. But I'm wondering if I can do it in a script. I have a script like this: date_stamp=$(date +"%Y-%m-%d" ...
1
vote
1answer
70 views

How to launch multiple instances of a program in parallel using parallel?

I would like to execute a command: curl example.com multiple times in parallel using GNU parallel. I tried: parallel -j 3 curl example.com but it doesn't work.
1
vote
2answers
33 views

How can I count the time that needs systemd to boot a default target and then graph it?

When I wanted to know how long systemd actually needed to boot the default target, how would I do that ? An then, is it possible to create a graph to show which unit takes how much time to initialize ...
0
votes
1answer
91 views

Running shell script on cluster in parralel

I thought dsh(Distributed Shel) will be a good option but when i ran my shell script across my nodes, I didnt get expected output, dsh -aM -c bash /home/cloudera/bash_script.sh server2@x.x.x.110: ...
1
vote
1answer
22 views

Wine and parallel processing

I'm quite new to linux, right now, running LMDE on my machine. I was wondering if wine can support the parallel processing needs of some software. Anyone knows any performance benchmark of wine ? ...
1
vote
1answer
1k views

top command on multi core processor

I am using freescale IMX6 quad processor. I want to know if the top command lists the CPU usage of all 4 cores or of a single core. I am seeing an application's CPU usage being the same with 4 cores ...
2
votes
4answers
740 views

Number of processors in /proc/cpuinfo

While i was learning about cpu load i came to know that it depends on numbers of core. if i have 2 cores then load 2 will give 100% cpu utilization. So i tried to find out cores.( i already know that ...
3
votes
3answers
148 views

Is the output mixed like this because of xargs and how can I fix it?

I'm learning a bit of Bash by using xargs to list whois records for a bunch of IP addresses. The command used is: echo "$1" | tr "\n" "\0" | xargs -0 -n 1 -t -P 3 -I % sh -c 'echo "\n 44rBegin whois ...
2
votes
2answers
100 views

How to read commands from file?

I have som php scripts which I run sequential order like: php index.php import file1 --offline && php index.php import file2 --deleteUnused && php index.php import file3 Now I have ...
6
votes
3answers
372 views

Multiple tar processes writing to the same archive file at once

I am running many tasks on a Linux cluster. Each task creates many output files. When all tasks are finished, I run something like tar cf foo.tar output_files/to create a tar archive. This is a very ...
2
votes
2answers
156 views

How does a parallelized Python program look with top command?

I work on a shared cluster. I've seen people run parallelized c code on this cluster which, when I use top to see what processes are running, are shown to be using (for example) 400% of the CPU, ...
1
vote
2answers
2k views

MaxStartups and MaxSessions configurations parameter for ssh connections?

I am copying the files from machineB and machineC into machineA as I am running my below shell script on machineA. If the files is not there in machineB then it should be there in machineC for sure ...
6
votes
3answers
273 views

Is it safe to pipe the stdout of A to B in parallel

I'm running A file | B --params > file.txt. Since I want to accelerate the processing speed, I used the parallel -j+0< a.txt to run 20 jobs concurrently. a.txt contains all the commands: A ...
2
votes
0answers
892 views

How to use rsync with GNU parallel?

I am using scp to copy the files in parallel using GNU parallel with my below shell script and it is working fine. I am not sure how can I use rsync in place of scp in my below shell script. I am ...
1
vote
1answer
102 views

How to copy in two folders simultaneously using GNU parallel by spawning multiple threads?

I am trying to copy files from machineB and machineC into machineA as I am running my below shell script on machineA. If the files is not there in machineB then it should be there in machineC for ...
2
votes
3answers
623 views

parallel-ssh with Passphrase Protected SSH Key

I'd like to use my passphrase protected SSH key when performing tasks with parallel-ssh (and related tools). However, I can't get this to work. All the documentation, regarding parallel-ssh, show ...
1
vote
2answers
231 views

Variable value is not recognized after using gnu parallel?

I have a below shell script from which I am trying to copy 5 files in parallel. I am running my below shell script on machineA which tries to copy the file from machineB and machineC. If the file is ...
2
votes
3answers
108 views

How to split the array in set of five files and download them in parallel?

I am trying to copy files from testMachineB and testMachineC into testMachineA as I am running my shell script on testMachineA. If the file is not there in testMachineB, then it should be there in ...
1
vote
1answer
160 views

How to get exit code of a parallel process launched by xargs?

I'd made a script for launching parallel rsync process: #! /bin/bash LIST=$1 DEST_DIR=$2 RSYNC_OPTS=$3 #echo "rsyncing From=$SRC_DIR To=$DEST_DIR RSYNC_OPTS=$RSYNC_OPTS" echo $LIST|xargs -n1 -d, ...
2
votes
2answers
182 views

Can GNU Parallel execute more parallel processes?

Can I for example execute: parallel -j 200 < list0 Where "list" has: nice -n -20 parallel -j 100 < list2 nice -n -20 parallel -j 100 < list1 Would this be feasible/possible?
1
vote
1answer
54 views

Starting GNU Parallel in background?

Is it possible to start GNU Parallel in the background, or somehow, send it to the background? I want to do this so that I can use two commands that run parallel at the same time. And why dont I ...
1
vote
1answer
70 views

Why does parallel slow down after a while?

I'm executing 60 scripts with GNU parallel(they all have wget commands in there)but I have noticed that after a few hours execution will slow down a bit. What could be causing this? I'm executing ...
4
votes
3answers
2k views

Bash, how to let some background processes run but wait for others?

I have (yet) another wait, &, && control flow question.. Say I have a script something like this where I want to do as much work at the same time as possible: # may take some hours ...
7
votes
4answers
2k views

How to run parallel processes and combine outputs when both finished

I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file. I then do the same again for a separate similar dataset ...
8
votes
4answers
165 views

Virtual write-only file system for storing files in archive

I have an embarrassingly parallel process that creates a huge amount of nearly (but not completely) identical files. Is there a way to archive the files "on the fly", so that the data does not consume ...
2
votes
2answers
78 views

Background execution in parallel

I have a program ./pgm taking some arguments (say -a file1 -b val), which requires 2 seconds to execute. I would like to use all the processors on my machine to run this program on all the input files ...
3
votes
3answers
428 views

How to parallelize the for loop while scp the files?

I am running my below shell script from machineA which is copying the files machineB and machineC into machineA. If the files are not there in machineB, then it should be there in machineC. The below ...
1
vote
1answer
222 views

Gnu Parallel: more than one per CPU? [closed]

If I only have two cores, does parallel only run two processes at a time? I guess I'm still confused about processes, threads, and cores....
1
vote
1answer
38 views

Is it safe to resize a partition while writing to it?

I'm working on a script for automatically setting up Amazon Linux servers. I create them with 100gb virtual disks, but the main partition is always 8gb. No problem, I call sudo resize2fs /dev/sda1 at ...
0
votes
3answers
101 views

running an executable in parallel by dividing the input directory tree

Commands like find and grep would benefit greatly if they were to run in parallel. Is there a utility or a program that takes a command as an argument and makes it run in parallel by running multiple ...
2
votes
1answer
422 views

Can I force Linux to boot its initializations parallel? [closed]

I was wondering if I could manage to initialize drivers, services etc. (all these jobs what Linux does during startup) in parallel instead of sequentially. I want to aggressively lower the boot time. ...
5
votes
2answers
481 views

How can I parallelise the upload of a directory by FTP?

I need to upload a directory with a rather complicated tree (lots of subdirectories, etc.) by FTP. I am unable to compress this directory, since I do not have any access to the destination apart from ...
0
votes
1answer
97 views

GNU Parallel slows down

I'm learning GNU parallel usage and did the following: $ for i in {1.txt,2.txt}; do time wc -l $i; done 100 1.txt real 0m0.010s user 0m0.000s sys 0m0.010s 10000012 2.txt real 0m0.069s ...
1
vote
1answer
55 views

Memory-efficient utility to return N first sorted values

I would like to implement a very popular MapReduce example using only existing programs operating in a UNIX way. The problem is to find N most frequent values in an enormous amount of data. The ...
2
votes
1answer
232 views

No output using parallel in tandem with ag or ack

I have a list of java reserved words, first letter capitalised. $ tail -n5 ~/reservedjava.txt Break While True False Null I'm trying to look through all my java source code to find methods that ...
0
votes
1answer
97 views

ERROR: cube viewer is not available!

I have installed and configured the following: cube-4.2.1 opari2-1.1.1 papi-5.2.0 scalasca-2.0 scorep-1.2.1 I am trying to run scalasca -examine, but I get the following error: ERROR: cube viewer ...
2
votes
4answers
94 views

Parallel pausing and resuming?

I have a pretty strange thing I'm seeing as I run 8 GPG encryption jobs with GNU parallel: The command I've run is this: find . -type f -not -iname "*.gpg" | sort |parallel --gnu -j 8 --workdir ...
11
votes
1answer
1k views

GNU parallel vs & (I mean background) vs xargs -P

I'm confused about the difference or advantage (if any) of running a set of tasks in a .sh script using GNU parallel E.g. Ole Tange's answer: parallel ./pngout -s0 {} R{} ::: *.png rather than say ...
5
votes
3answers
3k views

Parallelize a Bash FOR Loop

I have been trying to parallelize the following script, specifically each of the three FOR loop instances, using GNU Parallel but haven't been able to. The 4 commands contained within the FOR loop run ...
2
votes
2answers
475 views

scp does not display output when used with gnu parallel

I have a short script that uses scp to copy files to a number of remote hosts (yes, I know about rdist and rsync; they both fail to work for a few of the hosts - that's not the point here; I'm only ...
0
votes
0answers
78 views

log4cxx/logger.h or libcql/cql.hpp fatal error while doing make -j4

I am running make -j4 and I am getting below exception... userName@phx5qa01c-4e23:~/build$ make -j4 [ 1%] [ 2%] Building CXX object nda/common/loadtest/CMakeFiles/loadtester.dir/LoadTester.cc.o ...
1
vote
1answer
192 views

Is there a simple way of limiting the number of child processes that execute in parallel?

I have a bash script that kicks off multiple rsync processes in parallel. I'd like to limit the number of simultaneous child processes but not serialize the whole script. Here's a simplified version ...
1
vote
1answer
123 views

Strategies for building distributed computing system with hybrid Mac and Linux systems

I want to build a distributed computing system to run Matlab, C and other programming languages for scientific computing. Now I've several old Mac machines with Lion Mac OS installed acted as web ...
3
votes
1answer
350 views

How can I issue parallel commands to remote nodes with different arguments?

I need to execute an application in parallel on multiple Ubuntu servers while supplying different arguments for different servers. I tried to google it, but could not get to the possible solution. I ...
4
votes
10answers
3k views

How do I run the same linux command in more than one tab/shell simultaneously?

Is there any tool/command in Linux that I can use to run a command in more than one tab simultaneously? I want to run the same command: ./myprog argument1 argument2 simultaneously in more than one ...
0
votes
1answer
778 views

Rsync from multiple folders to one

Is it safe to have multiple folders Rsync simultaneously into one? Each of the source folders contains a subset of the destination folder, and everybody just syncs into a master destination folder. I ...
3
votes
1answer
191 views

Variable declaration in parallel sh -c …

I've been trying to process the output of find with parallel, which in turn invoked a shell (some textual substitutions were needed). I observed some strange behaviour, which I cannot really explain ...
3
votes
2answers
212 views

Running up to X commands in parallel

I'm running something like this: find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do echo "Encrypting $file..." gpg --trust-model always --recipient "me@myself.com" ...
9
votes
2answers
2k views

Executing piped commands in parallel

Consider the following scenario. I have two programs A and B. Program A outputs to stdout lines of strings while program B process lines from stdin. The way to use these two programs is of course ...
1
vote
4answers
129 views

How can i run a piece of code in background?

Can I run a piece of code in background instead of using another script. [sesiv@itseelm-lx4151 ~]$ cat ./testback2 #!/bin/bash start_time=$(date +%s) for i in {1..5} do ./testscript & done wait ...