Questions tagged [parallelism]

Performing tasks in parallel, to make use of multiple processors

Filter by
Sorted by
Tagged with
48
votes
15answers
200k views

Automatically run commands over SSH on many servers

There is a list of IP addresses in a .txt file, ex.: 1.1.1.1 2.2.2.2 3.3.3.3 Behind every IP address there is a server, and on every server there is an sshd running on port 22. Not every server is ...
23
votes
4answers
16k views

Four tasks in parallel… how do I do that?

I have a bunch of PNG images on a directory. I have an application called pngout that I run to compress these images. This application is called by a script I did. The problem is that this script does ...
40
votes
5answers
38k views

How to determine the maximum number to pass to make -j option?

I want to compile as fast as possible. Go figure. And would like to automate the choice of the number following the -j option. How can I programmatically choose that value, e.g. in a shell script? ...
39
votes
2answers
3k views

How do I distribute a large download over multiple computers?

I need to download a large file (1GB). I also have access to multiple computers running Linux, but each is limited to a 50kB/s download speed by an admin policy. How do I distribute downloading this ...
10
votes
2answers
11k views

How to parallelize dd?

I'm currently having trouble with dd invoked with a sparse file as input (if) and a file as output (of) with conv=sparse. dd seems to be using one core of the CPU (Intel(R) Core(TM) i7-3632QM CPU @ 2....
9
votes
2answers
6k views

Parallelizing a for loop

I want to parallelize the for loops of the following code. How to do this? #!/bin/bash N=$1 n=$2 for (( i=1; i<=$N; i++ )); do min=100000000000000 //set min to some garbage value for (( j=1;...
7
votes
4answers
9k views

Parallel execution of a program on multiple files

I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's for file in ./folder/*; do ./bin/myProgram $file > ./done/$file ...
3
votes
2answers
3k views

Executing commands consequtively on multiple folders

I have parent folder "parent". Inside this folder I have subfolders, and a file named "names.txt". this file includes the names of these subfolders as follow: Parent_folder folder1 folder2 folder3 ...
6
votes
3answers
5k views

tarring in parallel

An Oceanographer friend at work needs to back up many months worth of data. She is overwhelmed so I volunteered to do it. There are hundreds of directories to be backed up and we want to tar/bzip ...
42
votes
15answers
71k views

How do I run the same linux command in more than one tab/shell simultaneously?

Is there any tool/command in Linux that I can use to run a command in more than one tab simultaneously? I want to run the same command: ./myprog argument1 argument2 simultaneously in more than one ...
20
votes
8answers
94k views

Calling multiple bash scripts and running them in parallel, not in sequence

Suppose that I have three (or more) bash scripts: script1.sh, script2.sh, and script3.sh. I would like to call all three of these scripts and run them in parallel. One way to do this is to just ...
5
votes
2answers
4k views

How to run x instances of a script parallel?

I have script I'd always like to run 'x' instances in parallel. The code looks a like that: for A in do for B in do (script1.sh $A $B;script2.sh $A $B) & done #B done #A The scripts ...
4
votes
5answers
11k views

Remove numbers from filenames

I've a problem modifying the files' names in my Music/ directory. I have a list of names like these: $ ls 01 American Idiot.mp3 01 Articolo 31 - Domani Smetto.mp3 01 Bohemian rapsody.mp3 01 Eye of ...
1
vote
2answers
338 views

Processing command parallel per batch

So, I have 10 CPU core and 20 data to process. I want to process the data in parallel but I am afraid if I just process 20 at once it will make some problem. So, I want to process 10 data 2 times. Is ...
42
votes
3answers
36k views

How to start multi-threaded grep in terminal?

I have a folder which has 250+ files of 2 GB each. I need to search for a string/pattern in those files and output the result in an output file. I know I can run the following command, but it is too ...
17
votes
2answers
18k views

Executing piped commands in parallel

Consider the following scenario. I have two programs A and B. Program A outputs to stdout lines of strings while program B process lines from stdin. The way to use these two programs is of course: ...
18
votes
6answers
25k views

using parallel to process unique input files to unique output files

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
25
votes
7answers
10k views

Simultaneously calculate multiple digests (md5, sha256)?

Under the assumption that disk I/O and free RAM is a bottleneck (while CPU time is not the limitation), does a tool exist that can calculate multiple message digests at once? I am particularly ...
13
votes
9answers
2k views

Spreading stdin to parallel processes

I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of ...
9
votes
1answer
7k views

Using only one cpu core

I need to run performance tests for my concurrent program and my requirement is that it should be run on only one CPU core. (I don't want to cooperative threads - I want always have a context ...
4
votes
2answers
4k views

How to run groups of commands in parallel?

I am trying to run groups of commands in parallel. There are 10 groups and each group has 3 commands. I want sequential execution within each group and parallel execution of the 10 groups. I am trying ...
16
votes
5answers
2k views

How to run scripts in parallel on a remote machine?

I can ssh into a remote machine that has 64 cores. Lets say I need to run 640 shell scripts in parallel on this machine. How do I do this? I can see splitting the 640 scripts into 64 groups each of ...
10
votes
1answer
39k views

Is it possible to run two commands at the same time in a shell script?

I know that on the command line I can use & to run a command in the background. But I'm wondering if I can do it in a script. I have a script like this: date_stamp=$(date +"%Y-%m-%d" --date='...
7
votes
1answer
2k views

Sort large CSV files (90GB), Disk quota exceeded

Here is what I do right now, sort -T /some_dir/ --parallel=4 -uo file_sort.csv -k 1,3 file_unsort.csv the file is 90GB,I got this error message sort: close failed: /some_dir/sortmdWWn4: Disk quota ...
5
votes
2answers
4k views

split a file, pass each piece as a param to a script, run each script in parallel

I have a words.txt with 10000 words (one to a line). I have 5,000 documents. I want to see which documents contain which of those words (with a regex pattern around the word). I have a script.sh that ...
8
votes
2answers
1k views

Poor Man's GNU Parallel implemented in ksh?

I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across ...
4
votes
1answer
4k views

Should I disable hyperthreading when concerned about performance of single-threaded applications?

I use a i5-2410M processor, which is setup to do hyperthreading by default on my laptop. Considering that this is a 2-core processor, this means it can do 4 threads at a time. This also means that ...
2
votes
2answers
310 views

Background execution in parallel

I have a program ./pgm taking some arguments (say -a file1 -b val), which requires 2 seconds to execute. I would like to use all the processors on my machine to run this program on all the input files ...
5
votes
2answers
107 views

Running commands at once

I have text file contain the following commands command1 file1_input; command2 file1_output command1 file2_input; command2 file2_output command1 file3_input; command2 file3_output command1 ...
4
votes
1answer
2k views

Can tar archive files in parallel?

I'm trying to move parts of a large directory (~40 GiB and ~8 million files) across multiple machines via Amazon S3, and due to needing to preserve symlinks I am tarring up the directory and then ...
6
votes
1answer
7k views

Is there a way to run process parallelly in the loop of a bash script [duplicate]

Possible Duplicate: Parallelizing a for loop The original code might look like this: for i in *; do something.py $i; done I was wondering whether I can run these jobs parallelly in the ...
4
votes
3answers
2k views

Running up to X commands in parallel

I'm running something like this: find . -maxdepth 1 -type f -note -iname "*.gpg" | sort | while read file ; do echo "Encrypting $file..." gpg --trust-model always --recipient "me@myself.com" -...
2
votes
1answer
4k views

Multi processing / Multi threading in BASH

I have a test file that looks like this 5002 2014-11-24 12:59:37.112 2014-11-24 12:59:37.112 0.000 UDP ...... 23.234.22.106 48104 101 0 0 8.8.8.8 53 68.0 1.0 1 0.0 0 68 0 48 Each line contains a ...
2
votes
3answers
3k views

Execute command on multiple files matching a pattern in parallel

Let's say I have a command accepting a single argument which is a file path: mycommand myfile.txt Now I want to execute this command over multiple files in parallel, more specifically, file matching ...
1
vote
1answer
1k views

tar in parallel [duplicate]

I have a directory with several hundred sub-directories. I want to tar and compress each sub-directory and name the resulting file <currentDirName>.tar.bz2. I have done: find ./dir -type f -...