Questions tagged [parallelism]

Performing tasks in parallel, to make use of multiple processors

Filter by
Sorted by
Tagged with
54
votes
2answers
193k views

top command on multi core processor

I am using freescale IMX6 quad processor. I want to know if the top command lists the CPU usage of all 4 cores or of a single core. I am seeing an application's CPU usage being the same with 4 cores ...
48
votes
15answers
200k views

Automatically run commands over SSH on many servers

There is a list of IP addresses in a .txt file, ex.: 1.1.1.1 2.2.2.2 3.3.3.3 Behind every IP address there is a server, and on every server there is an sshd running on port 22. Not every server is ...
42
votes
15answers
71k views

How do I run the same linux command in more than one tab/shell simultaneously?

Is there any tool/command in Linux that I can use to run a command in more than one tab simultaneously? I want to run the same command: ./myprog argument1 argument2 simultaneously in more than one ...
42
votes
3answers
36k views

How to start multi-threaded grep in terminal?

I have a folder which has 250+ files of 2 GB each. I need to search for a string/pattern in those files and output the result in an output file. I know I can run the following command, but it is too ...
40
votes
5answers
38k views

How to determine the maximum number to pass to make -j option?

I want to compile as fast as possible. Go figure. And would like to automate the choice of the number following the -j option. How can I programmatically choose that value, e.g. in a shell script? ...
39
votes
2answers
3k views

How do I distribute a large download over multiple computers?

I need to download a large file (1GB). I also have access to multiple computers running Linux, but each is limited to a 50kB/s download speed by an admin policy. How do I distribute downloading this ...
25
votes
7answers
10k views

Simultaneously calculate multiple digests (md5, sha256)?

Under the assumption that disk I/O and free RAM is a bottleneck (while CPU time is not the limitation), does a tool exist that can calculate multiple message digests at once? I am particularly ...
23
votes
4answers
16k views

Four tasks in parallel… how do I do that?

I have a bunch of PNG images on a directory. I have an application called pngout that I run to compress these images. This application is called by a script I did. The problem is that this script does ...
21
votes
2answers
6k views

How to stop xargs from badly merging output from multiple processes?

I'm using xargs with the option --max-args=0 (alternatively -P 0). However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end ...
20
votes
8answers
94k views

Calling multiple bash scripts and running them in parallel, not in sequence

Suppose that I have three (or more) bash scripts: script1.sh, script2.sh, and script3.sh. I would like to call all three of these scripts and run them in parallel. One way to do this is to just ...
18
votes
4answers
2k views

Are basic POSIX utilities parallelized?

In a common Linux distribution, do utilities like rm, mv, ls, grep, wc, etc. run in parallel on their arguments? In other words, if I grep a huge file on a 32-threaded CPU, will it go faster than on ...
18
votes
6answers
25k views

using parallel to process unique input files to unique output files

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to ...
17
votes
4answers
20k views

How to run parallel processes and combine outputs when both finished

I have a bash shell script in which I pipe some data through about 5 or 6 different programs then the final results into a tab delimited file. I then do the same again for a separate similar dataset ...
17
votes
2answers
18k views

Executing piped commands in parallel

Consider the following scenario. I have two programs A and B. Program A outputs to stdout lines of strings while program B process lines from stdin. The way to use these two programs is of course: ...
16
votes
6answers
16k views

Is there parallel wget? Something like fping but only for downloading?

I've found only puf (Parallel URL fetcher) but I couldn't get it to read urls from a file; something like puf < urls.txt does not work either. The operating system installed on the server is ...
16
votes
5answers
2k views

How to run scripts in parallel on a remote machine?

I can ssh into a remote machine that has 64 cores. Lets say I need to run 640 shell scripts in parallel on this machine. How do I do this? I can see splitting the 640 scripts into 64 groups each of ...
14
votes
1answer
2k views

How can I get the index of the xargs “parallel processor”?

Suppose I have two resources, named 0 and 1, that can only be accessed exclusively. Is there any way to recover the "index" of the "parallel processor" that xargs launches in order to use it as a ...
13
votes
9answers
2k views

Spreading stdin to parallel processes

I have a task that processes a list of files on stdin. The start-up time of the program is substantial, and the amount of time each file takes varies widely. I want to spawn a substantial number of ...
10
votes
1answer
39k views

Is it possible to run two commands at the same time in a shell script?

I know that on the command line I can use & to run a command in the background. But I'm wondering if I can do it in a script. I have a script like this: date_stamp=$(date +"%Y-%m-%d" --date='...
10
votes
4answers
3k views

Creating a single output stream out of three other streams produced in parallel

I have three kinds of data that are in different formats; for each data type, there is a Python script that transforms it into a single unified format. This Python script is slow and CPU-bound (to a ...
10
votes
2answers
11k views

How to parallelize dd?

I'm currently having trouble with dd invoked with a sparse file as input (if) and a file as output (of) with conv=sparse. dd seems to be using one core of the CPU (Intel(R) Core(TM) i7-3632QM CPU @ 2....
9
votes
1answer
7k views

Using only one cpu core

I need to run performance tests for my concurrent program and my requirement is that it should be run on only one CPU core. (I don't want to cooperative threads - I want always have a context ...
9
votes
2answers
6k views

Parallelizing a for loop

I want to parallelize the for loops of the following code. How to do this? #!/bin/bash N=$1 n=$2 for (( i=1; i<=$N; i++ )); do min=100000000000000 //set min to some garbage value for (( j=1;...
9
votes
1answer
21k views

Multi-Threading/Forking in a bash script

I have written a bash script which is in following format: #!/bin/bash start=$(date +%s) inFile="input.txt" outFile="output.csv" rm -f $inFile $outFile while read line do -- Block of Commands ...
9
votes
3answers
9k views

correct xargs parallel usage

I am using xargs to call a python script to process about 30 million small files. I hope to use xargs to parallelize the process. The command I am using is: find ./data -name "*.json" -print0 | ...
8
votes
1answer
1k views

Can $! cause race conditions when used in scripts running in parallel?

Say I have multiple bash scripts that run in parallel, with code like the following: #!/bin/bash tail -f /dev/null & echo "pid is "$! Is $! guaranteed to give me the PID of the most recent ...
8
votes
2answers
500 views

Bash: Multiple for loops in Background

Is this the correct way to start multiple sequential processings in the background? for i in {1..10}; do for j in {1..10}; do run_command $i $j; done & done; All j should be ...
8
votes
2answers
9k views

How can I parallelise the upload of a directory by FTP?

I need to upload a directory with a rather complicated tree (lots of subdirectories, etc.) by FTP. I am unable to compress this directory, since I do not have any access to the destination apart from ...
8
votes
4answers
630 views

Virtual write-only file system for storing files in archive

I have an embarrassingly parallel process that creates a huge amount of nearly (but not completely) identical files. Is there a way to archive the files "on the fly", so that the data does not consume ...
8
votes
2answers
5k views

How to make a function available to the command `parallel` (GNU)?

In Bash, let's consider a function that does nothing but echo the argument followed by "is an integer". f () { num="${!1}"; echo $num is an integer; } number=12 f number # 12 is an integer I would ...
8
votes
2answers
1k views

Poor Man's GNU Parallel implemented in ksh?

I'd like to use the feature of GNU parallel where it can execute the command and the list it's fed in parallel and spit it out after it's all done, however, I don't want to install GNU parallel across ...
7
votes
4answers
13k views

Download several files with wget in parallel

I have the following bash script: for i in {0800..9999}; do for j in {001..032}; do wget http://example.com/"$i-$j".jpg done done All photos are exist and in fact each iteration does ...
7
votes
1answer
5k views

Lots of red in htop — does that mean my tasks are tripping over each other?

I've read that the color red indicates "kernel processes." Does that mean little daemons that are regulating which task gets to use the CPU? And by extension, transaction costs in an oversubscribed ...
7
votes
4answers
9k views

Parallel execution of a program on multiple files

I have a small script that loops through all files of a folder and executes a (usually long lasting) command. Basically it's for file in ./folder/*; do ./bin/myProgram $file > ./done/$file ...
7
votes
1answer
2k views

Sort large CSV files (90GB), Disk quota exceeded

Here is what I do right now, sort -T /some_dir/ --parallel=4 -uo file_sort.csv -k 1,3 file_unsort.csv the file is 90GB,I got this error message sort: close failed: /some_dir/sortmdWWn4: Disk quota ...
6
votes
5answers
5k views

run a script in multiple folders in parallel

I have several sub-directories within on high level directory. Each sub-directory has several files and a for loop shell script. The same for loop script is present in each sub-directory. I want to go ...
6
votes
3answers
427 views

Is it safe to pipe the stdout of A to B in parallel

I'm running A file | B --params > file.txt. Since I want to accelerate the processing speed, I used the parallel -j+0< a.txt to run 20 jobs concurrently. a.txt contains all the commands: A ...
6
votes
2answers
3k views

Forcing GNU make to run commands in order

With the following Makefile, GNU make runs the two commands in parallel. Since the first one takes time to finish, rm *.log is run before the log file is created, and fails. dummy.pdf: dummy.tex ...
6
votes
2answers
648 views

How to make this search faster in fgrep/Ag?

I am thinking methods to make the search faster and/or better which principally uses fgrep or ag. Code which searches the word and case-insensitively at $HOME, and redirects a list of matches to vim ...
6
votes
3answers
1k views

Multiple tar processes writing to the same archive file at once

I am running many tasks on a Linux cluster. Each task creates many output files. When all tasks are finished, I run something like tar cf foo.tar output_files/to create a tar archive. This is a very ...
6
votes
3answers
5k views

tarring in parallel

An Oceanographer friend at work needs to back up many months worth of data. She is overwhelmed so I volunteered to do it. There are hundreds of directories to be backed up and we want to tar/bzip ...
6
votes
1answer
7k views

Is there a way to run process parallelly in the loop of a bash script [duplicate]

Possible Duplicate: Parallelizing a for loop The original code might look like this: for i in *; do something.py $i; done I was wondering whether I can run these jobs parallelly in the ...
5
votes
3answers
3k views

How to speed up my build

I am doing a build on a Linux machine with Ubuntu 10.04 on it. How can I really speed up my build? I have 4 CPUs and lots of RAM. I already reniced the process group to -20. Is there something else I ...
5
votes
5answers
378 views

How safe it is to output to <dir> simultaneously with rm <dir>/*

Sometimes I need to remove all the contents of a directory and create new files there. Can I do something like this and expect all new files to remain intact: % rm -rf regression/* & ( sleep 10 ; ...
5
votes
2answers
4k views

How to run x instances of a script parallel?

I have script I'd always like to run 'x' instances in parallel. The code looks a like that: for A in do for B in do (script1.sh $A $B;script2.sh $A $B) & done #B done #A The scripts ...
5
votes
2answers
107 views

Running commands at once

I have text file contain the following commands command1 file1_input; command2 file1_output command1 file2_input; command2 file2_output command1 file3_input; command2 file3_output command1 ...
5
votes
2answers
4k views

split a file, pass each piece as a param to a script, run each script in parallel

I have a words.txt with 10000 words (one to a line). I have 5,000 documents. I want to see which documents contain which of those words (with a regex pattern around the word). I have a script.sh that ...
5
votes
2answers
5k views

How can I run two commands in parallel and terminate them if ONE of them terminates with exit code 0?

I have 2 commands which are to be run simultaneously. And I want the script to terminate if one of them either exits with code 0 or 1. How can I achieve this in Linux(Ubuntu) cmd1 & cmd2 & ...
5
votes
5answers
2k views

How to create a bounded queue for shell tasks?

I have 1000 gzipped files which I want to sort. Doing this sequentially, the procedure looks pretty straightforward: find . -name *.gz -exec zcat {} | sort > {}.txt \; Not sure that the code ...
5
votes
2answers
3k views

How to assign environment variables in parallel in bash

I'm trying to set several environment variables with the results from command substitution. I want to run the commands in parallel with & and wait. What I've got currently looks something like ...

1
2 3 4 5