Questions tagged [parallelism]

Performing tasks in parallel, to make use of multiple processors

Filter by
Sorted by
Tagged with
2
votes
3answers
1k views

Can you make a process pool with shell scripts?

Say I have a great number of jobs (dozens or hundreds) that need doing, but they're CPU intensive and only a few can be run at once. Is there an easy way to run X jobs at once and start a new one ...
2
votes
1answer
600 views

How can I run this bash script in parallel?

I need this to be more efficient Right now it takes up to 20 hrs depending on the line (these are fairly large MCS datasets). Split large data file into its "shots" Creates a list of each shot name ...
2
votes
2answers
156 views

Run 2 commands in parallel but only wait for one command to finish before starting the next

I have set up a number of processes in the background that need to finish processing before the next process can be started. While I've got this set up, there is one step where 2 programs need to be ...
2
votes
2answers
2k views

How to process multiple files with pandoc?

files: $ ls a.md b.md c.md d.md e.md Command: pandoc file.md -f markdown file.pdf How would I parallely process two pandoc instances simulatneously? Possibly with xargs or parallel. It would work ...
2
votes
2answers
105 views

gzip *.txt vs gzip test.txt & gzip test2.txt &

I am looking to speedup gzip process. (server is AIX 7.1) More specificly, the current implementation is with gzip *.txt and it takes up to 1h to complete. (file extractions are quite big and we got ...
2
votes
2answers
130 views

Race Condition not working on Arch Linux

The following C program is supposed to illustrate race condition b/w child and parent processes : #include <stdio.h> #include <unistd.h> #include <sys/types.h> int main() { ...
2
votes
1answer
506 views

Can I expect linear scaling with GNU Parallel

I am using GNU Parallel to perform benchmarking using 20 cores to test speedup in comparison to doing the same job on only 1 core. Should I expect almost complete linear scaling i.e. will my ...
2
votes
1answer
4k views

Can I force Linux to boot its initializations parallel? [closed]

I was wondering if I could manage to initialize drivers, services etc. (all these jobs what Linux does during startup) in parallel instead of sequentially. I want to aggressively lower the boot time. ...
2
votes
1answer
2k views

How to run Abinit via MPI?

I'm trying to run Abinit in parallel, since in sequential mode it takes too much time. I have followed all steps in the abinit tutorials to run it in parallel, and they say to use OpenMPI, but I ...
2
votes
3answers
112 views

How to optimize GNU parallel for this use?

I created this script out of boredom with the sole purpose of using/testing GNU parallel so I know it's not particularly useful or optimized, but I have a script that will calculate all prime numbers ...
2
votes
1answer
153 views

How do I run an on-the-fly ffmpeg (pipe) conversion in parallel?

I want to do the following conversion: for f in *.m4a; do ( ffmpeg -i "$f" -f wav - | opusenc --bitrate 38 - "${f%.m4a}.opus" ) & done I know I could use ffmpeg directly to convert to ...
2
votes
1answer
110 views

parallel doesn't show intermediate output to terminal?

parallel -k python a1.py {} ::: {0..50..2} In the a1.py, I have some print statement. Running the parallel command doesn't show the print statement of my python-script. Is there any way for ...
2
votes
1answer
43 views

Find out where the allocated memory for a process resides

I would like to investigate where the memory for a specific process is allocated. To be more specific: I am running an OpenMP parallel Fortran binary on a ccNUMA machine with two physical CPUs. My ...
2
votes
1answer
3k views

Run GNU Octave script on multiple cores

I am computing Monte-Carlo simulations using GNU Octave 4.0.0 on my 4-core PC. The simulation takes almost 4 hours to compute the script for 50,000 times (specific to my problem), which is a lot of ...
2
votes
1answer
180 views

multicore CLI commands question

I have a collection of 150 CLI commands to execute in a file, on my 12 core machine. Running all 150 tasks in the background bogs the system down and I have found by experience that having 12-18 tasks ...
2
votes
2answers
982 views

Uncompressed .lzo files in parallel and then delete the original .lzo files

So I have .lzo files in /test01/primary folder which I need to uncompress and then delete all the .lzo files. Same thing I need to do in /test02/secondary folder as well. I will have around 150.lzo ...
2
votes
3answers
413 views

How to split the array in set of five files and download them in parallel?

I am trying to copy files from testMachineB and testMachineC into testMachineA as I am running my shell script on testMachineA. If the file is not there in testMachineB, then it should be there in ...
2
votes
2answers
43 views

How do I synchronize appending writes in shell scripts? [duplicate]

I want to write a shell script that accepts a URL and an output markdown file, and adds that URL plus some metadata to the end of that file. It is possible that this script is invoked concurrently, ...
2
votes
3answers
70 views

Shell parallel processing: appending values

I am trying to understand parallel processing in shell scripting and sequentially appending values deterministically (no random order) in the output through a simple example. Below is the code snippet:...
2
votes
2answers
707 views

Run commands in parallel, but kill both if parent is killed?

Is there any way to run two commands in parallel in Bash such that, if the shell is killed forcefully, then both of the commands are guaranteed to be also killed? A simple & won't work because if ...
2
votes
1answer
362 views

Use files from find command in parallel batches

I have this code to search for dirs in current folder and then tar it find . -type d -maxdepth 1 -mindepth 1 -print -exec tar czf {}.tar.gz {} \; I have 100s of folders and currently it does one by ...
2
votes
1answer
4k views

Multi processing / Multi threading in BASH

I have a test file that looks like this 5002 2014-11-24 12:59:37.112 2014-11-24 12:59:37.112 0.000 UDP ...... 23.234.22.106 48104 101 0 0 8.8.8.8 53 68.0 1.0 1 0.0 0 68 0 48 Each line contains a ...
2
votes
1answer
285 views

parallelize shell script

I have a following line: for length in "$(ls $OUT_SAMPLE)" do $CODES/transform_into_line2.rb -c $OUT_SAMPLE -p 0 -f $length & done So,it should parallelize the for loop but somehow it still ...
2
votes
1answer
35 views

Run script in parallel with incremental variable

I would like to run a python script in parallel on several processors on the files in an INPUTDIR. I use the few lines below for that stuff: # define a function pythonprocessing(){ python myscript....
2
votes
2answers
232 views

Running multiple jobs: a combination of parallel and serial

I would like to run tasks a_1, a_2, b_1, b_2, c_1, c_2 in the following fashion: a_i, b_j, c_k (where i, j, k are 0 or 1) can be run in parallel. But a_2 should be run right after a_1 completion (they ...
2
votes
2answers
61 views

Run factor in parallel

how to run Linux's factor in parallel, i.e. utilize all CPU cores? I have tried to run factor <prime number> but unfortunately only one CPU core is being utilized.
2
votes
1answer
88 views

Parallel execution in splitscreen?

I'm very well aware, that I can use xargs to run commands in parallel. However since the consoleout of the commands is important I was wondering if I could split the console window similar to how ...
2
votes
0answers
875 views

How to time the execution of parallel commands

I am trying to write a bash script that launches n commands in the background and then waits for each command to finish. I can easily get the time it took to execute all the commands using time sh ...
2
votes
1answer
660 views

Strategies for building distributed computing system with hybrid Mac and Linux systems

I want to build a distributed computing system to run Matlab, C and other programming languages for scientific computing. Now I've several old Mac machines with Lion Mac OS installed acted as web ...
1
vote
2answers
338 views

Processing command parallel per batch

So, I have 10 CPU core and 20 data to process. I want to process the data in parallel but I am afraid if I just process 20 at once it will make some problem. So, I want to process 10 data 2 times. Is ...
1
vote
2answers
696 views

How can I run multiple pv commands in parallel?

I want to run a sequence of command pipelines with pv on each one. Here's an example: for p in 1 2 3 do cat /dev/zero | pv -N $p | dd of=/dev/null & done The actual commands in the pipe don't ...
1
vote
2answers
727 views

Do different screens on different CPU cores?

I am connecting to a (osX) server with many CPU's. I want to run my program for many different input files. Right now, i run one by one: for input in "${inputs[@]}" do .<my-program.sh> --...
1
vote
2answers
60 views

GNU Parallel linking arguments with alternating arguments

I made a Python simulator that runs on the basis of user-provided arguments. To use the program, I run multiple random simulations (controlled with a seed value). I use GNU parallel to run the ...
1
vote
3answers
59 views

Terminal command follows the lifetime of another terminal command

What I would like to achieve is a bash script or even better a single bash line that can run two terminal based apps on parallel. I am aware of the commands & and ; but in my case they are not ...
1
vote
4answers
106 views

Converting for loops on script that is called by another script into GNU parallel commands

Currently I have the following script for using the HaploTypeCaller program on my Unix system on a repeatable environment I created: #!/bin/bash #parallel call SNPs with chromosomes by GATK for i in ...
1
vote
3answers
100 views

GNU Parallel alternating jobs

I want to run a task where I specify two commands which will be run in an alternating fashion with different parameters. E.g: 1. exec --foo $inputfile1 $inputfile.outfile1 2. exec --bar $inputfile2 $...
1
vote
3answers
326 views

make a loop parallel

How to make this loop parallel : for a in $(seq 1 3) do for b in 0.1 0.2 do echo process with a=$a and b=$b & done done is it in parallel or not ? In fact, I want to run the ...
1
vote
1answer
363 views

Parallelize recursive deletion with find

I want to recursively delete all files that end with .in. This is taking a long time, and I have many cores available, so I would like to parallelize this process. From this thread, it looks like it's ...
1
vote
1answer
801 views

Running a for loop in parallel [closed]

I have a question regarding linux parallel scripts: Context: I need to process many .dat files Code I have so far: #!/bin/bash #PBS -S /bin/bash # set parallel parameters #PBS -r n #PBS -l ...
1
vote
1answer
145 views

option to parallel build an app

These are the debian/rules from an app. called i-nex. It is a CPU-Z alternative for GNU/Linux and has a debian sub-directory having the following files - ┌─[shirish@debian] - [~/games/I-Nex] - [4454] ...
1
vote
2answers
1k views

Using `sem` to make a script run in parallel

I have the following shell script (one liner), which I wanted to use to identify directories that have the exact same contents. I'm using it, to identify and remove duplicate (child) directories. ...
1
vote
1answer
97 views

submitting jobs to get 3 nodes running parallel executions

I have a submit script looks like below, it tries to run a large number of instances of csce.py in backgrounds with 3 nodes.... in a laptop, this usually could successfully automatically distribute ...
1
vote
3answers
874 views

bash multi-threads

I have a list of IPs and I need to check them for opened ports using nmap. So far, my script is like this: #!/bin/bash filename="$1" port="$2" echo "STARTING NMAP" while IFS= read -r line do nmap -...
1
vote
3answers
1k views

New signal arrives while execution is in a signal handler, how to decide which is first?

I am working on a tool handling a massive amount of signals (with different signums) with sigaction(). I need to handle the case, if a new signal is coming, while the previous was in a signal handler....
1
vote
1answer
721 views

In a Makefile, how can I begin processing a file before it has finished building?

I'm using make to automate a project that generates lots of data, then processes it, with recipes roughly like this: processed-data: data data-processing-program $^ > $@ data: output-data ...
1
vote
3answers
1k views

running an executable in parallel by dividing the input directory tree

Commands like find and grep would benefit greatly if they were to run in parallel. Is there a utility or a program that takes a command as an argument and makes it run in parallel by running multiple ...
1
vote
1answer
26 views

Can I write the same tar archive to two tapes at once?

Is it possible to simultaneously write a tar archive to two tape drives using a single command, by directing the same output to both drives? I want to save time by performing the operation in ...
1
vote
1answer
855 views

How does find -execdir <command> + work?

I understand that find -tests -execdir <command> '{}' ';' runs command for every matching file against the test(s) specified. The command, when using -execdir, is executed in the same parent ...
1
vote
1answer
42 views

How to run multiple 'serial' processes under control considering CPUs available? [duplicate]

Suppose you have a binary that has to be run on maaaaaaaaany files (suppose files are numbered from 1 to N). Each file has to be processed by making a call to this binary (say.... something like ...
1
vote
3answers
842 views

Parallel Python scripts on a remote machine

Similar to this thread, I have a remote machine with 8 cores that I want to use for running scripts in parallel (1 script per core at a time). However, I don't have multiple bash scripts but a single ...