Questions tagged [parallelism]

Performing tasks in parallel, to make use of multiple processors

Filter by
Sorted by
Tagged with
1
vote
1answer
25 views

Can I write the same tar archive to two tapes at once?

Is it possible to simultaneously write a tar archive to two tape drives using a single command, by directing the same output to both drives? I want to save time by performing the operation in ...
2
votes
2answers
43 views

How do I synchronize appending writes in shell scripts? [duplicate]

I want to write a shell script that accepts a URL and an output markdown file, and adds that URL plus some metadata to the end of that file. It is possible that this script is invoked concurrently, ...
1
vote
1answer
21 views

How can I use fork to calculate partial results of a calculation?

The task that has been given to me is that I have to find all numbers that divide a number (let's call it x) and find out how many times does it divide it. I have to use fork to do the task. What I ...
0
votes
0answers
13 views

How does KCOV handle multi-processor systems?

The documentation of KCOV states that tracing is enabled only for one task (the one associated with the userland code that asked to enable tracing). However, I don't understand how it works. From ...
-1
votes
1answer
50 views

Wildcard variables in xargs for parallel processing

Issue I have an issue using an input variable VAR in a function (i.e. ls) with the wilcard *, when this funciton is within an xargs bash call. This xargs function repeats over each member of a list ...
1
vote
2answers
54 views

GNU Parallel linking arguments with alternating arguments

I made a Python simulator that runs on the basis of user-provided arguments. To use the program, I run multiple random simulations (controlled with a seed value). I use GNU parallel to run the ...
1
vote
1answer
25 views

Pairwise merging of files in parallel until only 1 file left over

I'd like to combine a bunch of language model (LM) count files using SRILM's ngram-merge program. With that program, I could combine a whole directory of count files, e.g. ngram-merge -write ...
1
vote
3answers
56 views

Terminal command follows the lifetime of another terminal command

What I would like to achieve is a bash script or even better a single bash line that can run two terminal based apps on parallel. I am aware of the commands & and ; but in my case they are not ...
3
votes
3answers
119 views

Bash consecutive and parallel loops/commands

I want to run some simulations using a Python tool that I had made. The catch is that I would have to call it multiple times with different parameters/arguments and everything. For now, I am using ...
0
votes
1answer
25 views

Thread safe next number (or round-robin) service in linux

I have an Ubuntu machine and a scenario in which I need to generate the next number (starting from X) between 0 and 20 in a round-robin way. I need a way to get the next number which will be thread ...
2
votes
1answer
35 views

Run script in parallel with incremental variable

I would like to run a python script in parallel on several processors on the files in an INPUTDIR. I use the few lines below for that stuff: # define a function pythonprocessing(){ python myscript....
1
vote
0answers
44 views

Assign result of a command to variables, but don't wait for the command to finish [duplicate]

I am trying to parallelize a bash script that looks like this: var01=$(ssh $USER@host01 'bash -s' < ./node.sh) & var02=$(ssh $USER@host02 'bash -s' < ./node.sh) & var03=$(ssh $USER@...
5
votes
2answers
150 views

How to extract a tar.gz file to two destinations concurrently?

I have some tar.gz files. I need to extract these tar.gz files to two USB harddisks. With the command tar -xvzf I can only extract the tar.gz to one harddisk at a time. Grateful if I can be advised ...
2
votes
3answers
109 views

How to optimize GNU parallel for this use?

I created this script out of boredom with the sole purpose of using/testing GNU parallel so I know it's not particularly useful or optimized, but I have a script that will calculate all prime numbers ...
1
vote
1answer
78 views

GNU Parallel Python Semaphore

I have a Python script parallelized through GNU parallel which finds a certain result which I would like to output to a file, which I currently do through standard Python file IO. The issue is that I ...
0
votes
0answers
44 views

Allow bash script to run commands on multiple processors

I have a script compute_all_matrices.sh which launches deepTools' computeMatrix for a whole bunch of different files with various option. The user manual for computeMatrix states: --...
3
votes
2answers
76 views

Adding a timeout to a parallelized call in Bash

I have a simple Bash script I am running to parallelize and automate the execution of a program written in Sage MATH: #!/bin/bash for i in {1..500}; do echo Spinning up threads... echo Round $i ...
0
votes
1answer
106 views

Is it possible to make -jN dynamic based on CPU load?

For example, would it not be beneficial to have -jN > nrcpus during the configure phase, where CPU load is low? Then have -jN = nrcpus during compilation, where load should be evenly distributed to ...
0
votes
1answer
90 views

Are subshells run in parallel by default?

This question follows from this one and one of its answers' recommendation to read Linuxtopia - Chapter 20. Subshells. I'm a bit confused by this statement at the Linuxtopia site: subshells let ...
8
votes
2answers
429 views

Bash: Multiple for loops in Background

Is this the correct way to start multiple sequential processings in the background? for i in {1..10}; do for j in {1..10}; do run_command $i $j; done & done; All j should be ...
8
votes
1answer
1k views

Can $! cause race conditions when used in scripts running in parallel?

Say I have multiple bash scripts that run in parallel, with code like the following: #!/bin/bash tail -f /dev/null & echo "pid is "$! Is $! guaranteed to give me the PID of the most recent ...
2
votes
2answers
212 views

Running multiple jobs: a combination of parallel and serial

I would like to run tasks a_1, a_2, b_1, b_2, c_1, c_2 in the following fashion: a_i, b_j, c_k (where i, j, k are 0 or 1) can be run in parallel. But a_2 should be run right after a_1 completion (they ...
1
vote
1answer
36 views

Take similarly named fastq files on a folder and amke them input for an script(on pairs)

I have a set of files named like this inside a folder: AM11_BW415_R1.fastq.gz NAM13_BW968_R2.fastq.gz NAM17_AC_Barrie_R1.fastq.gz NAM3_PI648600_R2.fastq.gz NAM7_Glenlea_R1.fastq.gz ...
1
vote
4answers
97 views

Converting for loops on script that is called by another script into GNU parallel commands

Currently I have the following script for using the HaploTypeCaller program on my Unix system on a repeatable environment I created: #!/bin/bash #parallel call SNPs with chromosomes by GATK for i in ...
3
votes
2answers
156 views

Parallel Recursive Grep While Omitting String

Problem I cannot perform a recursive parallel zgrep while also omitting a string from the results. I am zgrepping through about 640GB of compressed firewall logs and almost %30 of the lines have the ...
0
votes
0answers
36 views

parallelizing a for loop containing a linux command which has more than one command line arguments [duplicate]

Hi all I have the following bash script that calls hmmscan from hmmer3 software. hmmscan requires to specify 6 command line arguments in this case the code that I have written is as follows: while ...
1
vote
0answers
37 views

Old version of of R Cannot Find Packages on RHEL 7

I am working on a Redhat 7 linux server and utilizing an extremely old version of R (3.0.3 to be exact). I had to install this version of R from a source file. I properly configured the Renviron file ...
0
votes
0answers
21 views

How RAM is distributed among processes in MPI?

For example for a 4GB RAM and 2 processes, each process will take 2 GB of memory?
1
vote
0answers
20 views

Why do we use dynamic memory in ccNUMA systems when we talk about data distribution into locality domains by first touch policy?

In many books, when they talk about first touch policy in ccNUMA systems they are using dynamic memory allocation when they distribute data across locality domains. What if, for example, we have an ...
1
vote
0answers
17 views

Warning Message on batch submission of matlab script on Linux-based cluster

I recently began using a super-cluster at my university and have succeeded with serial matlab scripts. Now I am running code using parfor loops, and have been getting this warning in the .out file. I ...
3
votes
1answer
528 views

Parallel running with only limited CPU cores

I have a 12 cores server and I want to run a script which take a file as input argument and does processing on it. I want to use 8 core and no further to do parallel processing on 8 files at a time on ...
2
votes
1answer
141 views

How do I run an on-the-fly ffmpeg (pipe) conversion in parallel?

I want to do the following conversion: for f in *.m4a; do ( ffmpeg -i "$f" -f wav - | opusenc --bitrate 38 - "${f%.m4a}.opus" ) & done I know I could use ffmpeg directly to convert to ...
2
votes
3answers
68 views

Shell parallel processing: appending values

I am trying to understand parallel processing in shell scripting and sequentially appending values deterministically (no random order) in the output through a simple example. Below is the code snippet:...
4
votes
1answer
342 views

Make GNU Parallel not delay before executing arguments from STDIN

GNU Parallel, without any command line options, allows you to easily parallelize a command whose last argument is determined by a line of STDIN: $ seq 3 | parallel echo 2 1 3 Note that parallel does ...
1
vote
3answers
97 views

GNU Parallel alternating jobs

I want to run a task where I specify two commands which will be run in an alternating fashion with different parameters. E.g: 1. exec --foo $inputfile1 $inputfile.outfile1 2. exec --bar $inputfile2 $...
0
votes
1answer
63 views

Could the situation: many CPUs simultaneously read the same single file, slow down the speed of reading of every CPU?

I am running cat to separately combine file_X with a numerous number of files e.g. file_1 to file_100000000000. Due to the large number, I distributed the job to a node with 64 CPUs to run parallelly ...
0
votes
1answer
107 views

Live changing bjobs output

When using LSF command bjobs, I would like to get instantly changing output if I submit another job, because I feel stressful to run the same command again and again. I would like something like top ...
2
votes
1answer
103 views

parallel doesn't show intermediate output to terminal?

parallel -k python a1.py {} ::: {0..50..2} In the a1.py, I have some print statement. Running the parallel command doesn't show the print statement of my python-script. Is there any way for ...
2
votes
2answers
681 views

Run commands in parallel, but kill both if parent is killed?

Is there any way to run two commands in parallel in Bash such that, if the shell is killed forcefully, then both of the commands are guaranteed to be also killed? A simple & won't work because if ...
4
votes
7answers
760 views

Alternative to wait -n (because server has old version of bash)

I would like to solve the following issue about submitting a job that has been parallelised to a specific node. Let me start with explaining the structure of my problem I have two very simple Matlab ...
0
votes
2answers
1k views

Parallelise a for loop in a bash file

I would like to parallelise the for loop in the bash script below. I have two Matlab scripts 1) main.m clear rng default P=2; grid=randn(2,3); jobs=1; 2) f.m sgetasknum_grid=grid(jobs*(...
1
vote
2answers
305 views

Processing command parallel per batch

So, I have 10 CPU core and 20 data to process. I want to process the data in parallel but I am afraid if I just process 20 at once it will make some problem. So, I want to process 10 data 2 times. Is ...
2
votes
2answers
1k views

How to process multiple files with pandoc?

files: $ ls a.md b.md c.md d.md e.md Command: pandoc file.md -f markdown file.pdf How would I parallely process two pandoc instances simulatneously? Possibly with xargs or parallel. It would work ...
1
vote
1answer
783 views

How does find -execdir <command> + work?

I understand that find -tests -execdir <command> '{}' ';' runs command for every matching file against the test(s) specified. The command, when using -execdir, is executed in the same parent ...
1
vote
2answers
662 views

How can I run multiple pv commands in parallel?

I want to run a sequence of command pipelines with pv on each one. Here's an example: for p in 1 2 3 do cat /dev/zero | pv -N $p | dd of=/dev/null & done The actual commands in the pipe don't ...
2
votes
2answers
59 views

Run factor in parallel

how to run Linux's factor in parallel, i.e. utilize all CPU cores? I have tried to run factor <prime number> but unfortunately only one CPU core is being utilized.
1
vote
1answer
640 views

How to optimise Aria2 when downloading thousands of small files?

My starting point is this question. Now aria2 is downloading the files but it is not as fast as I expected it (I am running this on a Macbook Pro with an i7, gigabit connection and AC-wifi. I am most ...
4
votes
3answers
2k views

Running local shell script on multiple remote server in parallel and get the status

I need to run a local script on remote servers. Running script to run in parallel is important. [ec2-user@ip-172-31-43-140 ~]$ cat hosts.txt ec2-user@18.218.191.143 ec2-user@...
3
votes
1answer
50 views

Is an optimal user-time to real-time ratio an indicator for efficient parallelization?

In some unix shells there is the time command which prints how much time a given command takes to be executed. The output looks like real  1m0.000s user 10m0.000s sys   0m0....
4
votes
2answers
1k views

multicore equivalent for '| sort | uniq -c | sort -n' command

I would like to ask if there is a out of the box multicore equivalent for a '| sort | uniq -c | sort -n' command? I know that I can use below procedure split -l5000000 data.tsv '_tmp'; ls -1 _tmp* |...

1
2 3 4 5