Questions tagged [parallelism]

Performing tasks in parallel, to make use of multiple processors

1
vote
0answers
6 views

Warning Message on batch submission of matlab script on Linux-based cluster

I recently began using a super-cluster at my university and have succeeded with serial matlab scripts. Now I am running code using parfor loops, and have been getting this warning in the .out file. I ...
3
votes
1answer
47 views

Parallel running with only limited CPU cores

I have a 12 cores server and I want to run a script which take a file as input argument and does processing on it. I want to use 8 core and no further to do parallel processing on 8 files at a time on ...
2
votes
1answer
29 views

How do I run an on-the-fly ffmpeg (pipe) conversion in parallel?

I want to do the following conversion: for f in *.m4a; do ( ffmpeg -i "$f" -f wav - | opusenc --bitrate 38 - "${f%.m4a}.opus" ) & done I know I could use ffmpeg directly to convert to ...
0
votes
0answers
17 views

How to make p7zip paralell on SLES 12?

If I do a: 7z a -t7z -mx1 testfile.raw.7z testfile.raw I can see 7z isn't using all the available CPUs, only just one ("top" cmd, then "1"). Versions: # rpm -qi p7zip-9.20.1-3.2.x86_64 Name ...
2
votes
3answers
58 views

Shell parallel processing: appending values

I am trying to understand parallel processing in shell scripting and sequentially appending values deterministically (no random order) in the output through a simple example. Below is the code snippet:...
3
votes
1answer
92 views

Make GNU Parallel not delay before executing arguments from STDIN

GNU Parallel, without any command line options, allows you to easily parallelize a command whose last argument is determined by a line of STDIN: $ seq 3 | parallel echo 2 1 3 Note that parallel does ...
1
vote
3answers
36 views

GNU Parallel alternating jobs

I want to run a task where I specify two commands which will be run in an alternating fashion with different parameters. E.g: 1. exec --foo $inputfile1 $inputfile.outfile1 2. exec --bar $inputfile2 $...
0
votes
1answer
23 views

Could the situation: many CPUs simultaneously read the same single file, slow down the speed of reading of every CPU?

I am running cat to separately combine file_X with a numerous number of files e.g. file_1 to file_100000000000. Due to the large number, I distributed the job to a node with 64 CPUs to run parallelly ...
0
votes
1answer
24 views

Live changing bjobs output

When using LSF command bjobs, I would like to get instantly changing output if I submit another job, because I feel stressful to run the same command again and again. I would like something like top ...
2
votes
1answer
36 views

parallel doesn't show intermediate output to terminal?

parallel -k python a1.py {} ::: {0..50..2} In the a1.py, I have some print statement. Running the parallel command doesn't show the print statement of my python-script. Is there any way for ...
2
votes
2answers
245 views

Run commands in parallel, but kill both if parent is killed?

Is there any way to run two commands in parallel in Bash such that, if the shell is killed forcefully, then both of the commands are guaranteed to be also killed? A simple & won't work because if ...
4
votes
7answers
360 views

Alternative to wait -n (because server has old version of bash)

I would like to solve the following issue about submitting a job that has been parallelised to a specific node. Let me start with explaining the structure of my problem I have two very simple Matlab ...
0
votes
2answers
361 views

Parallelise a for loop in a bash file

I would like to parallelise the for loop in the bash script below. I have two Matlab scripts 1) main.m clear rng default P=2; grid=randn(2,3); jobs=1; 2) f.m sgetasknum_grid=grid(jobs*(...
1
vote
2answers
83 views

Processing command parallel per batch

So, I have 10 CPU core and 20 data to process. I want to process the data in parallel but I am afraid if I just process 20 at once it will make some problem. So, I want to process 10 data 2 times. Is ...
2
votes
2answers
374 views

How to process multiple files with pandoc?

files: $ ls a.md b.md c.md d.md e.md Command: pandoc file.md -f markdown file.pdf How would I parallely process two pandoc instances simulatneously? Possibly with xargs or parallel. It would work ...
1
vote
1answer
195 views

How does find -execdir <command> + work?

I understand that find -tests -execdir <command> '{}' ';' runs command for every matching file against the test(s) specified. The command, when using -execdir, is executed in the same parent ...
0
votes
2answers
263 views

How can I run multiple pv commands in parallel?

I want to run a sequence of command pipelines with pv on each one. Here's an example: for p in 1 2 3 do cat /dev/zero | pv -N $p | dd of=/dev/null & done The actual commands in the pipe don't ...
0
votes
0answers
52 views

>100% CPU usage of a process in htop, then lots (>nthreads) of little processes using CPU% amounting to main process

I'm running tensorflow through R Studio. It's multithreaded in some way that I didn't program through one of R's forking packages like foreach, so I don't really understand how it works. The main ...
2
votes
2answers
49 views

Run factor in parallel

how to run Linux's factor in parallel, i.e. utilize all CPU cores? I have tried to run factor <prime number> but unfortunately only one CPU core is being utilized.
0
votes
1answer
397 views

How to optimise Aria2 when downloading thousands of small files?

My starting point is this question. Now aria2 is downloading the files but it is not as fast as I expected it (I am running this on a Macbook Pro with an i7, gigabit connection and AC-wifi. I am most ...
4
votes
3answers
1k views

Running local shell script on multiple remote server in parallel and get the status

I need to run a local script on remote servers. Running script to run in parallel is important. [ec2-user@ip-172-31-43-140 ~]$ cat hosts.txt ec2-user@18.218.191.143 ec2-user@...
3
votes
1answer
42 views

Is an optimal user-time to real-time ratio an indicator for efficient parallelization?

In some unix shells there is the time command which prints how much time a given command takes to be executed. The output looks like real  1m0.000s user 10m0.000s sys   0m0....
4
votes
2answers
742 views

multicore equivalent for '| sort | uniq -c | sort -n' command

I would like to ask if there is a out of the box multicore equivalent for a '| sort | uniq -c | sort -n' command? I know that I can use below procedure split -l5000000 data.tsv '_tmp'; ls -1 _tmp* |...
0
votes
0answers
24 views

Simple multithreaded diffing on macOS

I run diff -rq on terabyte-size directories in macOS. Is there a simple way to parallelize this process?
11
votes
1answer
879 views

How can I get the index of the xargs “parallel processor”?

Suppose I have two resources, named 0 and 1, that can only be accessed exclusively. Is there any way to recover the "index" of the "parallel processor" that xargs launches in order to use it as a ...
0
votes
2answers
344 views

Simultaneously run curl and a process timer

I am trying to simultaneously run a curl command and, essentially, run a timer against how long it takes for it to complete. We have been having some issues with the URLs' response time and I would ...
1
vote
1answer
39 views

Find out where the allocated memory for a process resides

I would like to investigate where the memory for a specific process is allocated. To be more specific: I am running an OpenMP parallel Fortran binary on a ccNUMA machine with two physical CPUs. My ...
0
votes
3answers
105 views

Script Processes in Parallel

I would like to parse apache access log w.r.t IPs. I have used the following code, but it took nearly 90sec. grep "^$CLIENT_IP" /var/log/http/access.log > /tmp/access-$CLIENT_IP.log Then i tried ...
0
votes
1answer
86 views

using top to identify cpu core number with qsub pbspro

I have a shared memory server with PBSpro installed as the job manager. There are 320 cores total, and pbs is configured so there is 1 job queue having 30 of the 32 cpu's so 300 physical cores to be ...
3
votes
2answers
907 views

Combine parallel and sequential commands

1. Summary I don't understand, how I can combine parallel and sequential commands in Linux. 2. Expected behavior Pseudocode: pip install pipenv sequential pipenv install --dev parallel task npm ...
5
votes
1answer
3k views

Lots of red in htop — does that mean my tasks are tripping over each other?

I've read that the color red indicates "kernel processes." Does that mean little daemons that are regulating which task gets to use the CPU? And by extension, transaction costs in an oversubscribed ...
0
votes
1answer
160 views

Running two git commands in parallel

What happens if two git pull command are run simultaneously in the same directory?
3
votes
2answers
1k views

parallelism in bash scripting

Let us consider n statements s1, s2, s3,..., sn. Each statement has infinite steps of execution and I want to run all in parallel. my shell script looks like s1 s2 s3 . . . sn How to change my ...
2
votes
2answers
297 views

Pipeline as parallel command

Normally, pipelines in Unix are used to connect two commands and use the output of the first command as the input of the second command. However, I recently come up with the idea (which may not be new,...
0
votes
0answers
74 views

Is it possible to pin a process to a core which has no other processes pinned to it? [duplicate]

In a parallel programming library, we pin each process to a different core. However, we overlooked the fact that a given core might already be busy and have another CPU-intensive process already ...
1
vote
0answers
52 views

Parallel execution in splitscreen?

I'm very well aware, that I can use xargs to run commands in parallel. However since the consoleout of the commands is important I was wondering if I could split the console window similar to how ...
4
votes
2answers
997 views

How can I run GNU parallel in record per job, with 1 process per core

What I'm really trying to do is run X number of jobs, with X amount in parallel for testing an API race condition. I've come up with this echo {1..10} | xargs -n1 | parallel -m 'echo "{}"'; which ...
4
votes
2answers
3k views

Parallel running of functions

I have 3 functions, like function WatchDog { sleep 1 #something } function TempControl { sleep 480 #somthing } function GPUcontrol { sleep 480 #somethimg } And i am runing it like WatchDog | ...
0
votes
1answer
448 views

slurm: srun & sbatch different performance with the same settings

In a slurm system, when I use srun command to run the program. It runs very slow and seems like only one processor works. srun --pty -A free -J test -N 1 -n 1 -c 1 mpirun -np 16 $FEAPHOME8_3/...
0
votes
1answer
145 views

Start multiple vagrant boxes in parallel via Makefile

I have three vagrant boxes I usually need simultaneously so I created a Makefile to start them all at once. Well, one after the other actually. The Makefile looks like this basically: up: cd ...
0
votes
1answer
124 views

ssh multiple server for tar operation [duplicate]

I have a file containing IP address of 10 servers. e.g host.txt I want to tar a directory and on 10 differents server from 1 server. How can I run ssh parallel to those server which can reduce my ...
0
votes
2answers
354 views

Do different screens on different CPU cores?

I am connecting to a (osX) server with many CPU's. I want to run my program for many different input files. Right now, i run one by one: for input in "${inputs[@]}" do .<my-program.sh> --...
4
votes
2answers
2k views

How to run groups of commands in parallel?

I am trying to run groups of commands in parallel. There are 10 groups and each group has 3 commands. I want sequential execution within each group and parallel execution of the 10 groups. I am trying ...
1
vote
1answer
84 views

bash parallel encoding and uploading

I am working on a bash script for encoding videos with ffmpeg in a special format. To save the resulting videos I want to upload the video as soon as the encoding is finished. Therefore I want to use ...
1
vote
1answer
40 views

How to run multiple 'serial' processes under control considering CPUs available? [duplicate]

Suppose you have a binary that has to be run on maaaaaaaaany files (suppose files are numbered from 1 to N). Each file has to be processed by making a call to this binary (say.... something like ...
6
votes
4answers
9k views

Download several files with wget in parallel

I have the following bash script: for i in {0800..9999}; do for j in {001..032}; do wget http://example.com/"$i-$j".jpg done done All photos are exist and in fact each iteration does ...
0
votes
1answer
181 views

run the same script for multiple files, scripts requires 2 input files

I have a folder containing 500 genotype files, like this: xaa.txt xab.txt xac.txt xad.txt I want to run a script for each genotype file separately at the same file; I know this question has been ...
2
votes
2answers
86 views

gzip *.txt vs gzip test.txt & gzip test2.txt &

I am looking to speedup gzip process. (server is AIX 7.1) More specificly, the current implementation is with gzip *.txt and it takes up to 1h to complete. (file extractions are quite big and we got ...
1
vote
0answers
52 views

Optimal value for make -j <value> [duplicate]

What is the optimal value of the -j flag for make ? I know that all my source is free of errors. Is it the number of cores? Or the number of logical processors?
1
vote
0answers
94 views

Monitor MPSSH Progress, maybe PV?

Does anyone know of a good way to monitor the progress of mpssh? While making a large number of parallel connections it can take some time to complete. I'm trying to figure out a good way to monitor ...