Questions tagged [parallelism]

Performing tasks in parallel, to make use of multiple processors

Filter by
Sorted by
Tagged with
0 votes
0 answers
11 views

Any good tool/library for queue processing with good monitoring/management interface?

I'm trying to run an experiment that involves transforming a lot of files through a pipeline like this. A and B take a file as input and produces a file, or some text through stdout. C takes those ...
user avatar
  • 101
1 vote
0 answers
35 views

Task takes 15 minutes with one processor and 20 minutes with two processors

This is the command, run at the root of an NTFS partition from a Debian 12 VM (VirtualBox) $ (find . -type f -exec cat {} \;) | pv | wc -c I was trying to check that all files are readable. It's a ...
user avatar
  • 365
-1 votes
2 answers
62 views

Is there a simple way to run multiple instances of the same C++ program on Linux?

I have a C++ program I'd like to run in parallel on a Linux machine, with each instance being a completely independent process that doesn't communicate with the others. Is there a straightforward way ...
user avatar
  • 1
0 votes
2 answers
40 views

How might I execute this nested for loop in parallel?

#!/usr/bin/bash TARGETS=( "81.176.235.2" "81.176.70.2" "78.41.109.7" ) myIPs=( "185.164.100.1" "185.164.100.2" "185.164.100.3" "185....
user avatar
3 votes
2 answers
183 views

How to merge data from multiple background jobs back to a single data stream in bash

I've run into a couple of similar situations where I can break a single-core bound task up into multiple parts and run each part as separate job in bash to parallelize it, but I struggle with ...
user avatar
  • 67
3 votes
1 answer
30 views

Safely redirect output to a single file with xargs parallel mode

I have a bunch of bzipped JSON file that I read with xargs in parallel, do some light processing with jq and redirect the output to a file as follows: # Number of workers is one less than the number ...
user avatar
1 vote
1 answer
29 views

Serialize all rules in GNU make: best practise?

I have a makefile and want to make sure that all the rules are executed sequentially, that is, that no parallel execution is performed. I believe I have three ways of achieving this: With ....
user avatar
  • 366
1 vote
1 answer
52 views

How to kill all background and spawned processes of a bash script in its pre-exit handler?

I'm using the wait -n technique to perform max_jobs parallel tasks: #!/usr/bin/env bash cleanup() { echo "cleaning up..." } trap "cleanup" EXIT do_task() { echo "...
user avatar
0 votes
0 answers
66 views

Making multiple PAM modules work together with success as soon as any of them return successfully

I basically want to unlock any sudo authentication with any of these criteria met (whatever completes first successfully): a usb as a security key matched (custom script). fingerprint matched. ...
user avatar
0 votes
1 answer
61 views

Is 'cmd1 [newline] cmd2' as concurrent as 'cmd1 & cmd2' in a Bash script?

Does the following script #!/usr/bin/bash while true do ab -c 700 -n 100000000000 -r http://crimea-post.ru/ >>crimea-post & ab -c 700 -n 100000000000 -r http://nalog.gov.ru/ >>nalog ...
user avatar
0 votes
0 answers
44 views

How to keep track of PID of multiple active scripts that spawn multiple bacground processes

I have some programs that that need to run in order with a common task between each program that modifies the data and sends it to another location for processing. each modify_data script spawns ...
user avatar
  • 67
2 votes
1 answer
45 views

Linux server: How does memory get allocated to parallel processes?

I'm a complete newbie, so please excuse my ignorance and/or potentially wrong terminology. I'm using an Ubuntu server through ssh for brain image processing (one command with several programs, ...
user avatar
  • 29
1 vote
1 answer
239 views

How to run one command on several cores

I have a command which I want to run on all free cores to speed up the execution time. Specifically I am running the Pitman-Yor Adaptor-Grammar Sampler software I downloaded from here ./py-cfg/py-cfg-...
user avatar
  • 171
0 votes
1 answer
142 views

Running script in parallel using xargs and cat

I'm inserting a lot of CSV files into database. I want to do it in parallel, for example run 4 processes. Right now I do it with the script like this: find . -name "*.csv" | xargs -n 1 -P 4 ....
user avatar
  • 115
-1 votes
1 answer
368 views

Curl Parallel requests using links source file

I have this script to go through a list of URLs and the check return codes using Curl. Links file goes like this: https://link1/... https://link2/... https://link200/... (...) The script: INDEX=0 DIR=...
user avatar
5 votes
1 answer
170 views

Bash background execution not returning

I am trying to write a script that has the purpose to parallelize an execution (a program that creates some files) running the processes in background and, when all commands in the for loop are done, ...
user avatar
  • 247
1 vote
2 answers
142 views

How to run PDF to TIFF conversion in parallel?

I have this command here for batch converting PDF documents (first 2 pages) to TIFF files using pdftoppm. The goal is to put the TIFF images into its own folder with folder name matching the original ...
user avatar
0 votes
2 answers
110 views

Do multiple shell scripts sourcing a common other script interfere with each other when they modify a "sourced" variable?

If I source the same shell script from multiple other scripts, which I run in parallel, and modify a shell variable defined in the sourced script, will the sourcing scripts mess up each other's value ...
user avatar
  • 1
0 votes
1 answer
118 views

1TB drive compressed shows only 3.8GB, what did I do wrong?

On Linux Mint 20.2 Cinnamon I would like to create a disk image of my secondary disk drive (SATA) containing Windows 10, not that it matters now, directly gzip'ed using the Parallel gzip = pigz onto ...
user avatar
0 votes
0 answers
185 views

Parallel job submission of for loop

I have written for loop and parallelized it with & and limited the threshold to running 3 jobs at one time. Below is my script. I am reserving 32 cores and 256 GB memory through BSUB command. The ...
user avatar
1 vote
1 answer
89 views

How to parallelize a recursive bash function?

I have a bash function that mainly curls an endpoint for a set of links, and again curls each of those links (for another set of links) recursively. task() { link="$1" response=$(curl &...
user avatar
0 votes
2 answers
251 views

One-liner to run commands from a file in parallel using xargs

I have a script like this: #!/bin/csh command 1 \ -f \"input1\" \ -l input2 -other_swithes1 command 2 \ -f \"input1\" \ -m input2 \ -l input3 -other_swithes1 ...
user avatar
  • 9
1 vote
1 answer
40 views

Tools for batching N commands over L scripts (for N≫L)?

Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to M jobs either running or pending at any one time, of ...
user avatar
  • 13k
1 vote
1 answer
86 views

Is it with GNU parallel possible to split on character count, but provide full lines only?

I would like to split an input file on character count (ASCII is fine), combined with new lines as well. That is, every group of 10000 character should be seen as one record to be piped into the child ...
user avatar
1 vote
1 answer
132 views

How can I use the parallel command while getting parameters dynamically from a file?

Let's say I have a file listing the path of multiple files like the following: /home/user/file1.txt /home/user/file2.txt /home/user/file3.txt /home/user/file4.txt /home/user/file5.txt /home/user/file6....
user avatar
  • 319
2 votes
2 answers
1k views

How to combine multiple lines with xargs

I have a large input file which contains 30M lines, new lines in \r\n. I want to process this file in parallel by sending chunks of 1000 lines (or less, for the remainder of the file) to a REST API ...
user avatar
0 votes
1 answer
52 views

Problem running several programs or commands at the same time

I had a problem because I have to execute near 1700 times the same program with different data. That program do several calculations using iterations, and the answer is obtained 5 hours later ...
user avatar
2 votes
1 answer
179 views

How to make multiple parallel calls to function which makes post request in unix?

I am having a function A, which takes an argument fileName and then makes a curl post call with that file to employment Server.The function pseudocode will look as : function A(filename) { // read ...
user avatar
0 votes
2 answers
129 views

Bash script for reading multiple files

I have multiple R scripts to read (up to 3 i.e. tr1.R, tr2.R, tr3.R). The bash script for reading a single script is given below #!/bin/bash #PBS -l nodes=1:ppn=10,walltime=00:05:00 #PBS -M #PBS -m e ...
user avatar
3 votes
3 answers
424 views

How can I parallelize command sha256sum or other hashing commands?

I want to parallelize hash calculation process because I have a very large amount of file counts and sizes. When I see CPU usage of these commands, I get upset because they are only using one thread; ...
user avatar
1 vote
1 answer
73 views

Pausing and then resuming a piped command

I'm downloading a lot of data for my research. The data is being downloaded on one of my campus's supercomputers, but data downloads are interrupted every hour. When the OS pauses the pipeline, I have ...
user avatar
  • 111
1 vote
1 answer
90 views

Use GNU Parallel when file has a single (long) line

I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv. This file has 544609657 characters, is about 545 megabytes, all in a single line. ...
user avatar
0 votes
2 answers
147 views

How can I run many emerge jobs in paralell?

When I install many packages at once, I use emerge --jobs=8 firefox vim. This will install the packages, but it won't run everything in paralel. For example, it will wait compile rust before compiling ...
user avatar
-1 votes
1 answer
339 views

Using shell arrays to read every N lines in parallel of a file

So, i already know how to successfully read every N lines in parallel, and run a command on each of those lines: while read -r i && read -r a && read -r b && read -r c &&...
user avatar
0 votes
2 answers
200 views

create parallel gpg

I am trying to tar, compress and encrypt a large backup folder but the process is slow as I believe gpg is single threaded. I have been tryping to use split to divide the root folder so instead of ...
user avatar
  • 1
0 votes
0 answers
163 views

Increase Chrome/Firefox's max parallel worker limit?

Our company have a webpage dedicated to processing costs information for our customers locally on their machines via Javascript code. In the past, this processing was sequential, which could easily ...
user avatar
0 votes
1 answer
21 views

Is it possible to continuously feed parallel with more input?

What I want to do is something like this: inotifywait -m ./input -e close_write -e moved_to | parallel --max-args 1 "someSlowSingleThreadedTaskThatActsOnThisFile" I can't imagine that ...
user avatar
  • 142
1 vote
1 answer
36 views

parallel compare two arguments and send to command

I am trying to run recon-all command with GNU parallel. It should take one argument at {1} like this: parallel --progress subj_id='echo {1} | cut -d / -f 3' && recon-all -s $subj_id -i {1} -sd ...
user avatar
0 votes
1 answer
128 views

How to tune the linux Scheduler for parallel computation?

I have a linux machine dedicated to running some parallel computation, and I'm trying to understand how to choose / tune the scheduler, and perhaps other parameters, to extract the most performance (...
user avatar
  • 101
0 votes
2 answers
42 views

Will "$!" reliably return the correct ID with "&"?

In my tests, I always get the correct result so far with this: [fabian@manjaro ~]$ sleep 10 & echo $! [1] 302657 302657 But sleep and echo are getting executed simultaneously here, so I would ...
user avatar
1 vote
1 answer
460 views

What is the detriment to downloading too many files in parallel through lftp?

I'm downloading files in parallel through lftp onto a remote server, and I use the commands: lftp ftp:*url* cd into directory mirror --parallel=10 I'm downloading a total of 365 files separated ...
user avatar
1 vote
2 answers
508 views

FIFO-based semaphore explanation

I am trying to work on some parallelization of many processes (task send/to be executed on many (let's say hundreds) nodes). I came across this solution: https://unix.stackexchange.com/a/216475 # ...
user avatar
1 vote
3 answers
2k views

Concurrency and Parallelism on Bash

I'm a bit confused about concurrency and parallelism in the bash shell. As I understand it, when we run commands in more than one subshells at the same time, these commands run in parallel on ...
user avatar
  • 1,102
1 vote
0 answers
20 views

How to load test (emulate multiple users) a storage device with bonnie++?

I have just started using bonnie++ for testing the performance of storage devices. So far I have done that emulating a single user but it would be more realistic, or at least an interesting variation, ...
user avatar
  • 2,032
2 votes
2 answers
157 views

Process multiple inputs with multiple equivalent commands (multiple thread pools) in GNU parallel

I have a large dataset (>200k files) that I would like to process (convert files into another format). The algorithm is mostly single-threaded, so it would be natural to use parallel processing. ...
user avatar
  • 4,737
1 vote
1 answer
67 views

Bash parallel command is executing commands sequentially

I am trying to download multiple files parallelly in bash and I came across GNU parallel. It looks very simple and straight forward. But I am having a hard time getting GNU parallel working. What am I ...
user avatar
1 vote
2 answers
79 views

return code of individual script ,though they are running in parallel

I have jar file which is running in parallel with multiple table names as arguments using nohup command in unix nohup java -jar batchorch.jar $tablename & I have used below lines for getting ...
user avatar
0 votes
1 answer
99 views

SLURM: How do I submit multilple OpenMP parallel codes in a job script

I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.) #!/bin/bash #SBATCH -N 1 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=2 export OMP_NUM_THREADS=...
user avatar
1 vote
2 answers
204 views

How to get nth result from find?

I have access to a distributed computing/server farm with a job scheduler (Slurm) that gives each parallel job an integer ID from 1 to n (I know the value of n, in the example below, n = 10). I am ...
user avatar
1 vote
1 answer
73 views

appending values in a dynamic array

#!/bin/bash range=$(seq -f "cen%04g" 1052 1099) range1=$(seq -f "rh%04g" 1052 1099) check () { for node1 in ${range};do ping -q -c 1 -w 3 -s 10 $node1 >/dev/null if [ ...
user avatar
  • 37

1
2 3 4 5 6