Questions tagged [parallelism]
Performing tasks in parallel, to make use of multiple processors
273
questions
1
vote
2
answers
130
views
How can I run `$(nproc)`× parallel instances of `openssl dhparam` & when first instance exits with `0`, kill the other instances?
I'm running openssl dhparam -out dhparam4096.pem 4096 and it pegs a single core at 100% for the duration of the task (which can be considerable on some processors). I have 1 or more additional cores ...
- 131
3
votes
2
answers
731
views
Race condition not seen while two scripts write to a same file
I have two scripts running parallelly and they are echoing to the same file. One script is echoing +++++++++++++++ to the file while the other script is echoing =========== to the file.
Below is the ...
- 101
1
vote
1
answer
78
views
optipng in parallel using shell
I would like to run optipng in parallel with my 8-thread CPU using the shell.
I know the program itself is not multicore optimized, the only way will be to run 8 files with it.
I have 500+ PNG images ...
- 28.3k
2
votes
0
answers
260
views
How do you fetch a large file over http in parallel?
Question:
Since HTTP supports resuming at an offset, are there any tools (or existing options for commands like wget or curl) that will launch multiple threads to fetch the file in parallel with ...
- 303
0
votes
0
answers
331
views
Terminal closes itself when running many python processes
I was trying to run multiple machine learning experiments simultaneously to save time. My OS is Fedora 36, and I use emacs. To automate the experiment I used runs.py
import subprocess
subprocess.run(&...
- 103
4
votes
4
answers
360
views
Parallelize filling of a variable in Bash
I am trying to speed up a find command which accesses files on multiple harddrives by utilizing parallelization. Unfortunately, either the parallelization is ignored or the variable is not filled.
...
- 260
1
vote
1
answer
119
views
Does parallel limit cause some items to not be checked?
I have near 400 git repos on my machine.
And this is a script that I use to find their status collectively:
function Check()
{
gitFolder="$1"
parent=$(dirname $gitFolder);
...
- 629
1
vote
1
answer
158
views
How to parallelize a bash for loop and use the iteration variable for file IO
I have a bash script that more or less looks like this:
N=32
for i in $(seq -f "%06g" 0 ${LAST_NUM}) # LAST_NUM is an env variable
do
((j=j%N)); ((j++==0)) && wait # Wait for all ...
- 11
2
votes
1
answer
652
views
GNU Parallel with -j -N still uses one CPU
How can I get reasonable parallelisation on multi-core nodes without saturating resources? As in many other similar questions, the question is really how to learn to tweak GNU Parallel to get ...
- 1,460
1
vote
1
answer
86
views
Bash: how to check if any subshell failed?
Consider this code:
job()
{
local id=$1
sleep $id
}
do_job_in_parallel()
{
local pids=()
# run subshells
for id in $(seq 4)
do
job $id &
pids=("${pids[@]}" $!)
...
- 517
0
votes
0
answers
310
views
Increasing max_parallel_downloads is not effective in yum commands
To install all the updates in a Red Hat or CentOS machine, we use the following command
yum -y update --skip-broken --nobest
We are looking for ways to improve total running time of installing the ...
0
votes
1
answer
42
views
Kythe code indexing processes being throttled despite normal CPU temperature
I'm running into an issue with CPU throttling that only seems to trigger under a specific workload of running the Kythe indexer. Detailed repro steps at the end of the question. I'm going to give a ...
- 311
3
votes
1
answer
69
views
Is it possible to bundle subshell output?
I have this code to check the status of all of my git folders.
find / -maxdepth 3 -not -path / -path '/[[:upper:]]*' -type d -name .git -not -path "*/Trash/*" -not -path "*/Temp/*" ...
- 629
7
votes
6
answers
4k
views
Start 100 process at a time in bash script
In bash script, I have a program like this
for i in {1..1000}
do
foo i
done
Where I call the function foo 1000 times with parameter i
If I want to make it run in multi-process, but not all at once,...
- 173
0
votes
1
answer
251
views
How to avoid duplicate random number generation?
TL;DR:
A program draws the same random seed when started twice simultaneously. How did it happen?
Details
I'm running an MCMC statistical analysis, so I execute the program (phylobayes) twice to get 2 ...
- 619
0
votes
0
answers
54
views
Any good tool/library for queue processing with good monitoring/management interface?
I'm trying to run an experiment that involves transforming a lot of files through a pipeline like this. A and B take a file as input and produces a file, or some text through stdout. C takes those ...
- 101
1
vote
0
answers
52
views
Task takes 15 minutes with one processor and 20 minutes with two processors
This is the command, run at the root of an NTFS partition from a Debian 12 VM (VirtualBox)
$ (find . -type f -exec cat {} \;) | pv | wc -c
I was trying to check that all files are readable. It's a ...
- 407
-1
votes
2
answers
1k
views
Is there a simple way to run multiple instances of the same C++ program on Linux?
I have a C++ program I'd like to run in parallel on a Linux machine, with each instance being a completely independent process that doesn't communicate with the others. Is there a straightforward way ...
- 1
1
vote
2
answers
901
views
How might I execute this nested for loop in parallel?
#!/usr/bin/bash
TARGETS=(
"81.176.235.2"
"81.176.70.2"
"78.41.109.7"
)
myIPs=(
"185.164.100.1"
"185.164.100.2"
"185.164.100.3"
"185....
- 621
3
votes
2
answers
267
views
How to merge data from multiple background jobs back to a single data stream in bash
I've run into a couple of similar situations where I can break a single-core bound task up into multiple parts and run each part as separate job in bash to parallelize it, but I struggle with ...
- 77
3
votes
1
answer
752
views
Safely redirect output to a single file with xargs parallel mode
I have a bunch of bzipped JSON file that I read with xargs in parallel, do some light processing with jq and redirect the output to a file as follows:
# Number of workers is one less than the number ...
- 129
1
vote
1
answer
736
views
Serialize all rules in GNU make: best practise?
I have a makefile and want to make sure that all the rules are executed sequentially, that is, that no parallel execution is performed.
I believe I have three ways of achieving this:
With ....
- 328
1
vote
1
answer
225
views
How to kill all background and spawned processes of a bash script in its pre-exit handler?
I'm using the wait -n technique to perform max_jobs parallel tasks:
#!/usr/bin/env bash
cleanup() {
echo "cleaning up..."
}
trap "cleanup" EXIT
do_task() {
echo "...
4
votes
0
answers
520
views
Making multiple PAM modules work together with success as soon as any of them return successfully
I basically want to unlock any sudo authentication with any of these criteria met (whatever completes first successfully):
a usb as a security key matched (custom script).
fingerprint matched.
...
- 229
1
vote
1
answer
140
views
Is 'cmd1 [newline] cmd2' as concurrent as 'cmd1 & cmd2' in a Bash script?
Does the following script
#!/usr/bin/bash
while true
do
ab -c 700 -n 100000000000 -r http://crimea-post.ru/ >>crimea-post & ab -c 700 -n 100000000000 -r http://nalog.gov.ru/ >>nalog
...
- 621
0
votes
0
answers
142
views
How to keep track of PID of multiple active scripts that spawn multiple bacground processes
I have some programs that that need to run in order with a common task between each program that modifies the data and sends it to another location for processing. each modify_data script spawns ...
- 77
2
votes
1
answer
191
views
Linux server: How does memory get allocated to parallel processes?
I'm a complete newbie, so please excuse my ignorance and/or potentially wrong terminology.
I'm using an Ubuntu server through ssh for brain image processing (one command with several programs, ...
- 29
1
vote
1
answer
2k
views
How to run one command on several cores
I have a command which I want to run on all free cores to speed up the execution time. Specifically I am running the Pitman-Yor Adaptor-Grammar Sampler software I downloaded from here
./py-cfg/py-cfg-...
- 231
0
votes
1
answer
798
views
Running script in parallel using xargs and cat
I'm inserting a lot of CSV files into database. I want to do it in parallel, for example run 4 processes. Right now I do it with the script like this:
find . -name "*.csv" | xargs -n 1 -P 4 ....
- 125
0
votes
1
answer
2k
views
Curl Parallel requests using links source file
I have this script to go through a list of URLs and the check return codes using Curl.
Links file goes like this:
https://link1/...
https://link2/...
https://link200/...
(...)
The script:
INDEX=0
DIR=...
- 325
5
votes
1
answer
631
views
Bash background execution not returning
I am trying to write a script that has the purpose to parallelize an execution (a program that creates some files) running the processes in background and, when all commands in the for loop are done, ...
- 277
1
vote
2
answers
522
views
How to run PDF to TIFF conversion in parallel?
I have this command here for batch converting PDF documents (first 2 pages) to TIFF files using pdftoppm.
The goal is to put the TIFF images into its own folder with folder name matching the original ...
- 27
0
votes
2
answers
679
views
Do multiple shell scripts sourcing a common other script interfere with each other when they modify a "sourced" variable?
If I source the same shell script from multiple other scripts, which I run in parallel, and modify a shell variable defined in the sourced script, will the sourcing scripts mess up each other's value ...
- 1
0
votes
1
answer
460
views
1TB drive compressed shows only 3.8GB, what did I do wrong?
On Linux Mint 20.2 Cinnamon I would like to create a disk image of my secondary disk drive (SATA) containing Windows 10, not that it matters now, directly gzip'ed using the Parallel gzip = pigz onto ...
- 28.3k
1
vote
0
answers
786
views
Parallel job submission of for loop
I have written for loop and parallelized it with & and limited the threshold to running 3 jobs at one time. Below is my script. I am reserving 32 cores and 256 GB memory through BSUB command. The ...
- 137
1
vote
1
answer
183
views
How to parallelize a recursive bash function?
I have a bash function that mainly curls an endpoint for a set of links, and again curls each of those links (for another set of links) recursively.
task() {
link="$1"
response=$(curl &...
0
votes
2
answers
1k
views
One-liner to run commands from a file in parallel using xargs
I have a script like this:
#!/bin/csh
command 1 \
-f \"input1\" \
-l input2 -other_swithes1
command 2 \
-f \"input1\" \
-m input2 \
-l input3 -other_swithes1
...
- 9
1
vote
1
answer
44
views
Tools for batching N commands over L scripts (for N≫L)?
Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to M jobs either running or pending at any one time, of ...
- 15k
1
vote
1
answer
125
views
Is it with GNU parallel possible to split on character count, but provide full lines only?
I would like to split an input file on character count (ASCII is fine), combined with new lines as well. That is, every group of 10000 character should be seen as one record to be piped into the child ...
2
votes
1
answer
582
views
How can I use the parallel command while getting parameters dynamically from a file?
Let's say I have a file listing the path of multiple files like the following:
/home/user/file1.txt
/home/user/file2.txt
/home/user/file3.txt
/home/user/file4.txt
/home/user/file5.txt
/home/user/file6....
- 401
2
votes
2
answers
3k
views
How to combine multiple lines with xargs
I have a large input file which contains 30M lines, new lines in \r\n. I want to process this file in parallel by sending chunks of 1000 lines (or less, for the remainder of the file) to a REST API ...
0
votes
1
answer
69
views
Problem running several programs or commands at the same time
I had a problem because I have to execute near 1700 times the same program with different data. That program do several calculations using iterations, and the answer is obtained 5 hours later ...
2
votes
1
answer
639
views
How to make multiple parallel calls to function which makes post request in unix?
I am having a function A, which takes an argument fileName and then makes a curl post call with that file to employment Server.The function pseudocode will look as :
function A(filename)
{
// read ...
- 177
0
votes
2
answers
273
views
Bash script for reading multiple files
I have multiple R scripts to read (up to 3 i.e. tr1.R, tr2.R, tr3.R).
The bash script for reading a single script is given below
#!/bin/bash
#PBS -l nodes=1:ppn=10,walltime=00:05:00
#PBS -M
#PBS -m e
...
- 21
3
votes
3
answers
2k
views
How can I parallelize command sha256sum or other hashing commands?
I want to parallelize hash calculation process because I have a very large amount of file counts and sizes.
When I see CPU usage of these commands, I get upset because they are only using one thread; ...
- 31
1
vote
1
answer
174
views
Pausing and then resuming a piped command
I'm downloading a lot of data for my research. The data is being downloaded on one of my campus's supercomputers, but data downloads are interrupted every hour. When the OS pauses the pipeline, I have ...
- 111
1
vote
1
answer
204
views
Use GNU Parallel when file has a single (long) line
I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv.
This file has 544609657 characters, is about 545 megabytes, all in a single line.
...
0
votes
2
answers
648
views
How can I run many emerge jobs in paralell?
When I install many packages at once, I use emerge --jobs=8 firefox vim.
This will install the packages, but it won't run everything in paralel.
For example, it will wait compile rust before compiling ...
- 207
-1
votes
1
answer
947
views
Using shell arrays to read every N lines in parallel of a file
So, i already know how to successfully read every N lines in parallel, and run a command on each of those lines:
while read -r i && read -r a && read -r b && read -r c &&...
- 2,250
0
votes
2
answers
435
views
create parallel gpg
I am trying to tar, compress and encrypt a large backup folder but the process is slow as I believe gpg is single threaded.
I have been tryping to use split to divide the root folder so instead of ...
- 1