Questions tagged [parallelism]

Performing tasks in parallel, to make use of multiple processors

Filter by
Sorted by
Tagged with
0
votes
1answer
40 views

Running script in parallel using xargs and cat

I'm inserting a lot of CSV files into database. I want to do it in parallel, for example run 4 processes. Right now I do it with the script like this: find . -name "*.csv" | xargs -n 1 -P 4 ....
-1
votes
1answer
40 views

Curl Parallel requests using links source file

I have this script to go through a list of URLs and the check return codes using Curl. Links file goes like this: https://link1/... https://link2/... https://link200/... (...) The script: INDEX=0 DIR=...
5
votes
1answer
129 views

Bash background execution not returning

I am trying to write a script that has the purpose to parallelize an execution (a program that creates some files) running the processes in background and, when all commands in the for loop are done, ...
1
vote
2answers
75 views

How to run PDF to TIFF conversion in parallel?

I have this command here for batch converting PDF documents (first 2 pages) to TIFF files using pdftoppm. The goal is to put the TIFF images into its own folder with folder name matching the original ...
0
votes
2answers
52 views

Do multiple shell scripts sourcing a common other script interfere with each other when they modify a "sourced" variable?

If I source the same shell script from multiple other scripts, which I run in parallel, and modify a shell variable defined in the sourced script, will the sourcing scripts mess up each other's value ...
0
votes
1answer
88 views

1TB drive compressed shows only 3.8GB, what did I do wrong?

On Linux Mint 20.2 Cinnamon I would like to create a disk image of my secondary disk drive (SATA) containing Windows 10, not that it matters now, directly gzip'ed using the Parallel gzip = pigz onto ...
0
votes
0answers
75 views

Parallel job submission of for loop

I have written for loop and parallelized it with & and limited the threshold to running 3 jobs at one time. Below is my script. I am reserving 32 cores and 256 GB memory through BSUB command. The ...
1
vote
1answer
63 views

How to parallelize a recursive bash function?

I have a bash function that mainly curls an endpoint for a set of links, and again curls each of those links (for another set of links) recursively. task() { link="$1" response=$(curl &...
0
votes
2answers
108 views

One-liner to run commands from a file in parallel using xargs

I have a script like this: #!/bin/csh command 1 \ -f \"input1\" \ -l input2 -other_swithes1 command 2 \ -f \"input1\" \ -m input2 \ -l input3 -other_swithes1 ...
1
vote
1answer
38 views

Tools for batching N commands over L scripts (for N≫L)?

Let's say that I have access to a high-performance Linux cluster equipped with a scheduler (e.g. LSF, Slurm, etc.) that will allow me to have up to M jobs either running or pending at any one time, of ...
1
vote
1answer
75 views

Is it with GNU parallel possible to split on character count, but provide full lines only?

I would like to split an input file on character count (ASCII is fine), combined with new lines as well. That is, every group of 10000 character should be seen as one record to be piped into the child ...
1
vote
1answer
47 views

How can I use the parallel command while getting parameters dynamically from a file?

Let's say I have a file listing the path of multiple files like the following: /home/user/file1.txt /home/user/file2.txt /home/user/file3.txt /home/user/file4.txt /home/user/file5.txt /home/user/file6....
1
vote
2answers
493 views

How to combine multiple lines with xargs

I have a large input file which contains 30M lines, new lines in \r\n. I want to process this file in parallel by sending chunks of 1000 lines (or less, for the remainder of the file) to a REST API ...
0
votes
1answer
50 views

Problem running several programs or commands at the same time

I had a problem because I have to execute near 1700 times the same program with different data. That program do several calculations using iterations, and the answer is obtained 5 hours later ...
2
votes
1answer
95 views

How to make multiple parallel calls to function which makes post request in unix?

I am having a function A, which takes an argument fileName and then makes a curl post call with that file to employment Server.The function pseudocode will look as : function A(filename) { // read ...
0
votes
2answers
74 views

Bash script for reading multiple files

I have multiple R scripts to read (up to 3 i.e. tr1.R, tr2.R, tr3.R). The bash script for reading a single script is given below #!/bin/bash #PBS -l nodes=1:ppn=10,walltime=00:05:00 #PBS -M #PBS -m e ...
2
votes
3answers
247 views

How can I parallelize command sha256sum or other hashing commands?

I want to parallelize hash calculation process because I have a very large amount of file counts and sizes. When I see CPU usage of these commands, I get upset because they are only using one thread; ...
1
vote
1answer
54 views

Pausing and then resuming a piped command

I'm downloading a lot of data for my research. The data is being downloaded on one of my campus's supercomputers, but data downloads are interrupted every hour. When the OS pauses the pipeline, I have ...
1
vote
1answer
56 views

Use GNU Parallel when file has a single (long) line

I have a file which is an HTML document, containing a <table> I want to extract data from and output into a csv. This file has 544609657 characters, is about 545 megabytes, all in a single line. ...
0
votes
2answers
65 views

How can I run many emerge jobs in paralell?

When I install many packages at once, I use emerge --jobs=8 firefox vim. This will install the packages, but it won't run everything in paralel. For example, it will wait compile rust before compiling ...
-1
votes
1answer
144 views

Using shell arrays to read every N lines in parallel of a file

So, i already know how to successfully read every N lines in parallel, and run a command on each of those lines: while read -r i && read -r a && read -r b && read -r c &&...
0
votes
2answers
125 views

create parallel gpg

I am trying to tar, compress and encrypt a large backup folder but the process is slow as I believe gpg is single threaded. I have been tryping to use split to divide the root folder so instead of ...
0
votes
0answers
97 views

Increase Chrome/Firefox's max parallel worker limit?

Our company have a webpage dedicated to processing costs information for our customers locally on their machines via Javascript code. In the past, this processing was sequential, which could easily ...
0
votes
1answer
18 views

Is it possible to continuously feed parallel with more input?

What I want to do is something like this: inotifywait -m ./input -e close_write -e moved_to | parallel --max-args 1 "someSlowSingleThreadedTaskThatActsOnThisFile" I can't imagine that ...
1
vote
1answer
31 views

parallel compare two arguments and send to command

I am trying to run recon-all command with GNU parallel. It should take one argument at {1} like this: parallel --progress subj_id='echo {1} | cut -d / -f 3' && recon-all -s $subj_id -i {1} -sd ...
0
votes
1answer
102 views

How to tune the linux Scheduler for parallel computation?

I have a linux machine dedicated to running some parallel computation, and I'm trying to understand how to choose / tune the scheduler, and perhaps other parameters, to extract the most performance (...
0
votes
2answers
42 views

Will "$!" reliably return the correct ID with "&"?

In my tests, I always get the correct result so far with this: [fabian@manjaro ~]$ sleep 10 & echo $! [1] 302657 302657 But sleep and echo are getting executed simultaneously here, so I would ...
1
vote
1answer
302 views

What is the detriment to downloading too many files in parallel through lftp?

I'm downloading files in parallel through lftp onto a remote server, and I use the commands: lftp ftp:*url* cd into directory mirror --parallel=10 I'm downloading a total of 365 files separated ...
1
vote
1answer
274 views

FIFO-based semaphore explanation

I am trying to work on some parallelization of many processes (task send/to be executed on many (let's say hundreds) nodes). I came across this solution: https://unix.stackexchange.com/a/216475 # ...
1
vote
3answers
1k views

Concurrency and Parallelism on Bash

I'm a bit confused about concurrency and parallelism in the bash shell. As I understand it, when we run commands in more than one subshells at the same time, these commands run in parallel on ...
1
vote
0answers
16 views

How to load test (emulate multiple users) a storage device with bonnie++?

I have just started using bonnie++ for testing the performance of storage devices. So far I have done that emulating a single user but it would be more realistic, or at least an interesting variation, ...
2
votes
2answers
134 views

Process multiple inputs with multiple equivalent commands (multiple thread pools) in GNU parallel

I have a large dataset (>200k files) that I would like to process (convert files into another format). The algorithm is mostly single-threaded, so it would be natural to use parallel processing. ...
1
vote
1answer
55 views

Bash parallel command is executing commands sequentially

I am trying to download multiple files parallelly in bash and I came across GNU parallel. It looks very simple and straight forward. But I am having a hard time getting GNU parallel working. What am I ...
1
vote
2answers
36 views

return code of individual script ,though they are running in parallel

I have jar file which is running in parallel with multiple table names as arguments using nohup command in unix nohup java -jar batchorch.jar $tablename & I have used below lines for getting ...
0
votes
1answer
51 views

SLURM: How do I submit multilple OpenMP parallel codes in a job script

I wanted to run 4 python codes each using 2 processors. (Parallelisation is done inside the MKL.) #!/bin/bash #SBATCH -N 1 #SBATCH --ntasks-per-node=4 #SBATCH --cpus-per-task=2 export OMP_NUM_THREADS=...
1
vote
2answers
146 views

How to get nth result from find?

I have access to a distributed computing/server farm with a job scheduler (Slurm) that gives each parallel job an integer ID from 1 to n (I know the value of n, in the example below, n = 10). I am ...
1
vote
1answer
65 views

appending values in a dynamic array

#!/bin/bash range=$(seq -f "cen%04g" 1052 1099) range1=$(seq -f "rh%04g" 1052 1099) check () { for node1 in ${range};do ping -q -c 1 -w 3 -s 10 $node1 >/dev/null if [ ...
18
votes
4answers
3k views

Are basic POSIX utilities parallelized?

In a common Linux distribution, do utilities like rm, mv, ls, grep, wc, etc. run in parallel on their arguments? In other words, if I grep a huge file on a 32-threaded CPU, will it go faster than on ...
2
votes
2answers
73 views

parallel: Pausing (swapping out) long-running progress when above memory limit threshold

Let's say that I have 10 GBs of RAM and unlimited swap. I want to run 10 jobs in parallel (gnu parallel is an option but not the only one necessarily). These jobs progressively need more and more ...
1
vote
1answer
54 views

Can I write the same tar archive to two tapes at once?

Is it possible to simultaneously write a tar archive to two tape drives using a single command, by directing the same output to both drives? I want to save time by performing the operation in ...
2
votes
2answers
301 views

How do I synchronize appending writes in shell scripts? [duplicate]

I want to write a shell script that accepts a URL and an output markdown file, and adds that URL plus some metadata to the end of that file. It is possible that this script is invoked concurrently, ...
1
vote
1answer
165 views

How can I use fork to calculate partial results of a calculation?

The task that has been given to me is that I have to find all numbers that divide a number (let's call it x) and find out how many times does it divide it. I have to use fork to do the task. What I ...
-1
votes
1answer
319 views

Wildcard variables in xargs for parallel processing

Issue I have an issue using an input variable VAR in a function (i.e. ls) with the wilcard *, when this funciton is within an xargs bash call. This xargs function repeats over each member of a list ...
1
vote
2answers
198 views

GNU Parallel linking arguments with alternating arguments

I made a Python simulator that runs on the basis of user-provided arguments. To use the program, I run multiple random simulations (controlled with a seed value). I use GNU parallel to run the ...
1
vote
1answer
116 views

Pairwise merging of files in parallel until only 1 file left over

I'd like to combine a bunch of language model (LM) count files using SRILM's ngram-merge program. With that program, I could combine a whole directory of count files, e.g. ngram-merge -write ...
1
vote
3answers
92 views

Terminal command follows the lifetime of another terminal command

What I would like to achieve is a bash script or even better a single bash line that can run two terminal based apps on parallel. I am aware of the commands & and ; but in my case they are not ...
3
votes
3answers
1k views

Bash consecutive and parallel loops/commands

I want to run some simulations using a Python tool that I had made. The catch is that I would have to call it multiple times with different parameters/arguments and everything. For now, I am using ...
0
votes
1answer
70 views

Thread safe next number (or round-robin) service in linux

I have an Ubuntu machine and a scenario in which I need to generate the next number (starting from X) between 0 and 20 in a round-robin way. I need a way to get the next number which will be thread ...
2
votes
1answer
78 views

Run script in parallel with incremental variable

I would like to run a python script in parallel on several processors on the files in an INPUTDIR. I use the few lines below for that stuff: # define a function pythonprocessing(){ python myscript....
2
votes
0answers
342 views

Assign result of a command to variables, but don't wait for the command to finish [duplicate]

I am trying to parallelize a bash script that looks like this: var01=$(ssh $USER@host01 'bash -s' < ./node.sh) & var02=$(ssh $USER@host02 'bash -s' < ./node.sh) & var03=$(ssh $USER@...

1
2 3 4 5