Questions tagged [concurrency]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
68 views

Linux pipe/redirect creates file with size `0`; then eventually writes all data to file; how to avoid

I'm having a hard time googling this, so please point to available duplicates (better phrasing of title). I wonder, if I have a long running command redirect into a file, the file gets created ...
Joel's user avatar
  • 141
5 votes
3 answers
638 views

Lock a bash script based on parameter?

I'm trying to find a way to lock a script based on a parameter given, but was unsuccessful in finding a proper answer. What I'm trying to achieve is prevent another user from running a script based on ...
classicmusiclover's user avatar
0 votes
1 answer
147 views

Prevent process from overwriting files

The setting Let's say I have an executable file, let's call it program, whose source code is unavailable (maybe proprietary/legacy). Each time this program is executed, it generates a file, let's call ...
Álvaro G. Tenorio's user avatar
1 vote
1 answer
184 views

Run subset of commands in parallel; once one command is done, run another

Let's say I have N bash commands stored in a file: $ cat list_of_commands.txt do_thing_1 do_thing_2 ... do_thing_N The goal is to run them in parallel, but only 10 at a time (to avoid overloading the ...
rijkdw's user avatar
  • 13
2 votes
3 answers
859 views

Execute several scripts at the same time with nohup

I want to execute four scripts at the same time with nohup. These scripts are all in the same folder. Would something like this work?: nohup /.scrip1.sh & /.scrip2.sh & /.scrip3.sh & /....
RoyBatty279's user avatar
4 votes
1 answer
665 views

What is the minimum amount of memory required for starting a process on a Linux-based system?

Routines in both languages are inexpensive: goroutines are 2KB each, while Elixir processes are 0.5KB each. I understand that to start a process in BEAM requires 0.5KB of memory. This being so ...
John Smith's user avatar
1 vote
2 answers
799 views

How might I execute this nested for loop in parallel?

#!/usr/bin/bash TARGETS=( "81.176.235.2" "81.176.70.2" "78.41.109.7" ) myIPs=( "185.164.100.1" "185.164.100.2" "185.164.100.3" "185....
John Smith's user avatar
0 votes
1 answer
503 views

Testing file locking

I have a script which locks a file to avoid concurrent access to it, How can I execute this same script from two different terminals synchronously, to check if it works? Here is the script #!/bin/bash ...
Rob's user avatar
  • 101
0 votes
1 answer
59 views

Asynchronous downloading from virtual machine?

I have some files stored on a virtual machine that I'm downloading onto my PC. There are approximately 1 million files and I have been using the following command: scp vm_user@IP:/home/vm_user/...
dollar bill's user avatar
0 votes
1 answer
94 views

different process in Linux in a single core PC how been managed?

(this a dummy question)multiple processes are running in the background. my understanding is each CPU core can run only 1 process at a time. so someone has the ability to interrupt the current process(...
kankan256's user avatar
  • 103
3 votes
2 answers
210 views

Concurrency of a find, hash val, and replace across large amount of rows

I have a bunch of files and for each row there is a unique value I'm trying to obscure with a hash. However there are 3M rows across the files and a rough calculation of the time needed to complete ...
Liam Pieri's user avatar
-1 votes
1 answer
47 views

Does inetd simplify server programs, without affecting the part of concurrently handling multiple clients? [closed]

In The Linux Programming Interface, Chapter 60 talks about designing a server to concurrently handle multiple clients, by using sockets, multiple processes/threads or thread/process pools designing ...
Tim's user avatar
  • 99.1k
2 votes
3 answers
4k views

Concurrency and Parallelism on Bash

I'm a bit confused about concurrency and parallelism in the bash shell. As I understand it, when we run commands in more than one subshells at the same time, these commands run in parallel on ...
testter's user avatar
  • 1,310
1 vote
1 answer
191 views

concurrent loops without using a subprocess

I'm working on a script in which I'm trying to have two concurrent timers within the same process. In the following example, I'm trying to update the ip address every 60 seconds, while outputting the ...
Subbeh's user avatar
  • 430
1 vote
0 answers
39 views

Shell output of `ls | cat > new_file ` [duplicate]

The behavior of dash (and also bash) looks non-intuitive in this case. It seems that new_file is created before ls is executed. Is it specified in POSIX?
gadm's user avatar
  • 11
1 vote
1 answer
159 views

What happens if a file is renamed during a copy operation using cp on linux?

So I want to use... cp /var/log/somelogs/* ./temp ... to copy some log files to a temporary folder and analyse them. The problem is that the process generating the log files isn't under my control ...
Roger Heathcote's user avatar
4 votes
1 answer
3k views

Multiple processes redirecting to the same file with >

This is not a "how to append and not overwrite" question. I'm not looking for a file that combines the output of two commands. It's just a mistake I made and I would like to understand why the system ...
golimar's user avatar
  • 387
0 votes
0 answers
42 views

Can I turn kernel params to make the network stack using swap instead of physical memory?

I'm doing a test about high-currency, and I only have a machine with limited memory. My goal is to setup more than 10M TCP connections on this memory limited machine. It seems that the network stack ...
Myrfy's user avatar
  • 101
3 votes
2 answers
142 views

Is file creation totally ordered?

On Linux, the openat syscall can be used to create files and to test for their existence. Speaking in terms of the C/C++ memory model, creating a file and verifying its existence creates a ...
Kaz Wesley's user avatar
0 votes
0 answers
178 views

Background processes in bash preserving order

From a GUI terminal emulator running bash, I would like to open a tabbed instance, get its X window ID, and use it to embed four st terminal instances inside tabbed, with titles from 1 to 4, in ...
BowPark's user avatar
  • 4,821
8 votes
2 answers
4k views

How to run multiple processes and exit if any of them exits or fails

I am running multiple processes and want to exit with the appropriate exit code (this means error on failure, success otherwise) if any of them fails or exists. Additionally if any child process ...
Olivenbaum's user avatar
0 votes
1 answer
96 views

Is extracting the first line from a file that is accessed via NFS thread safe?

If I mount a directory via NFS on multiple servers each running a process that reads and deletes the first line of a specific file in that directory, can it somehow be guaranteed that no line is read ...
sge's user avatar
  • 101
0 votes
0 answers
992 views

How to find concurrent number of connections per second with Apache?

A web-server hosting an application for a client ; which is high-volume or transaction server, it is required to find out "number of concurrent connections" currently in use by the server. This is ...
asadz's user avatar
  • 233
5 votes
2 answers
1k views

Can two asynchronous subshell commands safely write to a shared stdout?

Can stdout be overwritten by two bourne (or bash, if that matters) subshell commands running asynchronously? (tail -f ./file1 & tail -f ./file2) | cat I don’t care about the line order, just ...
Daniel's user avatar
  • 1,207
1 vote
1 answer
53 views

How to run multiple 'serial' processes under control considering CPUs available? [duplicate]

Suppose you have a binary that has to be run on maaaaaaaaany files (suppose files are numbered from 1 to N). Each file has to be processed by making a call to this binary (say.... something like ...
eftshift0's user avatar
  • 555
2 votes
1 answer
2k views

Lost wakeup problem - how does UNIX deal with it [closed]

(I give context for my question first, the question itself is at the bottom where it says QUESTION in bold). Take two processes A and B. A checks a condition, sees that it isn't satisfied, and goes ...
G. Bach's user avatar
  • 121
3 votes
1 answer
380 views

Why does this gawk coprocess hang?

While having a go at Convert date in bash shell, I tried GNU awk's coprocess feature: gawk -F, -v cmd='date +"%Y-%m-%d %H:%M:%S" -f-' '{print $5 |& cmd; cmd |& getline d; $5 = d}1' foo This ...
muru's user avatar
  • 70.2k
5 votes
2 answers
2k views

Prevent a shell fork from living longer than its initiator?

If I have a Bash script like: function repeat { while :; do echo repeating; sleep 1 done } repeat & echo running once running once is printed once but repeat's fork lives forever,...
cat's user avatar
  • 3,438
3 votes
2 answers
404 views

Terminate second (parallel) process if first one is terminated

Let's assume we have two shell scripts running in parallel driver.sh: ./proc1.sh& ./proc2.sh& how can I terminate a process if the other one exited with non-zero code? (terminate driver.sh)
Mahyar's user avatar
  • 131
3 votes
2 answers
332 views

Can you read an updated mtime, but when reading the contents they have yet to be updated?

Assuming touch etc. is not involved, just standard open, write, close, link system calls. One process is updating files, another is periodically stating to see if they've changed and reloading the ...
OrangeDog's user avatar
  • 871
6 votes
1 answer
4k views

Are pipe reads not greater than PIPE_BUF atomic?

The GNU C library manual briefly mentioned that both reads and writes for a pipe are atomic Reading or writing pipe data is atomic if the size of data written is not greater than PIPE_BUF. However,...
Igor Bukanov's user avatar
12 votes
2 answers
22k views

Concurrent writing to a log file from many processes

In mimicing MacOS's terminal's open command or Window's terminal's start, the comments of this answer suggest appending stdout and stderr to ~/.xsession-errors, something like (bash): alias open='&...
Tom Hale's user avatar
  • 28.9k
4 votes
2 answers
2k views

Is using tar while source directory is being updated safe?

I have to ask question similar to this. In scenarios where you're backing up directory with tar and new files/dirs are being added current files/dirs are being edited and deleted can you expect ...
Miloš Đakonović's user avatar
2 votes
1 answer
1k views

Parallel read the contents of a zipped file without extraction

I have the following zip archive structure: $ unzip -l Undetermined_S0_L004_R1_001_fastqc.zip Archive: Undetermined_S0_L004_R1_001_fastqc.zip Length Date Time Name -------- ---- ---...
user977828's user avatar
1 vote
1 answer
511 views

Weird results of triple piping to STDOUT & tee /dev/null >(wc -l > tmp.txt) & piping again embedding `cat tmp.txt`

$ seq 1 12773 | tee /dev/null >(wc -l > tmp.txt) | head -$((0x`openssl rand -hex 7` % `cat tmp.txt` + 1))|tail -1 --> 8473 (random between 1~12773) $ cat tmp.txt --> 8473 $ seq 1 12774 | tee ...
user58029's user avatar
0 votes
1 answer
3k views

lock bash script to prevent parallel running from within the script itself?

I am aware of using flock and solo (not sure the difference though) when running from a cron job. However, is it possible to prevent parallel running from within the script itself?
eekfonky's user avatar
  • 655
3 votes
1 answer
2k views

Will an `unlink` or `rename` portably and atomically make a `link` fail?

Question Suppose I have some non-directory (file, named pipe/socket, whatever) at the pathname /tmp/foo and some other non-directory at the pathname /tmp/bar. Then two (or more) processes start ...
mtraceur's user avatar
  • 1,146
2 votes
2 answers
2k views

concurrent curl could not resolve host

I try to run cocurrent curl but it can easily report "Could not resolve host". To run curl parallel, I use "parallel". parallel :::: ./a.sh ./a.sh from api server % Total % Received % Xferd ...
cwhsu's user avatar
  • 123
7 votes
1 answer
7k views

Multiple appenders writing to the same file on NFS share

If I have 100 servers execute a command, and each redirects their output to the same file on an NFS share, will the resultant file be interleaved neatly or can it become corrupt? For example, if this ...
Fidel's user avatar
  • 405
2 votes
1 answer
2k views

Does linux tar block write access to files

I have a large (>10GB) folder full of images on a live webserver that I need to back up and transfer. I'm worried that if I tar the folder, the files will be blocked for reading by the webserver, ...
Ucinorn's user avatar
  • 123
1 vote
2 answers
2k views

Allow one entry at a time in incron job?

I have a directory to monitor. When it is updated, I run some command via incron. When multiple files are copied in this directory, incron execute multiple commands at the same time. Is there any way ...
Hafiz Muhammad Shafiq's user avatar
1 vote
2 answers
28 views

running codes simultaneously while outputting result one by one

I have a scripts looks like: for simplify in 0.1 ;do for lmbda in 0.9 1.1 1.3;do for mu in 2.1 3.4 4.2;do rm eci.out; csce.py --mu $mu --lmbda $lmbda --simplify $simplify \ ...
user40780's user avatar
  • 1,841
7 votes
2 answers
11k views

Delete file while it is in use

I have a process that receives a video file (RAW) and transcodes it with FFMPEG generating three (different resolutions) resultant files. I'm using a distributed task queue system (Celery) to process ...
Mauro Baraldi's user avatar
3 votes
1 answer
2k views

Multiple simultaneous instances of apt-get

E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable) E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? What does these ...
Mahesh Dhoni's user avatar
1 vote
0 answers
35 views

Limit cpu usage to one core [duplicate]

I want to run a server with user-written scripts. But these scripts should only use one core (which should be the one with the lowest usage because there will be many concurrent scripts). Also I need ...
Armin's user avatar
  • 131
14 votes
4 answers
6k views

How to atomically allocate a loop device?

I am writing some shell scripts to handle some disk image stuff, and I need to use loop devices to access some disk images. However, I am not sure how to properly allocate a loop device without ...
AJMansfield's user avatar
1 vote
2 answers
1k views

Background processes mix up order of shell script execution

Can someone explain me why the script echo one && echo two & echo three && echo four gives me [1] 3692 three four one two [1]+ Done echo one && ...
nnn's user avatar
  • 1,151
10 votes
2 answers
748 views

How bash treats "> >()"

While experimenting with output redirection and process substitution I stumbled upon the following command and its resulting output: me@elem:~$ echo foo > >(cat); echo bar bar me@elem:~$ ...
Stanley Yu's user avatar
2 votes
2 answers
977 views

Linux lockfile starvation

I want to use lockfile functionality for a bash script that will run in parallel. The problem is that in a functionality as in the script below, a starvation can happen in case that the first one to ...
csny's user avatar
  • 1,475
3 votes
3 answers
2k views

Atomic removal of directory

Because rename(2) is called by mv, is it safe to assume the following would be atomic? $ mv /home/me/someDir /tmp/toBeDeleted $ rm -rf /tmp/toBeDeleted
jibsales's user avatar
  • 285