Questions tagged [concurrency]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
32
votes
4answers
36k views

How to launch a set of program inside tmux or gnome-terminal within a script?

I wanted to launch a few commands simultaneously in tmux or gnome-terminal or xfterminal , every different tab with a single command running , and close that tab when that command finishes. Any ...
23
votes
1answer
21k views

cannot create regular file 'filename': File exists

I've been getting this strange error message in one of my build scripts - cp fails, returning the error "File exists". I've even tried using cp -f, which should overwrite the file if it exists, but ...
19
votes
1answer
12k views

Is using rsync while source is being updated safe?

Is it safe to use rsync when the source filesystem is mounted and writeable, assuming that it may very well be modified while rsync is running? I presume that under race conditions I may not get the ...
16
votes
3answers
8k views

tee + cat: use an output several times and then concatenate results

If I call some command, for instance an echo I can use the results from that command in several other commands with tee. Example: echo "Hello world!" | tee >(command1) >(command2) >(command3)...
12
votes
2answers
6k views

communication between multiple processes

I have a bash script, which runs manager() function as a separate process for x-times. How is it possible to forward messages to all manager() processes from within the script? I've read about ...
11
votes
4answers
3k views

How to atomically allocate a loop device?

I am writing some shell scripts to handle some disk image stuff, and I need to use loop devices to access some disk images. However, I am not sure how to properly allocate a loop device without ...
10
votes
1answer
173 views

Moving a file to a file that is being read

When a file is moved and the destination file already exists and is currently being read, will the operation succeed? Will it depend on how long the reading operation on the destination file lasts?
9
votes
3answers
1k views

Why do package managers need lock files?

dpkg uses a lock file (/var/lib/dpkg/lock), when in use. Why are these lockfiles needed? Why are multiple instances not possible?
9
votes
2answers
429 views

How bash treats “> >()”

While experimenting with output redirection and process substitution I stumbled upon the following command and its resulting output: me@elem:~$ echo foo > >(cat); echo bar bar me@elem:~$ ...
7
votes
2answers
20k views

How do you make a cross-process locking in Linux (C/C++)?

I need to make sure one process executes only in one instance at a time. On Windows you could use named mutex. But I have no idea what to use on Linux. I think I've seen an approach were app creates ...
6
votes
1answer
1k views

Is `ln` atomic and reliable on NFS? Could NFS replace GFS in this use case?

I have a cluster with a bunch of servers with a shared disk containing a GFS global file system that all nodes access simultaneously. Each node in the cluster run the same program (a shell script is ...
5
votes
2answers
3k views

Can there be multiple kernels executing at the same time?

I know that Linux OS's are typically multi-programmed, which means that multiple processes can be active at the same time. Can there be multiple kernels executing at the same time?
5
votes
2answers
3k views

Concurrent writing to a log file from many processes

In mimicing MacOS's terminal's open command or Window's terminal's start, the comments of this answer suggest appending stdout and stderr to ~/.xsession-errors, something like (bash): alias open='&...
5
votes
1answer
3k views

What's the use dirsync option for mount?

Today I stumbled upon this mount's option: dirsync All directory updates within the filesystem should be done synchronously. This affects the following system calls: creat, link, unlink, ...
5
votes
2answers
111 views

Is file creation totally ordered?

On Linux, the openat syscall can be used to create files and to test for their existence. Speaking in terms of the C/C++ memory model, creating a file and verifying its existence creates a ...
5
votes
1answer
1k views

Are pipe reads not greater than PIPE_BUF atomic?

The GNU C library manual briefly mentioned that both reads and writes for a pipe are atomic Reading or writing pipe data is atomic if the size of data written is not greater than PIPE_BUF. However,...
4
votes
3answers
2k views

pipe line with non-deterministic output

I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The ...
4
votes
1answer
2k views

Multiple appenders writing to the same file on NFS share

If I have 100 servers execute a command, and each redirects their output to the same file on an NFS share, will the resultant file be interleaved neatly or can it become corrupt? For example, if this ...
4
votes
2answers
2k views

rm -rf failing if deleting in parallel

rm -rf will fail if something tries to delete the same file tree (I think because rm enumerates the files first, then deletes). A simple test: # Terminal 1 for i in `seq 1 1000`; do mkdir -p /tmp/...
4
votes
2answers
348 views

Can two asynchronous subshell commands safely write to a shared stdout?

Can stdout be overwritten by two bourne (or bash, if that matters) subshell commands running asynchronously? (tail -f ./file1 & tail -f ./file2) | cat I don’t care about the line order, just ...
4
votes
2answers
446 views

How to run multiple processes and exit if any of them exits or fails

I am running multiple processes and want to exit with the appropriate exit code (this means error on failure, success otherwise) if any of them fails or exists. Additionally if any child process ...
4
votes
1answer
1k views

Advantages of using named pipes and sockets rather than temporary files

I have two cooperating programs. One program just writes its output to a file and the other one then reads from the file and spits the data out for the front end to work with. I have been reading ...
4
votes
1answer
131 views

Edge cases in filesystem operations during backup and file copy

I have a question about file access in Linux that I cannot sum it up completely. Consider that I have a home directory that I backup it using rsync triggered via cron. My home directory is on an EXT4 ...
3
votes
2answers
914 views

Is using tar while source directory is being updated safe?

I have to ask question similar to this. In scenarios where you're backing up directory with tar and new files/dirs are being added current files/dirs are being edited and deleted can you expect ...
3
votes
2answers
764 views

Prevent a shell fork from living longer than its initiator?

If I have a Bash script like: function repeat { while :; do echo repeating; sleep 1 done } repeat & echo running once running once is printed once but repeat's fork lives forever,...
3
votes
3answers
720 views

Atomic removal of directory

Because rename(2) is called by mv, is it safe to assume the following would be atomic? $ mv /home/me/someDir /tmp/toBeDeleted $ rm -rf /tmp/toBeDeleted
3
votes
2answers
1k views

Am I hitting a race condition in bash?

I have a script which compares the output of a command with the output of the same command as it was ran previously, it works most of the time, but every now and then it doesn't work as expected. I'...
3
votes
2answers
79 views

Can you read an updated mtime, but when reading the contents they have yet to be updated?

Assuming touch etc. is not involved, just standard open, write, close, link system calls. One process is updating files, another is periodically stating to see if they've changed and reloading the ...
3
votes
1answer
1k views

Controlling bash script concurrency, flock inheritance

I have a bash deployment script that handles deploying updated code to a Tomcat instance on CentOS, however, both Chef and RunDeck may invoke the script, and since Chef runs periodically there is a ...
3
votes
1answer
539 views

How can the order of execution in the shell as it relates to redirection be upset about 1 time out of a 1000 on my system? [duplicate]

Considering this Q&A about the order of execution in the shell as it relates to redirection, and despite the fact that if a file doesn't exist it gets created first so that cat example.txt | shuf &...
3
votes
1answer
137 views

Why does this gawk coprocess hang?

While having a go at Convert date in bash shell, I tried GNU awk's coprocess feature: gawk -F, -v cmd='date +"%Y-%m-%d %H:%M:%S" -f-' '{print $5 |& cmd; cmd |& getline d; $5 = d}1' foo This ...
3
votes
2answers
176 views

Terminate second (parallel) process if first one is terminated

Let's assume we have two shell scripts running in parallel driver.sh: ./proc1.sh& ./proc2.sh& how can I terminate a process if the other one exited with non-zero code? (terminate driver.sh)
2
votes
5answers
5k views

Prevent a second instance of my software from starting

I wrote a service script to run a specific software at boot time. How can I restrict this software so that it does not run manually again? I mean software has to have just one instance in the memory....
2
votes
1answer
739 views

Will an `unlink` or `rename` portably and atomically make a `link` fail?

Question Suppose I have some non-directory (file, named pipe/socket, whatever) at the pathname /tmp/foo and some other non-directory at the pathname /tmp/bar. Then two (or more) processes start ...
2
votes
1answer
823 views

Does linux tar block write access to files

I have a large (>10GB) folder full of images on a live webserver that I need to back up and transfer. I'm worried that if I tar the folder, the files will be blocked for reading by the webserver, ...
2
votes
2answers
3k views

Delete file while it is in use

I have a process that receives a video file (RAW) and transcodes it with FFMPEG generating three (different resolutions) resultant files. I'm using a distributed task queue system (Celery) to process ...
2
votes
2answers
979 views

concurrent curl could not resolve host

I try to run cocurrent curl but it can easily report "Could not resolve host". To run curl parallel, I use "parallel". parallel :::: ./a.sh ./a.sh from api server % Total % Received % Xferd ...
2
votes
1answer
835 views

2 or more programs in same USB serial converter

Having a network of lets say 10 devices in RS-485, connected to the linux PC with a RS-485 to USB Converter. Already there is a C program interacting with the serial converter at /dev/ttyUSB0. My ...
2
votes
1answer
795 views

Parallel read the contents of a zipped file without extraction

I have the following zip archive structure: $ unzip -l Undetermined_S0_L004_R1_001_fastqc.zip Archive: Undetermined_S0_L004_R1_001_fastqc.zip Length Date Time Name -------- ---- ---...
2
votes
2answers
491 views

Linux lockfile starvation

I want to use lockfile functionality for a bash script that will run in parallel. The problem is that in a functionality as in the script below, a starvation can happen in case that the first one to ...
2
votes
1answer
835 views

Concurrency problems with GNU parallel

My scripts are having trouble with correctly running things in GNU parallel. I have a sub_script like so (all these are actually simplified versions): #! /bin/bash input=$1 # input is a date in ...
2
votes
1answer
60 views

Does time window still exist when the signal is not blocked

Many books say that a time window exists between the generation and delivery of the signal when the signal is not blocked. So I wonder whether the time window still exists when the signal is not ...
1
vote
1answer
1k views

Multiple simultaneous instances of apt-get

E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable) E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it? What does these ...
1
vote
1answer
58 views

Scanning a file by multiply antiviruses

I start needing to scan for viruses all files uploaded to my website. I've implemented a pretty simple prototype which I tested locally and it seemed work well. However, I need to make it more complex ...
1
vote
1answer
3k views

Capture user input while doing other things

Is there a way to have the output of a subprocess (something other than 'sleep' below) be dovetailed with the output from a foreground command-loop? For example: while true do echo "updating ...
1
vote
1answer
40 views

How to run multiple 'serial' processes under control considering CPUs available? [duplicate]

Suppose you have a binary that has to be run on maaaaaaaaany files (suppose files are numbered from 1 to N). Each file has to be processed by making a call to this binary (say.... something like ...
1
vote
2answers
484 views

Background processes mix up order of shell script execution

Can someone explain me why the script echo one && echo two & echo three && echo four gives me [1] 3692 three four one two [1]+ Done echo one && ...
1
vote
1answer
2k views

Running commands after ampersand symbol & in a remote ssh session

I have a shell script that initiates a long, resource-intensive command on several different machines. In order to execute the script on each machine in parallel, I have an ampersand symbol after ...
1
vote
1answer
593 views

Lost wakeup problem - how does UNIX deal with it [closed]

(I give context for my question first, the question itself is at the bottom where it says QUESTION in bold). Take two processes A and B. A checks a condition, sees that it isn't satisfied, and goes ...
1
vote
2answers
1k views

Allow one entry at a time in incron job?

I have a directory to monitor. When it is updated, I run some command via incron. When multiple files are copied in this directory, incron execute multiple commands at the same time. Is there any way ...