Questions tagged [concurrency]
The concurrency tag has no usage guidance.
66
questions
1
vote
1answer
42 views
What happens if a file is renamed during a copy operation using cp on linux?
So I want to use...
cp /var/log/somelogs/* ./temp
... to copy some log files to a temporary folder and analyse them. The problem is that the process generating the log files isn't under my control ...
0
votes
0answers
17 views
Is adding crontabs safe if it's done via a multithreadded application like a web server?
I'm using cronjobs to send email reminders to my clients. What happens is that when an email is first sent to a user, a reminder is scheduled at the same time by adding a cronjob for it using this ...
0
votes
0answers
28 views
bash script update during execution changes behavior
I ran some tests with a simple example script:
#!/bin/bash
sleep 60
echo 'foo'
The test is, run this script in the background, and while it is sleeping, alter the content of the echo command inside ...
3
votes
1answer
300 views
Multiple processes redirecting to the same file with >
This is not a "how to append and not overwrite" question. I'm not looking for a file that combines the output of two commands. It's just a mistake I made and I would like to understand why the system ...
0
votes
0answers
17 views
Can I turn kernel params to make the network stack using swap instead of physical memory?
I'm doing a test about high-currency, and I only have a machine with limited memory. My goal is to setup more than 10M TCP
connections on this memory limited machine. It seems that the network stack ...
5
votes
2answers
114 views
Is file creation totally ordered?
On Linux, the openat syscall can be used to create files and to test for their existence. Speaking in terms of the C/C++ memory model, creating a file and verifying its existence creates a ...
0
votes
0answers
52 views
Background processes in bash preserving order
From a GUI terminal emulator running bash, I would like to open a tabbed instance, get its X window ID, and use it to embed four st terminal instances inside tabbed, with titles from 1 to 4, in ...
5
votes
2answers
821 views
How to run multiple processes and exit if any of them exits or fails
I am running multiple processes and want to exit with the appropriate exit code (this means error on failure, success otherwise) if any of them fails or exists.
Additionally if any child process ...
0
votes
1answer
48 views
Is extracting the first line from a file that is accessed via NFS thread safe?
If I mount a directory via NFS on multiple servers each running a process that reads and deletes the first line of a specific file in that directory, can it somehow be guaranteed that no line is read ...
0
votes
0answers
622 views
How to find concurrent number of connections per second with Apache?
A web-server hosting an application for a client ; which is high-volume or transaction server, it is required to find out "number of concurrent connections" currently in use by the server. This is ...
4
votes
2answers
509 views
Can two asynchronous subshell commands safely write to a shared stdout?
Can stdout be overwritten by two bourne (or bash, if that matters) subshell commands running asynchronously?
(tail -f ./file1 & tail -f ./file2) | cat
I don’t care about the line order, just ...
1
vote
1answer
42 views
How to run multiple 'serial' processes under control considering CPUs available? [duplicate]
Suppose you have a binary that has to be run on maaaaaaaaany files (suppose files are numbered from 1 to N). Each file has to be processed by making a call to this binary (say.... something like ...
1
vote
1answer
911 views
Lost wakeup problem - how does UNIX deal with it [closed]
(I give context for my question first, the question itself is at the bottom where it says QUESTION in bold).
Take two processes A and B. A checks a condition, sees that it isn't satisfied, and goes ...
3
votes
1answer
176 views
Why does this gawk coprocess hang?
While having a go at Convert date in bash shell, I tried GNU awk's coprocess feature:
gawk -F, -v cmd='date +"%Y-%m-%d %H:%M:%S" -f-' '{print $5 |& cmd; cmd |& getline d; $5 = d}1' foo
This ...
3
votes
2answers
1k views
Prevent a shell fork from living longer than its initiator?
If I have a Bash script like:
function repeat {
while :; do
echo repeating; sleep 1
done
}
repeat &
echo running once
running once is printed once but repeat's fork lives forever,...
3
votes
2answers
245 views
Terminate second (parallel) process if first one is terminated
Let's assume we have two shell scripts running in parallel
driver.sh:
./proc1.sh&
./proc2.sh&
how can I terminate a process if the other one exited with non-zero code? (terminate driver.sh)
3
votes
2answers
108 views
Can you read an updated mtime, but when reading the contents they have yet to be updated?
Assuming touch etc. is not involved, just standard open, write, close, link system calls.
One process is updating files, another is periodically stating to see if they've changed and reloading the ...
5
votes
1answer
2k views
Are pipe reads not greater than PIPE_BUF atomic?
The GNU C library manual briefly mentioned that both reads and writes for a pipe are atomic
Reading or writing pipe data is atomic if the size of data written is not greater than PIPE_BUF.
However,...
6
votes
2answers
6k views
Concurrent writing to a log file from many processes
In mimicing MacOS's terminal's open command or Window's terminal's start, the comments of this answer suggest appending stdout and stderr to ~/.xsession-errors, something like (bash):
alias open='&...
4
votes
2answers
1k views
Is using tar while source directory is being updated safe?
I have to ask question similar to this.
In scenarios where you're backing up directory with tar and
new files/dirs are being added
current files/dirs are being edited and deleted
can you expect ...
2
votes
1answer
939 views
Parallel read the contents of a zipped file without extraction
I have the following zip archive structure:
$ unzip -l Undetermined_S0_L004_R1_001_fastqc.zip
Archive: Undetermined_S0_L004_R1_001_fastqc.zip
Length Date Time Name
-------- ---- ---...
1
vote
1answer
359 views
Weird results of triple piping to STDOUT & tee /dev/null >(wc -l > tmp.txt) & piping again embedding `cat tmp.txt`
$ seq 1 12773 | tee /dev/null >(wc -l > tmp.txt) | head -$((0x`openssl rand -hex 7` % `cat tmp.txt` + 1))|tail -1
--> 8473 (random between 1~12773)
$ cat tmp.txt
--> 8473
$ seq 1 12774 | tee ...
0
votes
1answer
2k views
lock bash script to prevent parallel running from within the script itself?
I am aware of using flock and solo (not sure the difference though) when running from a cron job. However, is it possible to prevent parallel running from within the script itself?
3
votes
1answer
935 views
Will an `unlink` or `rename` portably and atomically make a `link` fail?
Question
Suppose I have some non-directory (file, named pipe/socket, whatever) at the pathname /tmp/foo and some other non-directory at the pathname /tmp/bar. Then two (or more) processes start ...
2
votes
2answers
1k views
concurrent curl could not resolve host
I try to run cocurrent curl but it can easily report "Could not resolve host". To run curl parallel, I use "parallel".
parallel :::: ./a.sh ./a.sh
from api server
% Total % Received % Xferd ...
4
votes
1answer
3k views
Multiple appenders writing to the same file on NFS share
If I have 100 servers execute a command, and each redirects their output to the same file on an NFS share, will the resultant file be interleaved neatly or can it become corrupt?
For example, if this ...
2
votes
1answer
1k views
Does linux tar block write access to files
I have a large (>10GB) folder full of images on a live webserver that I need to back up and transfer.
I'm worried that if I tar the folder, the files will be blocked for reading by the webserver, ...
1
vote
2answers
1k views
Allow one entry at a time in incron job?
I have a directory to monitor. When it is updated, I run some command via incron. When multiple files are copied in this directory, incron execute multiple commands at the same time. Is there any way ...
0
votes
2answers
22 views
running codes simultaneously while outputting result one by one
I have a scripts looks like:
for simplify in 0.1 ;do
for lmbda in 0.9 1.1 1.3;do
for mu in 2.1 3.4 4.2;do
rm eci.out;
csce.py --mu $mu --lmbda $lmbda --simplify $simplify \
...
2
votes
2answers
5k views
Delete file while it is in use
I have a process that receives a video file (RAW) and transcodes it with FFMPEG generating three (different resolutions) resultant files. I'm using a distributed task queue system (Celery) to process ...
1
vote
1answer
1k views
Multiple simultaneous instances of apt-get
E: Could not get lock /var/lib/dpkg/lock - open (11: Resource temporarily unavailable)
E: Unable to lock the administration directory (/var/lib/dpkg/), is another process using it?
What does these ...
1
vote
0answers
30 views
Limit cpu usage to one core [duplicate]
I want to run a server with user-written scripts. But these scripts should only use one core (which should be the one with the lowest usage because there will be many concurrent scripts). Also I need ...
11
votes
4answers
4k views
How to atomically allocate a loop device?
I am writing some shell scripts to handle some disk image stuff, and I need to use loop devices to access some disk images. However, I am not sure how to properly allocate a loop device without ...
1
vote
2answers
609 views
Background processes mix up order of shell script execution
Can someone explain me why the script
echo one &&
echo two &
echo three &&
echo four
gives me
[1] 3692
three
four
one
two
[1]+ Done echo one && ...
9
votes
2answers
486 views
How bash treats “> >()”
While experimenting with output redirection and process substitution I stumbled upon the following command and its resulting output:
me@elem:~$ echo foo > >(cat); echo bar
bar
me@elem:~$ ...
2
votes
2answers
589 views
Linux lockfile starvation
I want to use lockfile functionality for a bash script that will run in parallel. The problem is that in a functionality as in the script below, a starvation can happen in case that the first one to ...
3
votes
3answers
927 views
Atomic removal of directory
Because rename(2) is called by mv, is it safe to assume the following would be atomic?
$ mv /home/me/someDir /tmp/toBeDeleted
$ rm -rf /tmp/toBeDeleted
3
votes
2answers
2k views
Am I hitting a race condition in bash?
I have a script which compares the output of a command with the output of the same command as it was ran previously, it works most of the time, but every now and then it doesn't work as expected.
I'...
1
vote
2answers
2k views
How to run “&” operation sequentially
From my script I am running following commands:
SCRIPT_1="ksh -x script1.sh & ksh -x script2.sh";
SCRIPT_2="ksh -x script3.sh & ksh -x script4.sh";
eval $SCRIPT_1;
sleep 20s;
eval $...
2
votes
1answer
948 views
Concurrency problems with GNU parallel
My scripts are having trouble with correctly running things in GNU parallel.
I have a sub_script like so (all these are actually simplified versions):
#! /bin/bash
input=$1
# input is a date in ...
1
vote
1answer
2k views
Running commands after ampersand symbol & in a remote ssh session
I have a shell script that initiates a long, resource-intensive command on several different machines.
In order to execute the script on each machine in parallel, I have an ampersand symbol after ...
4
votes
1answer
2k views
Advantages of using named pipes and sockets rather than temporary files
I have two cooperating programs. One program just writes its output to a file and the other one then reads from the file and spits the data out for the front end to work with.
I have been reading ...
2
votes
5answers
5k views
Prevent a second instance of my software from starting
I wrote a service script to run a specific software at boot time. How can I restrict this software so that it does not run manually again?
I mean software has to have just one instance in the memory....
4
votes
3answers
2k views
pipe line with non-deterministic output
I have happened upon a command that sometimes works and sometimes does not, even when executed multiple times in rapid succession in a bash shell (I have not tested the behavior in other shells). The ...
3
votes
1answer
1k views
Controlling bash script concurrency, flock inheritance
I have a bash deployment script that handles deploying updated code to a Tomcat instance on CentOS, however, both Chef and RunDeck may invoke the script, and since Chef runs periodically there is a ...
10
votes
3answers
1k views
Why do package managers need lock files?
dpkg uses a lock file (/var/lib/dpkg/lock), when in use.
Why are these lockfiles needed?
Why are multiple instances not possible?
13
votes
2answers
7k views
communication between multiple processes
I have a bash script, which runs manager() function as a separate process for x-times. How is it possible to forward messages to all manager() processes from within the script?
I've read about ...
-3
votes
1answer
2k views
Run series of command independent of each other
If I have a file with
cmd1
cmd2
cmd3
And I want to run them independent of each other, and independent whether it outputs an exit status or not (that means do not expect to end, just keep ...
6
votes
1answer
1k views
Is `ln` atomic and reliable on NFS? Could NFS replace GFS in this use case?
I have a cluster with a bunch of servers with a shared disk containing a GFS global file system that all nodes access simultaneously.
Each node in the cluster run the same program (a shell script is ...
2
votes
1answer
61 views
Does time window still exist when the signal is not blocked
Many books say that a time window exists between the generation and delivery of the signal when the signal is not blocked.
So I wonder whether the time window still exists when the signal is not ...
