I have two co-operating programs X and Y running on the same system. X some times uses Y to calculate some information. I use file based communication between the programs. When X wants Y to compute something, the process goes as follows:
- X writes a job description into a file job.txt and places it into a shared directory. Then X starts to poll for existence of a file ready.txt in that same directory.
- Y polls for existence of the file job.txt. When the file appears, Y first reads the job information from the file and deletes it. Then Y executes the job and places the result into result.txt. Finally Y creates a file ready.txt and again starts to poll for existence of job.txt.
- When X notices the appearance of ready.txt, it reads result.txt, deletes both ready.txt and result.txt, and then continues to do something else.
The problem is that X sometimes receives only an empty or partial result file. Y uses a bash script that currently does something like this:
rm -f tmp_result.txt for ((i=first; i <= last; i++)) # Each iteration produces 1 or more result lines do # Compute something... echo "One result line with e.g. $values" >> tmp_result.txt done cp tmp_result.txt result.txt touch ready.txt
That is, the contents of the result file are printed one line at a time into a temporary file using echo and I/O redirection. When the contents are (or should be!) ready, they are copied into the final result.txt file and X is notified about this by creating ready.txt.
Even after quite extensive experimentation and googling around, I have been unable to find a way to ensure that X always receives complete results. Perhaps 1 to 5 times out of 100, X will see result.txt as completely empty (the most common case) or containing only partial results.
A couple of clarifications about the logic of the script: I originally echoed the result lines directly to result.txt, but this was very unreliable (perhaps 1 out of 2 results were incomplete). Then I changed to first echo the lines into tmp_result.txt and then move (rename) tmp_result.txt into result.txt. This resulted in perhaps 1 out of 10 results being incomplete. Copying the file, as shown above, has worked best, but still occasionally fails.
So how can I ensure that all echoed lines have been properly copied into result.txt before X starts to read the file? Y has only some core bash facilities available.
2 3 additional comments
- I have verified (e.g. by not deleting tmp_result.txt and then inspecting it afterwards) that the problem is not in the original results but in how they are transferred to X.
- Both tmp_result.txt and result.txt reside on tmpfs-based RAM disks, so I would expect copying / moving files to be very fast.
- I first ran into the problem after starting to use tmpfs (originally all files resided in a normal hard disk partition).