3

I have two co-operating programs X and Y running on the same system. X some times uses Y to calculate some information. I use file based communication between the programs. When X wants Y to compute something, the process goes as follows:

  1. X writes a job description into a file job.txt and places it into a shared directory. Then X starts to poll for existence of a file ready.txt in that same directory.
  2. Y polls for existence of the file job.txt. When the file appears, Y first reads the job information from the file and deletes it. Then Y executes the job and places the result into result.txt. Finally Y creates a file ready.txt and again starts to poll for existence of job.txt.
  3. When X notices the appearance of ready.txt, it reads result.txt, deletes both ready.txt and result.txt, and then continues to do something else.

The problem is that X sometimes receives only an empty or partial result file. Y uses a bash script that currently does something like this:

rm -f tmp_result.txt
for ((i=first; i <= last; i++)) # Each iteration produces 1 or more result lines
do
  # Compute something...
  echo "One result line with e.g. $values" >> tmp_result.txt
done
cp tmp_result.txt result.txt
touch ready.txt

That is, the contents of the result file are printed one line at a time into a temporary file using echo and I/O redirection. When the contents are (or should be!) ready, they are copied into the final result.txt file and X is notified about this by creating ready.txt.

Even after quite extensive experimentation and googling around, I have been unable to find a way to ensure that X always receives complete results. Perhaps 1 to 5 times out of 100, X will see result.txt as completely empty (the most common case) or containing only partial results.

A couple of clarifications about the logic of the script: I originally echoed the result lines directly to result.txt, but this was very unreliable (perhaps 1 out of 2 results were incomplete). Then I changed to first echo the lines into tmp_result.txt and then move (rename) tmp_result.txt into result.txt. This resulted in perhaps 1 out of 10 results being incomplete. Copying the file, as shown above, has worked best, but still occasionally fails.

So how can I ensure that all echoed lines have been properly copied into result.txt before X starts to read the file? Y has only some core bash facilities available.

EDIT: 2 3 additional comments

  1. I have verified (e.g. by not deleting tmp_result.txt and then inspecting it afterwards) that the problem is not in the original results but in how they are transferred to X.
  2. Both tmp_result.txt and result.txt reside on tmpfs-based RAM disks, so I would expect copying / moving files to be very fast.
  3. I first ran into the problem after starting to use tmpfs (originally all files resided in a normal hard disk partition).

closed as unclear what you're asking by Gilles, Jakuje, user79743, G-Man, slm Feb 9 '16 at 0:38

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    What you're doing should work: assuming that ready.txt doesn't exist at the beginning of the script, it isn't created until result.txt is fully written. Moving tmp_result.txt to result.txt should also work, and is the usual way to do this. This makes me think that you oversimplified the problem, and what you're doing is in fact not exactly what you describe. Please post complete code that reproduces the problem, both the producer and the consumer. What OS are you running this on? – Gilles Feb 8 '16 at 0:01
  • The underlying programs consist of hundreds and hundreds of lines of Java code, so perhaps I will not post them... But I am quite sure that the problem has something to do with buffering etc. because there have been cases where X first sees result.txt as empty, but setting X to try to access the file again results in at least some content to be available. I wonder if it is important that the problem appeared when I started to use tmpfs instead of hard disk for storing the result and ready files? The OS is Ubuntu 14.04. – user2781185 Feb 8 '16 at 20:58
  • 1
    Given that the part you posted works correctly (I can tell both because I understand the semantics of these programs, and because I tested your snippet with a reasonable implementation of the parts you didn't post), by elimination, the problem must be in some part that you didn't post. So please post a small but complete code that allows us to reproduce the problem. Until you do that, we can't really help you. – Gilles Feb 8 '16 at 21:11
  • 1
    In what you posted, the problem cannot possibly be buffering, because no buffering is going on at any point that matters, unless you're running into an implausible Linux bug (implausible because the OS you're running is used by millions of people and a bug of this magnitude would surely have been noticed and fixed a long time ago). (The problem could be buffering on the consumer side, but since you didn't post that, there's no way for us to tell.) Switching to tmpfs probably didn't cause the problem, but made it more common because of the speed. – Gilles Feb 8 '16 at 21:13
2

It looks like you could use FIFOs for this:

mkfifo tasks

Producer:

for i in {1..10}; do echo "task$i"; done > tasks

Consumer:

while read task; do echo "received $task"; done  <tasks

With this, you don't have to take care of synchronization or deletion, and you don't waste any CPU time on polling -- if there's no data or if the producer hasn't opened the FIFO yet, the consumer will block (and thereby save CPU time). If the FIFO is full or the consumer hasn't opened the FIFO, the producer will block.

  • 1
    My thoughts exactly. A named pipe aka First In First Out is the best solution for this problem. – Wildcard Feb 8 '16 at 21:57
0

Try to put sync between your cp command and touch ready.txt.

  • I tested this, and it seems to work fairly well. Only about 1 case out of 150 failed. Looking at sync man page, the problem here still seems to be that the sync command only notifies the kernel about the sync request and immediately returns: "On Linux, sync is only guaranteed to schedule the dirty blocks for writing; it can actually take a short time before all the blocks are finally written". I would like to find a "blocking" command that allows the script to proceed to the next command only after the flush/sync has been completed. – user2781185 Feb 7 '16 at 23:23
  • 1
    @user2781185 Running sync is completely irrelevant here. All sync does is to flush writes to the disk, which actually does nothing for tmpfs. If running sync helps, it's only because it introduces a delay, and a small sleep would achieve as much. – Gilles Feb 8 '16 at 0:02
  • I can only agree. tmpfs does change the game here. – Thomas Feb 8 '16 at 9:33
  • I first made a mistake: in fact the final result.txt was on hard disk. But the situation did not really change when I placed it to the same tmpfs as the tmp_result.txt. – user2781185 Feb 8 '16 at 20:44

Not the answer you're looking for? Browse other questions tagged or ask your own question.