Context: I am using POSIX shared memory to provide a set of processes with a shared memory space. I have used this scheme for some time now in order to share data, and it's working okay. However, I recently ran into an odd problem with a certain class of programs.
Problem: I wrote a program in which each process has to contribute a value to a shared sum in the shared memory space. The sum was initialized to zero when the shared object was mapped into memory earlier. However, when each process tries to add it's part to the shared sum, it can see the latest value, but the result of the addition is always as if it added it's own value with zero. See below:
[21017] Adding 6 to 0!
[21020] Adding 33 to 0!
[21016] Adding 15 to 0!
[21018] Adding 24 to 0!
[21017] Got access! (0x7fe953fcb000 = 0)
[21017] Done (0x7fe953fcb000 = 6)
[21016] Got access! (0x7fe953fcb000 = 6)
[21016] Done (0x7fe953fcb000 = 15)
[21018] Got access! (0x7fe953fcb000 = 15)
[21018] Done (0x7fe953fcb000 = 24)
[21020] Got access! (0x7fe953fcb000 = 24)
[21020] Done (0x7fe953fcb000 = 33)
Sum = 33
Each process "sees" the latest value written, but somehow, after adding it's own component, seems to have ignored the existing value. You can see that each access is ordered sequentially, as there is an access control system managing who gets to write to the shared memory space. The test program used looks as follows (though I do not expect the reader to run it):
int main (void) {
int local_sum = 0, gid = -1;
volatile int *sum;
// Fork for four processes.
for (int i = 1; i < 4; i++) {
if (fork() == 0) break;
}
// Initialize the DSM. Set GID.
sum = (int *)dsm_init(&cfg);
gid = dsm_get_gid();
// Compute range.
for (int i = 0; i < 3; i++) {
local_sum += array[(gid * 3) + i];
}
// Add to local sum.
printf("[%d] Adding %d to %d!\n", getpid(), local_sum, *sum);
*sum = *sum + local_sum;
// Barrier.
dsm_barrier();
// Print sum if process zero.
if (gid == 0) printf("Sum = %d\n", *sum);
// Exit.
dsm_exit();
}
Why is it that each process can "see" the correct value at the address 0x7fe953fcb000
in the shared space, yet after adding, behaves as if the value at that address during the addition was still zero?
Here is what troubles me about this problem:
- If it was a caching issue, how is it that I can print the correct value before the arithmetic operation, and it is still incorrect?
- I am adding to a shared value on the process heap. The compiler could not have assumed that the value there would be zero, and optimized anything out.
Is there some explanation for why this might happen under the hood? I've attempted to use GDB with my program to see what is going on. But as far as I can tell, it is simply moving the value at memory addresses into the registers. I cannot see any optimization problems yet.
cfg
?* sum += some_value
; in the absence of any kind of synchronization it is perfectly possible that they all see* sum
as being zero before the addition. At first sight this program looks very much like the "don't do it like this" example in concurrent programming 101.*sum = *sum + local_sum
is compiled intoLOAD *sum into R0
,LOAD local_sum into R1
,ADD R1 to R0
,STORE R0 to *sum
. The four processes race to execute this sequence. It is perfectly possible that they all doLOAD *sum into R0
before any of them has had a chance to doSTORE R0 to *sum
; in fact, given that, as you say, there is a system call (and thus a replanification point) triggered bySTORE R0 to *sum
, there is a very good chance. You need to synchronize accesses to shared variables, using semaphores for instance.