Processor not seeing changes to POSIX shared memory?

Question

Context: I am using POSIX shared memory to provide a set of processes with a shared memory space. I have used this scheme for some time now in order to share data, and it's working okay. However, I recently ran into an odd problem with a certain class of programs.

Problem: I wrote a program in which each process has to contribute a value to a shared sum in the shared memory space. The sum was initialized to zero when the shared object was mapped into memory earlier. However, when each process tries to add it's part to the shared sum, it can see the latest value, but the result of the addition is always as if it added it's own value with zero. See below:

[21017] Adding 6 to 0!
[21020] Adding 33 to 0!
[21016] Adding 15 to 0!
[21018] Adding 24 to 0!
[21017] Got access! (0x7fe953fcb000 = 0)
[21017] Done (0x7fe953fcb000 = 6)
[21016] Got access! (0x7fe953fcb000 = 6)
[21016] Done (0x7fe953fcb000 = 15)
[21018] Got access! (0x7fe953fcb000 = 15)
[21018] Done (0x7fe953fcb000 = 24)
[21020] Got access! (0x7fe953fcb000 = 24)
[21020] Done (0x7fe953fcb000 = 33)
Sum = 33

Each process "sees" the latest value written, but somehow, after adding it's own component, seems to have ignored the existing value. You can see that each access is ordered sequentially, as there is an access control system managing who gets to write to the shared memory space. The test program used looks as follows (though I do not expect the reader to run it):

int main (void) {
    int local_sum = 0, gid = -1;
    volatile int *sum;

    // Fork for four processes.
    for (int i = 1; i < 4; i++) {
        if (fork() == 0) break;
    }

    // Initialize the DSM. Set GID.
    sum = (int *)dsm_init(&cfg);
    gid = dsm_get_gid();

    // Compute range.
    for (int i = 0; i < 3; i++) {
        local_sum += array[(gid * 3) + i];
    }

    // Add to local sum.
    printf("[%d] Adding %d to %d!\n", getpid(), local_sum, *sum);
    *sum = *sum + local_sum;

    // Barrier.
    dsm_barrier();

    // Print sum if process zero.
    if (gid == 0) printf("Sum = %d\n", *sum);

    // Exit.
    dsm_exit();
}

Why is it that each process can "see" the correct value at the address 0x7fe953fcb000 in the shared space, yet after adding, behaves as if the value at that address during the addition was still zero?

---

Here is what troubles me about this problem:

• If it was a caching issue, how is it that I can print the correct value before the arithmetic operation, and it is still incorrect?
• I am adding to a shared value on the process heap. The compiler could not have assumed that the value there would be zero, and optimized anything out.

Is there some explanation for why this might happen under the hood? I've attempted to use GDB with my program to see what is going on. But as far as I can tell, it is simply moving the value at memory addresses into the registers. I cannot see any optimization problems yet.

Answer 1

As far as I can see, four processes get spawned in quick succession, and each of them tries to do * sum += some_value; it is perfectly possible that they all see * sum as being zero before the addition.

Let's use an abstract assembler syntax. The C statement

*sum = *sum + local_sum

is compiled into

LOAD *sum into R0
LOAD local_sum into R1
ADD R1 to R0
STORE R0 to *sum

The four processes race to execute this sequence. It is perfectly possible that they all do LOAD *sum into R0 before any of them has had a chance to do STORE R0 to *sum; in fact, given that, as you say, there is a system call (and thus a replanification point) triggered by STORE R0 to *sum, there is a very good chance. You need to synchronize accesses to shared variables, using semaphores for instance.