2

Context: I am using POSIX shared memory to provide a set of processes with a shared memory space. I have used this scheme for some time now in order to share data, and it's working okay. However, I recently ran into an odd problem with a certain class of programs.

Problem: I wrote a program in which each process has to contribute a value to a shared sum in the shared memory space. The sum was initialized to zero when the shared object was mapped into memory earlier. However, when each process tries to add it's part to the shared sum, it can see the latest value, but the result of the addition is always as if it added it's own value with zero. See below:

[21017] Adding 6 to 0!
[21020] Adding 33 to 0!
[21016] Adding 15 to 0!
[21018] Adding 24 to 0!
[21017] Got access! (0x7fe953fcb000 = 0)
[21017] Done (0x7fe953fcb000 = 6)
[21016] Got access! (0x7fe953fcb000 = 6)
[21016] Done (0x7fe953fcb000 = 15)
[21018] Got access! (0x7fe953fcb000 = 15)
[21018] Done (0x7fe953fcb000 = 24)
[21020] Got access! (0x7fe953fcb000 = 24)
[21020] Done (0x7fe953fcb000 = 33)
Sum = 33

Each process "sees" the latest value written, but somehow, after adding it's own component, seems to have ignored the existing value. You can see that each access is ordered sequentially, as there is an access control system managing who gets to write to the shared memory space. The test program used looks as follows (though I do not expect the reader to run it):

int main (void) {
    int local_sum = 0, gid = -1;
    volatile int *sum;

    // Fork for four processes.
    for (int i = 1; i < 4; i++) {
        if (fork() == 0) break;
    }

    // Initialize the DSM. Set GID.
    sum = (int *)dsm_init(&cfg);
    gid = dsm_get_gid();

    // Compute range.
    for (int i = 0; i < 3; i++) {
        local_sum += array[(gid * 3) + i];
    }

    // Add to local sum.
    printf("[%d] Adding %d to %d!\n", getpid(), local_sum, *sum);
    *sum = *sum + local_sum;

    // Barrier.
    dsm_barrier();

    // Print sum if process zero.
    if (gid == 0) printf("Sum = %d\n", *sum);

    // Exit.
    dsm_exit();
}

Why is it that each process can "see" the correct value at the address 0x7fe953fcb000 in the shared space, yet after adding, behaves as if the value at that address during the addition was still zero?


Here is what troubles me about this problem:

  • If it was a caching issue, how is it that I can print the correct value before the arithmetic operation, and it is still incorrect?
  • I am adding to a shared value on the process heap. The compiler could not have assumed that the value there would be zero, and optimized anything out.

Is there some explanation for why this might happen under the hood? I've attempted to use GDB with my program to see what is going on. But as far as I can tell, it is simply moving the value at memory addresses into the registers. I cannot see any optimization problems yet.

7
  • 1
    Where are the "got access" and "done" messages printed? They are not in the posted code. And who is the mysterious cfg?
    – AlexP
    Jul 12, 2018 at 18:37
  • 1
    @AlexP I added those lines in a small library. They are positioned before and after the modifications to the address is made. This is the file with the modifications. The print statements are on lines 187 and 236. However, I wanted to avoid posting this because it is hard to understand what is going on when lots of library files are involved.
    – Micrified
    Jul 12, 2018 at 18:43
  • 1
    The thing is, I don't understand how this program is synchronizing accesses to the shared variable. As far as I can see, four processes get spawned in quick succession, and each of them tries to do * sum += some_value; in the absence of any kind of synchronization it is perfectly possible that they all see * sum as being zero before the addition. At first sight this program looks very much like the "don't do it like this" example in concurrent programming 101.
    – AlexP
    Jul 12, 2018 at 18:48
  • 1
    @AlexP The shared map is write-protected. Attempting to access triggers SIGSEGV. Within the handler, the process waits for a go-ahead message from a C&C process. If unblocks when it receives a go-ahead, and then writes. The print statements are positioned right after getting the go-ahead, and then after finishing the write operation. I flush stdout after each print statement to best preserve order. I've checked the C&C logs and this to determine whether it's truely controlled access. As far as I can tell, it is. I could be wrong though, but I don't know how to verify what is happening better.
    – Micrified
    Jul 12, 2018 at 18:52
  • 1
    Let's use an abstract assembler syntax. The C statement *sum = *sum + local_sum is compiled into LOAD *sum into R0, LOAD local_sum into R1, ADD R1 to R0, STORE R0 to *sum. The four processes race to execute this sequence. It is perfectly possible that they all do LOAD *sum into R0 before any of them has had a chance to do STORE R0 to *sum; in fact, given that, as you say, there is a system call (and thus a replanification point) triggered by STORE R0 to *sum, there is a very good chance. You need to synchronize accesses to shared variables, using semaphores for instance.
    – AlexP
    Jul 12, 2018 at 19:07

1 Answer 1

2

As far as I can see, four processes get spawned in quick succession, and each of them tries to do * sum += some_value; it is perfectly possible that they all see * sum as being zero before the addition.

Let's use an abstract assembler syntax. The C statement

*sum = *sum + local_sum

is compiled into

LOAD *sum into R0
LOAD local_sum into R1
ADD R1 to R0
STORE R0 to *sum

The four processes race to execute this sequence. It is perfectly possible that they all do LOAD *sum into R0 before any of them has had a chance to do STORE R0 to *sum; in fact, given that, as you say, there is a system call (and thus a replanification point) triggered by STORE R0 to *sum, there is a very good chance. You need to synchronize accesses to shared variables, using semaphores for instance.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .