Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Noticing a strange performance behaviour while using shared memory that are file backed (i.e open user defined file and mmap() the same into process space). While doing a memcpy() into the shared memory section, sometimes a delay is observed in the order of milliseconds. To be specific, in normal scenarios 2048 bytes are copied in 0.4us but sometimes the same number of bytes takes about 10 - 20 milliseconds to copy. This happens randomly any time. The next 2048 frame of data goes back to the normal time.

Kernel version is 2.6.27.23-0.1-preempt.

Appreciate any clues from anyone, about what is happening and why the delay is seen. I have even tried by using shm_open() and then mmap(), with no difference

The mmap calls look like this:

int fd = ::open("somefile", flags, 0666);
if(fd != -1) {
  myBasePointer = ::mmap(0, sizeInBytes,
                         PROT_READ|PROT_WRITE, MAP_SHARED,
                         fd, offsetInBytes);
} 

One process creates the file and mmaps, the other processes opens the file and mmaps itinto their address space. The file is a normal file on an ordinary filesystem.

share|improve this question
    
The file is a real file, following is the code snippet int fd = ::open("somefile", flags, 0666); if(fd != -1) { myBasePointer = ::mmap(0, sizeInBytes, PROT_READ|PROT_WRITE, MAP_SHARED, fd, offsetInBytes); } One process creates the file and mmap's, the other processes opens the file and mmap's into their address space. –  rakeshS Oct 4 '11 at 3:34
    
maybe you're catching the lag at times when the mmap()ed file is flushing to disk. you could try using msync() to for syncs at specific times, and see if you find latency outside of those events. also, depending how big the file is, how much ram you have, you may be paging<->disk periodically. –  Tim Kennedy Oct 4 '11 at 13:53
    
Hmm the RAM is around 16 GB and the file is around 72 MB, Checked with vmstat (see o/p below) ` procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 1811988 153288 12362956 0 0 1 41 4 58 14 0 86 0 ` –  rakeshS Oct 5 '11 at 12:42
add comment

1 Answer 1

Idea 1: Try adding the MAP_LOCKED in your mmap call, to ensure that the pages remain backed by real RAM at all times. Even though you have a lot of RAM, it's still possible that the part you need is getting paged out for some reason. Since you seem to be latency-sensitive, specifying this flag is advisable regardless of the amount of RAM free on your current hardware.

Idea 2: Also specify MAP_POPULATE to ensure that page-table entries are created at mmap time rather than at fault time. Although AFAIK, this would only cause latency at first page fault, not at subsequent ones. But again a good defensive practice.

Idea 3: Does using an anonymous shared mapping (as opposed to backed by a real file), thus eliminating the filesystem, cause the latency to disappear? If this helps, you could always write the anonymous shared region's contents to a file (perhaps at some lower frequency) from a helper thread.

Idea 4: Simply try with a newer kernel. The Linux kernel has changed greatly since you asked your question.

Finally, how are you measuring the memcpy latency? Are you confident that you're not:

  • Including some large delay due to fetching the current time? Eg syscall latency, if you're calling a syscall; nor
  • Including clock jumps caused by NTP corrections; nor
  • Including locking latency (a memcpy of 2k is not atomic); nor
  • Using a method of time measurement whose precision exceeds its accuracy?
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.