I am benchmarking the write performance in the case that Linux writes the data to the page cache without throttling the process or syncing the data to the disk. The simple experiment I am doing looks like the following:
long size = 1024;
int fd = open ("file", O_CREAT | O_RDWR | O_TRUNC, ...);
char *buffer = (char *) malloc (size)
**start_time = time.now ();**
write (fd, buffer, size);
**end_time = time.now ();**
close (fd);
printf ("write duration %d\n", end_time - start_time);
Here I expect to effectively observe the memory bandwidth since the data is only copied to the page cache of the OS, and nothing is synced to the disk. Also, the OS has not started with the background flushes or throttling the process, since, the dirty rate is way lower than the background_dirty_ratio. Though, when I compare it to the memory bandwidth (memcpy cost of the same size), it is way more expensive than a memcpy:
char *buffer1 = (char *) malloc (size)
char *buffer2 = (char *) malloc (size)
**start_time = time.now ();**
memcpy (buffer1, buffer2, size);
**end_time = time.now ();**
printf ("memcpy duration %d\n", end_time - start_time);
For instance on my system (Linux kernel version 4.2, CentOS), I see a memcpy bandwidth of almost 60GB/s, and the write bandwidth is almost 2GB/s. To my understanding, when write system call is invoked, it merely copies the data in the page cache (in memory) and returns as soon as the copy is done. So I expect to see a bandwidth near to memory bandwidth. I also tested the same experiment with larger data (before the OS starts to throttle the process) to mitigate the cost of making the system call. But still, I see almost the same results. Does anyone know why I am not observing the memory bandwidth when performing write on the page cache?