I am running a benchmark of HDD read that bypasses page cache. I have set O_DIRECT flag and memaligned my memory. This function attempts a random read within a file(lseek64() used). The data I am getting looks fine until certain point (32 MB). Please see the data below (averages): In particular I would like to know why do I have a such a large jump after 32 MB? I use Ubuntu 16.04 File system ext4.

I would really appreciate some help on this. Thank you.

KB      TIME
32      11.2452
64      22.3882
128     45.3915
256     89.6025
512     12.655
1024    402.332
2048    759.456
4096    1512.83
8192    2999.54
16384   5988.16
32768   **85358.8**

double readFileRan(std::string name, unsigned long bytes) {
   Time t;

   int ID = open(name.c_str(), O_RDONLY | O_DIRECT);


   if ( ID == -1) {
       std::cout << "can't open input file!" << std::endl;
       return 0;

   unsigned long reads = bytes / 512;
   std::vector<unsigned long> offsets;
   for(unsigned long i = 0; i < reads; i++) {
      offsets.push_back((rand() % reads) * 512);

   int BLKSIZE = 512;
   char* sector = (char*)memalign(BLKSIZE, BLKSIZE); //size of physical   sector
   unsigned long numRead = 0;
   unsigned long i = 0;
   off64_t result = 10;

   unsigned long long start = t.start();
   while(i <= reads)  {
      result = lseek64(ID, offsets[i] ,SEEK_SET);
      numRead = read(ID, sector, 512);
      i = i + 1;
   unsigned long long end = t.end();

   unsigned long long total = end - start;
   double mili = t.convertCyclesToSec(total);

   std::cout << mili << std::endl;
   return mili;

The time to read a sector depends on the rotation angle of the drive when you try the read, and your sample size is too small to avoid statistical fluctuations from this random process. You're reading every sector just once on average. That's fine when bytes is large and you are taking a lot of samples, but not so great when bytes is small. To get more interesting data, you should always read a fixed large number of sectors independently of the magnitude of bytes.

At some point there can be expected to be a jump in the access time when bytes exceeds the size of a cylinder, and the head has to move from track to track rather than just waiting for the correct sector to fly by (which also takes time, but less time). But this effect can be seen better when reading a partition raw rather than through a filesystem (which is free to map the file sectors non-linearly to the device sectors).

The cylinder sizes of modern disks is, of course, variable, as more sectors can fit on the longer outer tracks than on the shorter inner tracks closer to the spindle.

Trying to measure all of this is further complicated by the fact that you probably have an small memory cache on the disk itself, which is not disabled solely by using O_DIRECT.

  • You are absolutely right. I looked up the cache size and it is 32 MB. Thank you. Accepted your answer. – Illia Dec 2 '16 at 2:11
  • I have another quick question: I am reading this time sequentially a bunch of files. Everything goes good until I hit 2 MB. The numbers are the following: 1 MB = 300 ms, 2 MB = 200 ms and then from 2 MB byte everything looks consistent. I ran it many times and this pattern continues. Any suggestions? – Illia Dec 2 '16 at 2:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.