3

I'm attempting to time file I/O on my disks. In order to make sure all the I/O is happening off the hard drives and not from ram, before I time any read, I run a drop cache command. Specifically, I call fprintf in a C program to write 3 to /proc/sys/vm/drop_caches. Here's the source:

FILE *f = fopen("/proc/sys/vm/drop_caches", "w");

  if (!f)
  {
    perror("Opening of /proc/sys/vm/drop_caches failed");
    return 1;
  }

fprintf(f, "3");

fclose(f);

The code actually runs on many machines in a large cluster, and most of them have been having a strange problem. To run the code above will sometimes take hours. The workflow here is Read → Drop Cache → Repeat. My reads only take about 5 minutes, so there shouldn't be much cached again in that small amount of time.

The machines are nearly identical in software and hardware, but there is only one machine out of the approximately 20 which never seems to have trouble dropping its cache.

Is there any reason that this should be taking so long? In some cases, I think the program is just freezing all together. Any tips on how to troubleshoot this?

Edit: ==========================================================

I did a bit of troubleshooting on this and I just wanted to catalog what I was able to find in case anyone else has this problem in the future. The long and short of it is, we think it had something to do with Hadoop and HDFS.

1) I was able to run my C program by hand on the machine while another instance of the command was hung up and in some cases it would return before the hung program. That is to say that the actual cache dropping probably wasn't taking that long because the other instance of the program was able to drop the cache and return

2) The reason one of the machines was having no trouble is that hadoop or perhaps some other program was crashing on that node causing it not to be used by for hadoop. This is why we think it has something to do with HDFS.

  • Can you umount filesystem(s)? Doing so will also drop the cache for the file system being unmounted. You'll at least narrow down exactly where the problem is. – Andrew Henle Aug 3 '15 at 10:50
  • Thanks for the suggestion Andrew. I don't think it's the cache dropping per se that's the problem, I think it actually has to do with Hadoop, specifically the HDFS Fuse mount. I did a bit more trouble-shooting, which I'll catalog in an edit above. – Bobak Hashemi Aug 11 '15 at 17:24
  • Can you put your answer in an answer (you are allowed to answer your own questions). – ctrl-alt-delor Jan 7 at 9:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.