I have a script which creates a lot of files and directories. The script does black box tests for a program which works with a lot of files and directories. The test count grew and the tests were taking too long (over 2 seconds). I thought I run the tests in a ram disk.

I ran the test in /dev/shm. Strangely it did not run any faster. Average run time was about the same as on normal harddisk. I also tried in a fuse based ram disk written in perl. The website is gone but I found it in the internet archive. Average run time on the fuse ram disk is even slower. Perhaps because of the suboptimal implementation of the perl code.

Here is a simplified version of my script:

#! /bin/sh

preparedir() {
  mkdir foo
  mkdir bar
  touch bar/file
  mkdir bar/baz
  echo qux > bar/baz/file

systemundertest() {
  # here is the black box program that i am testing
  # i do not know what it does exactly
  # but it must be reading the files
  # since it behaves differently based on them
  find $1 -type f -execdir cat '{}' \; > /dev/null

singletest() {
  mkdir actual
  (cd actual; preparedir)
  systemundertest actual
  mkdir expected
  (cd expected; preparedir)
  diff -qr actual expected

manytests() {
  while read dirname; do
    rm -rf $dirname
    mkdir $dirname
    (cd $dirname; singletest)

seq 100 | manytests

The real script does a bit more error checking and result collecting and a summary. The find is a dummy for the actual program I am testing.

I wonder why my filesystem intensive script does not run faster on a memory backed filesystem. Is it because the linux kernel handles the filesystem cache so efficiently that it practically is a memory backed filesystem?

  • One reason why they might be slower on FUSE: Reads/Writes hit the filesystem cache in memory anyways (same in any modern OS), and pdflush comes around so often to actually send the I/O through the scheduler. – Bratchley Apr 22 '13 at 16:36
  • Also, why do you have to use a FUSE filesystem for this? Does your kernel not support tmpfs (like in Linux and FreeBSD)? – Bratchley Apr 22 '13 at 16:39
  • 2
    Probably because you're spawning lots and lots of processes. That should be considerably slower than, say, a Python or Go program that just does things directly. – frostschutz Apr 22 '13 at 17:33
  • For example, just by combining all your mkdir into a single call, it takes 0.6s instead of 1s for me on /dev/shm. – frostschutz Apr 22 '13 at 17:42
  • Probably because many of those actions in your script are small transactions and will generally be cacheable by the OS's fs buffer. – Wing Tang Wong Apr 22 '13 at 18:39

Quite generally speaking, all operations happen in RAM first - file systems are cached. There are exceptions to this rule, but these rather special cases usually arise from quite specific requirements. Hence until you start hitting the cache flushing, you won't be able to tell the difference.

Another thing is, that the performance depends a lot on the exact file system - some are targeting easier access to huge amounts of small files, some are efficient on real-time data transfers to and from big files (multimedia capturing/streaming), some emphasise data coherency and others can be designed to have small memory/code footprint.

Back to your use case: in just one loop pass you spawn about 20 new processes, most of which just create one directory/file (note that () creates a sub-shell and find spawns cat for every single match) - the bottleneck indeed isn't the file system (and if your system uses ASLR and you don't have a good fast source of entropy your system's randomness pool gets depleted quite fast too). The same goes for FUSE written in Perl - it's not the right tool for the job.

| improve this answer | |

A somewhat longer response than my comment about the tests being comprised mainly of small transactions.

Workload insufficient to test

If you want to stress test your filesystem you will need larger sets of work.

Depending on how much memory you have on your box, even 10's of thousands of folder creation operations won't show a noticeable difference between the two. So, modify your workload to sufficiently test the filesystems, taking into account your memory, which will get used as a buffer.

There are a variety of ways to devise a test that negates the benefits of your system ram and other factors that will skew your test results.

Or, you can used a standardized test suite, like bonnie++

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.