4

My company developed an application that picks up (watches for) xml files of <10kb in size from a directory, reads it in sends the body as an api call to an external service and then moves the file into a processed directory.

Due to the volume of files - roughly 2000/min we were getting dreadful performance out of NTFS. We were no where near able to keep up with the processing.

I'm a Linux guy through and through and from experience Linux would handle this situation a lot better especially with things like inotify which are leaps and bounds ahead of the ntfs api, that's why I've ported the code to .NET Core to give it a shot.

At home I use XFS on my Workstations and ZFS on my servers, so aside from ext4 - I have no real experience with any other filesystem.

So my question is - which filesystem (preferably in-tree) would be the most performant for this kind of workload.

  • 2
    Just try ext4. Don't over optimise early. – ctrl-alt-delor Nov 4 '18 at 20:17
  • 1
    What about the hardware (the drive, where your file system will reside)? – sudodus Nov 4 '18 at 20:24
  • Currently it's on hyper-v with shared vhdx, it's going to move over to ESXi - I don't know the exact specs of the hypervisor. – dcrdev Nov 4 '18 at 20:33
  • If the data is small in relation to the amount of memory then I would look at tmpfs; basically a RAMdisk. It's non-persistent, but is great for temporary files. So code your primary loop to work from tmpfs, and then have a batch (once an hour?) move processed files to persistent storage (if it's needed). – Stephen Harris Nov 4 '18 at 21:49
  • 1
    @StephenHarris what you are describing is caching. You will probably not be better at it then the kernel is: once you have read the data, it will all be in ram (cached). It will only fall out of cache if the data set is large. – ctrl-alt-delor Nov 5 '18 at 10:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.