I am running 64-bit Xubuntu 14.04, linux version 3.8.0-25 with 8GB of RAM.
I have a script (in MATLAB, for what it's worth) that loads a large number of data files (~23k) one at a time, for a total of around 45G of data. The trouble I'm having is that after each file is loaded, it remains in the file cache. Linux seems to prefer to keep these files cached in memory rather than any of the other memory contents, resulting in almost everything else being forced into swap, causing my system to slow to a crawl. I read several files a second, so this happens fairly quickly. I only read each file once, so I don't need the files to remain cached after I've finished with them.
I've tried turning the swap off, which works to an extent, but it seems like a poor solution (and it has already failed once when another program started using excessive memory). Is there a way that I can limit the amount of RAM linux uses for file caching?