In our bioinformatics application we have large files (up to 900MB, typically 100MB). These files contain a clear text representation of a genome, and are essentially a 1 line file with a sequence of characters.
Data is referenced by location, for example chromosome 7 starts at position 1 and ends at position 158937463. We typically extract a small section of around 400 characters, eg: from position 4,120,000 4,120,400.
We have a utility written in Ruby for this purpose: https://github.com/sfcarroll/bio-fasta-read and it works by reading through the file from the beginning.
We do these reads many times and they are slowing our application. I was wondering what options are available for caching? The source data will never change, but the ranges will frequently change. We are running Ubuntu Server 14 x64 on a machine with 128GB of RAM.
Is there some OS-level way of increasing performance? Perhaps loading the entire file into memory or somehow caching the requests?
EDIT
I should add if there are options such as somehow assigning more memory to the file cache that would help, pointers would be appreciated. We can consider a dedicated server for these file reads if it would need to be tuned in a specific way.
EDIT 2 We are running a Xeon E5-1650 Hexa-Core CPU and dual SSD's (that could be raided), along with 128GB RAM.
man 2 lseek
for the C documentation).