Increasing performance of clear text file reads

Question

In our bioinformatics application we have large files (up to 900MB, typically 100MB). These files contain a clear text representation of a genome, and are essentially a 1 line file with a sequence of characters.

Data is referenced by location, for example chromosome 7 starts at position 1 and ends at position 158937463. We typically extract a small section of around 400 characters, eg: from position 4,120,000 4,120,400.

We have a utility written in Ruby for this purpose: https://github.com/sfcarroll/bio-fasta-read and it works by reading through the file from the beginning.

We do these reads many times and they are slowing our application. I was wondering what options are available for caching? The source data will never change, but the ranges will frequently change. We are running Ubuntu Server 14 x64 on a machine with 128GB of RAM.

Is there some OS-level way of increasing performance? Perhaps loading the entire file into memory or somehow caching the requests?

EDIT

I should add if there are options such as somehow assigning more memory to the file cache that would help, pointers would be appreciated. We can consider a dedicated server for these file reads if it would need to be tuned in a specific way.

EDIT 2 We are running a Xeon E5-1650 Hexa-Core CPU and dual SSD's (that could be raided), along with 128GB RAM.

Answer 1

The linux kernel does cache management automatically. Everthing that is loaded into RAM, stays there until another process needs RAM and there is no more available. So in the linux kernel RAM should always be full. You system has 128GB of RAM, thats more than enough for a 100-1000MB file.

To load a huge file into RAM just cat it:

cat huge_file > /dev/null 2>&1

All output is sent to /dev/null, but for that it must pass the systems RAM. While doing this you can watch how Cached in /proc/meminfo increases.

When cat ends, execute the Ruby application. The Ruby application now reads the cached version of the huge file.

Answer 2

Use dd to read a section of a file without reading everything preceding it.  For your example (reading bytes 4,120,000-4,120,400) you could use

dd  bs=400  skip=10300  count=1  if=your_input_file  of=your_output_file

This defines a logical block size of 400 bytes, and then tells dd to skip the first 10300 “logical blocks” of the input file (if).  10300 is 4,120,000 ÷ 400.  Then it reads one block (count=1) of 400 bytes and writes it to the output file (of).  If you omit the of specification, dd will write to the standard output, so you can pipe it to something.

If the starting point (offset) is not guaranteed to be an integer multiple of the block size (or even if it is), you can do trickier stuff, like

(dd bs=10000 skip=412 count=0;  dd bs=400 count=1 of=your_output_file) < your_input_file

or

(dd bs=4120000 skip=1 count=0;  dd bs=400 count=1 of=your_output_file) < your_input_file

where

• Again, you can omit the of specification, and this will write to the standard output.
• If you run dd without an if specification, it will read from the standard input.  The standard input for the entire (dd …; dd …) command group comes from the < your_input_file at the end.
• The first dd command doesn’t read or write any data, because of the count=0; it only seeks.
• Because the two dd commands get their standard input from the same I/O redirection, the seek done by the first one will affect the file pointer that the second one sees.