2

In our bioinformatics application we have large files (up to 900MB, typically 100MB). These files contain a clear text representation of a genome, and are essentially a 1 line file with a sequence of characters.

Data is referenced by location, for example chromosome 7 starts at position 1 and ends at position 158937463. We typically extract a small section of around 400 characters, eg: from position 4,120,000 4,120,400.

We have a utility written in Ruby for this purpose: https://github.com/sfcarroll/bio-fasta-read and it works by reading through the file from the beginning.

We do these reads many times and they are slowing our application. I was wondering what options are available for caching? The source data will never change, but the ranges will frequently change. We are running Ubuntu Server 14 x64 on a machine with 128GB of RAM.

Is there some OS-level way of increasing performance? Perhaps loading the entire file into memory or somehow caching the requests?

EDIT

I should add if there are options such as somehow assigning more memory to the file cache that would help, pointers would be appreciated. We can consider a dedicated server for these file reads if it would need to be tuned in a specific way.

EDIT 2 We are running a Xeon E5-1650 Hexa-Core CPU and dual SSD's (that could be raided), along with 128GB RAM.

1
  • 3
    You'd probably be better off heading over to code review. However if you're saying your application reads through the entire file up until the specified offset, then that is definitely wasteful. You'd be better off doing seek operations (man 2 lseek for the C documentation).
    – phemmer
    Oct 17, 2014 at 13:05

2 Answers 2

2

The linux kernel does cache management automatically. Everthing that is loaded into RAM, stays there until another process needs RAM and there is no more available. So in the linux kernel RAM should always be full. You system has 128GB of RAM, thats more than enough for a 100-1000MB file.

To load a huge file into RAM just cat it:

cat huge_file > /dev/null 2>&1

All output is sent to /dev/null, but for that it must pass the systems RAM. While doing this you can watch how Cached in /proc/meminfo increases.

When cat ends, execute the Ruby application. The Ruby application now reads the cached version of the huge file.

10
  • We have some files which are more frequently read ... if we do this would the file be cycled out of cache, or is there some way to pin it?
    – port5432
    Oct 17, 2014 at 13:00
  • @ardochhigh It gets out of cache only when RAM is full. But there is also an option to mount a part of your RAM and store the file there: mount -t ramfs ramfs /mountpoint
    – chaos
    Oct 17, 2014 at 13:46
  • 1
    @ardochhigh Yes, but /mountpoint has to exist.
    – chaos
    Oct 17, 2014 at 14:07
  • 1
    @ardochhigh And don't forget that after a reboot the file in this /mountpoint are away. So don't move important things there with bo backup
    – chaos
    Oct 17, 2014 at 14:09
  • 1
    @ardochhigh ur welcome. And yes that is faster, because cating the file gives no guarantee that it stays in the RAM or get swapped.
    – chaos
    Oct 17, 2014 at 14:36
1

Use dd to read a section of a file without reading everything preceding it.  For your example (reading bytes 4,120,000-4,120,400) you could use

dd  bs=400  skip=10300  count=1  if=your_input_file  of=your_output_file

This defines a logical block size of 400 bytes, and then tells dd to skip the first 10300 “logical blocks” of the input file (if).  10300 is 4,120,000 ÷ 400.  Then it reads one block (count=1) of 400 bytes and writes it to the output file (of).  If you omit the of specification, dd will write to the standard output, so you can pipe it to something.

If the starting point (offset) is not guaranteed to be an integer multiple of the block size (or even if it is), you can do trickier stuff, like

(dd bs=10000 skip=412 count=0;  dd bs=400 count=1 of=your_output_file) < your_input_file

or

(dd bs=4120000 skip=1 count=0;  dd bs=400 count=1 of=your_output_file) < your_input_file

where

  • Again, you can omit the of specification, and this will write to the standard output.
  • If you run dd without an if specification, it will read from the standard input.  The standard input for the entire (dd …; dd …) command group comes from the < your_input_file at the end.
  • The first dd command doesn’t read or write any data, because of the count=0; it only seeks.
  • Because the two dd commands get their standard input from the same I/O redirection, the seek done by the first one will affect the file pointer that the second one sees.
2
  • This is the right approach, but why use dd? Using an external utility will slow things down. This should be done directly in the Ruby program. Oct 17, 2014 at 23:20
  • It wasn't clear to me whether the Ruby program is an all-in-one solution, or whether it just extracts the 400-byte section and hands it off to another program that does whatever analysis on it. In other words, I thought the above dd solution might replace the Ruby program. Oct 17, 2014 at 23:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .