0

Background

I'm trying to download about 150GB to a newly-created Linux box (AWS EC2) with 100gbps network connection at full speed (12.5GB/s) or close to that. The network end is working well. However, I'm struggling to find anywhere on the box that I can put all the data fast enough to keep up, even though the box has 192GB of RAM.

My most successful attempt so far is to use the brd kernel module to allocate a RAM block device large enough, and write into that in parallel. This works at the required speed (using direct io) when the block device has already been fully written to, for example using dd if=/dev/zero ...

Unfortunately, when the brd device is newly created, it will only accept a write rate of around 2GB/s.

My guess is that this is because brd hooks into 'normal' kernel-managed memory, and therefore when each new block is used for the first time, the kernel has to actually allocate it, which it does no faster than 2GB/s.

Everything I've tried so far has the same problem. Seemingly, tmpfs, ramfs, brd, and everything else that provides RAM storage hooks into the normal kernel memory allocation system.

Question

Is there any way in Linux to create a block device out of real memory, without going through the normal kernel's memory management?

I'm thinking that perhaps there is a kernel module out there that will split off an amount of memory at boot time, to be treated like a disk. This memory would not be considered normal memory to the kernel and so there would be no issue with it wanting to use it for anything else.

Alternatively, is there some way to get the kernel to fully initialise a brd ramdisk (or similar) quickly? I tried writing to the last block of the disk alone, but unsurprisingly that didn't help.

Non-RAM alternative

In theory, a RAID of NVMe SSDs could achieve the required write speed, although it seems likely there would be some kind of bottleneck preventing such high overall I/O. My attempts to use mdadm RAID 0 with 8 NVMe SSDs have been unsuccessful, partly I think because of difficulties around block sizes. To use direct io and bypass the kernel's caching (which seems necessary), the only block size that can be used is 4096, and this is apparently far too small to make efficient use of the SSDs themselves. Any alternative here would be appreciated.

Comments

I know 2GB/s sounds like a lot, and it only takes a couple of minutes to download the lot, but I need to go from no EC2 instance at all to an EC2 instance with 150GB loaded in less than a minute. In theory it should be completely possible: the network stack and the physical RAM are perfectly capable of transferring data that fast.

Thanks!

0
1

On a tmpfs filesystem I can copy 64 files of 1.6 GB (in total 100GB) in 7.8 sec by running 64 jobs in parallel. That is pretty close to your 100 Gbit/s.

So if you run this in parallel (meta code):

curl byte 1G..2G | write_to file.out position 1G..2G 

ẁrite_to could be implemented with mmap.

Maybe you can simply write to different files, use loop-devices, and use RAID in linear mode: https://raid.wiki.kernel.org/index.php/RAID_setup#Linear_mode

If you control both ends, then set up the source as 150 1 GB files used as loop-devices and RAID in linear mode. Then you should copy those in parallel and set up the RAID linear again.

2
  • Thanks - I'll try that today
    – gandaliter
    Feb 18 at 8:56
  • 1
    This is working, although it has to be multiple separate files in a tmpfs. Multiple threads writing to one file is limited around the same level as I described (2GB/s). I just need to find a way to map those files together into a single block device, but I think some combination of loopback and dm should do it.
    – gandaliter
    Feb 18 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.