Linux DAX (Direct Access) Details

Question

I was looking for DAX (Direct Access), and saw that it is introduced as a replacement of XIP (Execute In Place); however, I have doubts if it indeed makes applications executed without being copied to RAM. It says that it is "Direct Access for files", but an executable is also a file to kernel isn't it? So does it make the kernel execute files without copying them to RAM? If yes, how it works? Does it keep the .text region in place, but create a copy of .data region?

I have an experiment setup: I configured my Linux kernel 4.6.2, with DAX support. Created a ram backed block device. Mounted a ramdisk with dax option:

# mount -t ramfs -o dax,size=8m ext2 /ramdisk
# mount
rootfs on / type rootfs (rw,size=59124k,nr_inodes=14781)
proc on /proc type proc (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
ext2 on /ramdisk type ramfs (rw,relatime,dax,size=8m)
#

Now, I have ramfs mounted to /ramdisk, formatted with ext2 and has dax support. If I now copy an application to /ramdisk, and execute; how can I be sure that it is not copied to any other place on the RAM, and executed from there?

Unfortunately, dax has so little documentation. It's kernel explanation says that:

For block devices that are memory-like, the page cache pages would be unnecessary copies of the original storage. The DAX code removes the extra copy by performing reads and writes directly to the storage device. For file mappings, the storage device is mapped directly into userspace.

This seems as if it will give me the execution without copying the executable to a second place in RAM. However, it is also said that:

Even if the kernel or its modules are stored on a filesystem that supports DAX on a block device that supports DAX, they will still be copied into RAM.

In summary, I am confused with DAX feature, and curious if it can provide me a way to execute applications, without copying them to another place on RAM (Copying to cache is out of topic for me). I would be pleased to hear explanations of how it works.

Answer 1

Before discussing your example, a little disclaimer: this is a somewhat simplified version of the reality, There are a lot of corner cases and exceptions I do not explain, but it should be enough to make you understand what is going on...

---

Block Devices

What is confusing you is the indiscriminate use of "block device" term. Block devices are typically HDDs, CDs, SSDs, ... As the name itself says, you cannot read/write an individual byte to these devices, you need to write them in blocks (typically 512 bytes in size).

Block devices have a few registers and small memory regions mapped to the processor address space which can be used to read device status, send commands, etc. However, they (usually) do not provide a means to access the data they hold directly. This is normally done by sending commands to the device and waiting for a hardware interruption signaling that a DMA operation (to read or write) has been completed.

So you see, with these kind of devices it is somewhat difficult (if not impossible) not to use the main memory (DRAM), since their operation involves DMA operations and the like. What DAX does in these cases is to remove some of the overheads involved in accessing the data, but not much more than that.

DIMM-format* NVMs

However, recently some DIMM-format* non-volatile memories (NVMs) have been introduced to the market. These devices map their whole contents to the processor address space and can therefore be accessed directly by the processor via store and load instructions. The kernel doesn't even need to know these devices are being accessed: by all intents and purposes it is as if the process is accessing a regular DRAM-backed memory page.

*DIMM-format was just one example. There are also some other existing interfaces such as PCI doing just that...

The Confusing Part

Here comes the confusion... Up until recently "storage device" was practically a synonym to "block device". The Linux kernel recognizes these new NVMs as storage devices/block devices and treats them accordingly by creating an entry in /dev the same way it would with a SSD, for instance. (If you do not have one of these NVM backed devices, you could simulate that by specifying that a given memory range of your regular DRAM should be treated like NVM. See here for more info on how to do that.)

If you create a file system on a device like that, it will work just as if it was using a regular HDD. It will try to improve performance by caching contents to DRAM. What DAX-enabled file systems do is to avoid the creation of caches, which were meant to speed up access but that, in these cases would most likely worsen performance.

---

Even if the kernel or its modules are stored on a file system that supports DAX on a block device that supports DAX, they will still be copied into RAM.

I couldn't really find a solid reason for this behavior, however I'd guess it is done for security and performance reasons to guarantee that the kernel and the kernel modules won't be running from a slow (slower than DRAM) device and that their contents would not be messed with during the kernel execution.

There shouldn't be, however, any kind of problem to run your executable directly from the NVM using only NVM-backed memory as long it remains in user space.

Take a look at the projects Pmem.io from Intel and Atlas from HP. They are programming interfaces created specifically for this kind of thing.

---

Now about your example:

# mount -t ramfs -o dax,size=8m ext2 /ramdisk
# mount
rootfs on / type rootfs (rw,size=59124k,nr_inodes=14781)
proc on /proc type proc (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
ext2 on /ramdisk type ramfs (rw,relatime,dax,size=8m)
#

You are not creating a RAM-Backed EXT2 file system. You are creating a RAM-backed file system using ramfs with a dummy name ext2. It would make no difference if you mounted it like this:

# mount -t ramfs -o dax,size=8m winter_is_coming /ramdisk

Answer 2

Does it keep the .text region in place, but create a copy of .data region?

In any case, exec() works the same as mmap() with MAP_PRIVATE. The pages are mapped into the processes' virtual address space as read-only. Therefore writes cause a page fault interrupt. The way these page faults are handled is described as Copy On Write.

In the case of DAX, the virtual pages start out as mapped to physical pages on the device. But write page faults on MAP_PRIVATE will copy the page data to a new page in RAM. (Then the processes' mapping is updated accordingly, and the interrupted program instruction is restarted).

Note that DAX is a generalization of XIP to allow for writes as well as reads i.e. MAP_SHARED as well as MAP_PRIVATE. MAP_SHARED can be used for database files, for example.

---

In fact, .text may also be written in the case of shared libraries. libraries which are not Position Independent Executables and contain references to themself, need these references to be updated according to what address the relevant library has been loaded at. This process is referred to as "relocation". Libraries also reference other libraries e.g. libc; updating these references is referred to as "symbol resolution".

Even if the kernel or its modules are stored on a filesystem that supports DAX on a block device that supports DAX, they will still be copied into RAM.

Kernel modules are special. They require symbol resolution too. But, the kernel does not use COW. (More generally, it does not use demand-paging for it's code and data segments). Page faults inside the kernel are fatal, because handling them could cause infinite recursion! Therefore pre-DAX, it was clear that kernel modules needed to be copied into RAM in their entirety. The kernel code and data segments are small; when DAX was implemented there would have been no benefit to mucking around with this on servers with byte-addressable storage.

The kernel itself is historically compressed, obviously it is decompressed into RAM.

That said, XIP is supported for uncompressed kernels. This would generally be used on an "embedded" system, i.e. quite limited hardware. At that point, it's probably not a problem to make most of the required code built-in, as opposed to using loadable modules.