I am trying to figure out how and when Linux gives the permission to perform direct memory access (DMA) to the peripheral devices on PCIe. I have read how DMA is initiated in the kernel's "DMA API HOWTO", but still would like to clarify a couple points.

This question is related to the question How to harden linux against DMA attacks? Maybe the comment about IOMMU is the answer. The HOWTO mentions the IOMMU in passing, saying that

I/O devices use a third kind of address: a "bus address". If a device has registers at an MMIO address, or if it performs DMA to read or write system memory, the addresses used by the device are bus addresses. In some systems, bus addresses are identical to CPU physical addresses, but in general they are not. IOMMUs and host bridges can produce arbitrary mappings between physical and bus addresses.

But since the answers to the question about DMA attacks suggest to mitigate the attacks without referring to IOMMU, I wonder how common are systems with IOMMU? Is it the usual architecture for servers, but not so for PCs?

If the access is defined by the IOMMU mappings, then when are they set up?

In "What memory is DMA'able?" of the HOWTO they write

The first piece of information you must know is what kernel memory can be used with the DMA mapping facilities. There has been an unwritten set of rules regarding this, and this text is an attempt to finally write them down.

If you acquired your memory via the page allocator (i.e. __get_free_page*()) or the generic memory allocators (i.e. kmalloc() or kmem_cache_alloc()) then you may DMA to/from that memory using the addresses returned from those routines.

But they do not mention giving some permissions to the devices and actually enabling the DMA. This makes me wonder if a device can write to this memory immediately without any permission given to it. If so, it should mean that a device can write to any other memory too, since the allocations are software business in the OS.

The question is when a chunk of main memory is enabled to be written to from the PCIe bus?

But next goes something that could be the answer:

For correct operation, you must set the DMA mask to inform the kernel about your devices DMA addressing capabilities.

This is performed via a call to dma_set_mask_and_coherent()::

int dma_set_mask_and_coherent(struct device *dev, u64 mask);

which will set the mask for both streaming and coherent APIs together.

These calls usually return zero to indicated your device can perform DMA properly on the machine given the address mask you provided, but they might return an error if the mask is too small to be supportable on the given system. If it returns non-zero, your device cannot perform DMA properly on this platform, and attempting to do so will result in undefined behavior. You must not use DMA on this device unless the dma_set_mask family of functions has returned success.

Aha! So, does the DMA mask define the access to the memory? Is the dma_set_mask family of functions the API that enables the DMA to the main memory, whatever the way it is allocated? Basically, what do they mean by "For correct operation, you must set the DMA mask to inform the kernel about your devices DMA addressing capabilities", what are you informing the kernel about with the DMA mask? Does it define the access to the physical memory from the PCIe bus, with some software or hardware technology (like IOMMU), or it just informs the kernel that a chunk of memory can be used by external devices, therefore the kernel must not use it for its own purposes, like for the virtual memory of the system. The last case seems more likely, since you only "inform .. about your devices DMA addressing capabilities". Then the system is still open to random DMA access from the external devices.

Also, do I understand right that IOMMU works just like MMU: it sets a memory mapping for each device? Therefore, the IOMMU will just block any device that tries to access the memory outside the allowed mapping. Does it send some interrupt to the CPU, like segfault? In this case there is no "undefined behavior", the access is just blocked and that is all.

Probably, the question is also related to Can't access RAM from PCIe device, where the RAM is accessible from an FPGA on PCIe when the CPU runs Windows or under Linux on another PC, but there is no access on the main PC under Linux. Hence, Windows somehow allows the access to RAM, but Linux does not. What must Linux do to allow the DMA?


You must log in to answer this question.

Browse other questions tagged .