loop-device performance issue

Question

I am trying to work with dm-mapper and it doesnt accept files. so i need a device.

So i have created a 1500M file on ram(ramfs filesystem) at /dev/shm/f and mapped it to /dev/loop0(by losetup).

But i have about a fourth performance on loop-device compared to the file.

File on ram benchmark:

# dd if=/dev/shm/f of=/dev/null bs=16k 
96000+0 records in
96000+0 records out
1572864000 bytes (1.6 GB, 1.5 GiB) copied, 0.379551 s, 4.1 GB/s

Loopback device benchmark:

# dd if=/dev/loop0 of=/dev/null bs=16k 
96000+0 records in
96000+0 records out
1572864000 bytes (1.6 GB, 1.5 GiB) copied, 1.62812 s, 966 MB/s

Is there any better way to chage the file into the device?

If no is there anything i can do to make loop-device faster?

Edit 1: System specs: ram ddr2 667 and cpu e5500

I have tried created ramdisk and /dev/shm/f with random data and for testing changed bs=16k to bs=512k. hear are the result:

1. the file on ramfs without direct io(doesnt accept the direct flag) 3.5GB :

# dd if=/dev/shm/f of=/dev/null bs=512K status=progress 
2048+0 records in
2048+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.3091 s, 3.5 GB/s

2. ramdisk without and with direct io ( 1GB/s and 3.4GB/s) :

# dd if=/dev/ram0 of=/dev/null bs=512K status=progress 
937951232 bytes (938 MB, 894 MiB) copied, 1.00036 s, 938 MB/s
2048+0 records in
2048+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.22576 s, 876 MB/s
# dd if=/dev/ram0 of=/dev/null bs=512K status=progress iflag=direct
2048+0 records in
2048+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.319873 s, 3.4 GB/s

3)loop device without and with direct (954MB/s and 2.5GB/s):

# dd if=/dev/loop0 of=/dev/null bs=512k status=progress   
2048+0 records in
2048+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.1009 s, 975 MB/s
# dd if=/dev/loop0 of=/dev/null bs=512k status=progress iflag=direct
2048+0 records in
2048+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.436886 s, 2.5 GB/s

4. ramdisk in device mapper (920MB/s and 2.4GB/s)

mount /dev/mapper/ram-snap /mnt/sta1
cat /dev/urandom >/mnt/sta1/tst
# dd if=/mnt/sta1/tst of=/dev/null bs=512K count=1500 status=progress  conv=sync                 
1500+0 records in                                                                                                                                 
1500+0 records out
786432000 bytes (786 MB, 750 MiB) copied, 0.850705 s, 924 MB/s
# dd if=/mnt/sta1/tst of=/dev/null bs=512K count=1500 status=progress iflag=direct conv=sync
1500+0 records in
1500+0 records out
786432000 bytes (786 MB, 750 MiB) copied, 0.327174 s, 2.4 GB/s

5. loop device on dm without and with direct(950MB/s and 1.7GB/s)

# dd if=/mnt/sta1/tst of=/dev/null bs=512K count=1500 status=progress 
1500+0 records in
1500+0 records out
786432000 bytes (786 MB, 750 MiB) copied, 0.827005 s, 951 MB/s
# dd if=/mnt/sta1/tst of=/dev/null bs=512k count=1500 status=progress iflag=direct 
1500+0 records in
1500+0 records out
786432000 bytes (786 MB, 750 MiB) copied, 0.451071 s, 1.7 GB/s

Well roughly it seems even in the case that you can use direct flag there is about 30% reduction in performance in (case 3, 4 and 5) and in practical situation you dont have control on direct flag so the real performance drop is 60%-70%(read speed without direct on my system around 800-1000 MB/s).

Considering that some ssd have the same read speeds, i think something is wrong.

Is there any way to make read from loop device(not loop file attached to loop device ) direct by defult?

Answer 1

It's rather strange to put a file system in RAM, just to create a block device-backing file on it... Plus, ramfs has been less popular in the recent years, in favor of the better tunable tmpfs. I don't think with kernel hugepages support you should see much of a real-world difference in speed, but still, your ramfs is relatively slow, as usually, ramfs is faster than tmpfs, and my tmpfs (on oldish DDR3!) with the same bs does >6 GB/s out of the box.

Instead of using ramfs/tmpfs, just create a ramdisk. Many of the instructions you find here are a bit old – you might or might not need the brd kernel module to create a /dev/ram0 (or ram1, 2,...) device that you can directly use. I'd recommend this answer:

sudo modprobe brd rd_nr=1 rd_size=$((4 * 1048576))

Also, another aspect: 16k is a really small size for this. You might be much faster working in blocks of 512k or 1M. "But," you say, "that's not sensible, as my dm will use 16k blocks!".

Well, that's something you can affect through what you do atop of md. Also, userland speed might not matter much here – the speed at which you copy will heavily be influenced by the number of context switches between kernel and userland.

For a preview how fast you can get, try adding iflag=direct to your dd calls from ramdisk (and weep):

sudo dd if=/dev/ram0 of=/dev/null iflag=direct bs=16k
262144+0 records in
262144+0 records out
4294967296 bytes (4,3 GB, 4,0 GiB) copied, 0,457708 s, 9,4 GB/s
sudo dd if=/dev/ram0 of=/dev/null iflag=direct bs=1M
4096+0 records in
4096+0 records out
4294967296 bytes (4,3 GB, 4,0 GiB) copied, 0,220438 s, 19,5 GB/s

Answer 2

Reading a loop device incurs overhead which will be noticable when benchmarking it, moreso when you are essentially benchmarking ram speed. If you expect the loop device to be just as fast as the direct transfer, then that means the kernel is not allowed to copy data multiple times internally even though you're passing through multiple layers of abstraction which probably involves tons of additional syscalls... Apparently, the kernel implementation just isn't good enough for that.

Regular overhead aside, loop devices on tmpfs files can also have possible side effects. So performance may be bad depending on how the file was created.

# truncate -s 1500M /dev/shm/f
# du /dev/shm/f
0   /dev/shm/f

Creating a file with truncate makes it sparse, so it does not actually occupy any space.

Nothing changes when you read the file directly:

# cat /dev/shm/f > /dev/null
# du /dev/shm/f
0   /dev/shm/f

However, it's different with a loop device:

# losetup --find --show /dev/shm/f
/dev/loop1
# cat /dev/loop1 > /dev/null
# du /dev/shm/f
1536000 /dev/shm/f

So, reading the loop device caused the tmpfs file to be allocated. Which means RAM was allocated for it, so possibly other RAM had to be freed for it. If there is swap space, even swapping might be involved in this process.

This is a possible side effect which can be unexpected.

You can try a proper ramdisk device instead of using tmpfs with loop devices and hope it will have less overhead than loop on a file.