2

I'm trying to find a way to speed up my IO to/from a USB stick on my Debian system, maybe with some caching.

Background: I've written a script that does debootstrap, i.e. installing a Debian system to some disk. I want to use that mostly for USB sticks. But it's terribly slow (I don't know why, but it's a different topic). I found a way to make it faster by more than a magnitude: Execute the procedure on a loop device instead, backed by some temporary image file on the harddrive. Then dd that image to the usb stick afterwards.

My question: Can I make that equally fast (or even faster?) without that temporary image file on the harddrive? It's not trivial to estimate how large it has to be. To me it feels more complicated than it should be. And slower than it has to be. There is enough RAM, and a big swap space. Can I tell the kernel to cache all that more aggressively, when I just run the procedure directly on the USB stick? It could actually do all that in RAM, and just write to the stick in background, as fast as possible, but without blocking my procedure. In the end I would run sync, wait for it, and unplug the stick.

Ideally it should be something non-invasive. Custom kernels aren't an option, and I'm not going to reconfigure my global system settings in ways that are problematic (it's my main workstation and I don't want to make it less reliable).

1
  • That's not cache (which is when the system remembers what it's read), that's buffering (which is when the system is asked to write something, it doesn't wait for the device to acknowledge the write before proceeding). Nov 13, 2021 at 13:04

2 Answers 2

1

You loop device probably discards ATA cache flushes / sync requests, which are very slow on common USB stick. You can obtain a similar result by disabling flushes at the USB disk level (by issuing something as echo "write through" > /sys/block/$device/queue/write_cache) and/or mounting the USB stick with the nobarrier option (for ext3/4 filesystem).

0

The Debian package management tools call fsync1 frequently so that if a system crashes during an update, it ends up in a consistent state. Sync-type calls require waiting for the device to finish the write and guarantee that if the system crashes at that point, the data will still be there. On a USB drive, this is very slow (flash-type memory is fast to read but slow to write).

Install eatmydata and use the eatmydata command to run debootstrap. This makes the fsync calls do nothing, so writes will be lot faster. Commands that are not run through eatmydata are unaffected. If the system crashes during the installation, the content that was written from the debootstrap command may be unusable, which is bad if that's your main system but not a problem if it's a USB stick and you can just try again from scratch.

Note that eatmydata does not persist across privilege elevation, e.g. you need to run sudo eatmydata debootstrap … and not eatmydata sudo debootstrap ….

1 And/or fdatasync, etc.

You must log in to answer this question.