Make disk/disk copy slower

Question

Is there a method of slowing down the copy process on Linux?

I have a big file, say 10GB, and I'd like to copy it to another directory, but I don't want to copy it with full speed. Let's say I'd like to copy it with the speed of 1mb/s, not faster. I'd like to use a standard Linux cp command.

Is this possible? (If yes, how?)

Edit: so, I'll add more context to what I'm trying to achieve.

I have a problem on the ArchLinux system when copying large files over USB (to a pendrive, usb disk, etc). After filling up the usb buffer cache, my system stops responding (even the mouse stops; it moves only sporadically). The copy operation is still ongoing, but it takes 100% resources of the box. When the copy operation finishes, everything goes back to normal -- everything is perfectly responsive again.

Maybe it's a hardware error, I don't know, but I do know I have two machines with this problem (both are on ArchLinux, one is a desktop box, second is a laptop).

Easiest and fastest "solution" to this (I agree it's not the 'real' solution, just an ugly 'hack') would be to prevent this buffer from filling up by copying the file with an average write speed of the USB drive, for me that would be enough.

Answer 1

You can throttle a pipe with pv -qL (or cstream -t provides similar functionality)

tar -cf - . | pv -q -L 8192 | tar -C /your/usb -xvf -

-q removes stderr progress reporting.

The -L limit is in bytes.

More about the --rate-limit/-L flag from the man pv:

-L RATE, --rate-limit RATE

    Limit the transfer to a maximum of RATE bytes per second.
    A suffix of "k", "m", "g", or "t" can be added to denote
    kilobytes (*1024), megabytes, and so on.

This answer originally pointed to throttle but that project is no longer available so has slipped out of some package systems.

Answer 2

Instead of cp -a /foo /bar you can also use rsync and limit the bandwidth as you need.

From the rsync's manual:

--bwlimit=KBPS

limit I/O bandwidth; KBytes per second

So, the actuall command, also showing the progress, would look like this:

rsync -av --bwlimit=100 --progress /foo /bar

Answer 3

I would assume you are trying not to disrupt other activity. Recent versions of Linux include ionice which does allow you to control the scheduling of IO.

Besides allowing various priorities, there is an additional option to limit IO to times when the disk is otherwise idle. The command man ionice will display the documentation.

Try copying the file using a command like:

ionice -c 3 cp largefile /new/directory

If the two directories are on the same device you may find linking the file does what you want. If you are copying for backup purposes, do not use this option. ln is extremely fast as the file itself does not get copied. Try:

ln largefile /new/directory

Or if you just want to access it from a directory on a different device try:

ln -s largefile /new/directory

Answer 4

If the ionice solution is not enough (whyever) and you really want to limit I/O to an absolute value there are several possibilities:

1. the probably easiest: ssh. It has a built-in bandwidth limit. You would use e.g. tar (instead of cp) or scp (if that's good enough; I don't know how it handles symlinks and hard links) or rsync. These commands can pipe their data over ssh. In case of tar you write to /dev/stdout (or -) and pipe that into the ssh client which executes another tar on the "remote" side.

2. elegant but not in the vanilla kernel (AFAIK): The device mapper target ioband. This, of course, works only if you can umount either the source or target volume.

3. some self-written fun: grep "^write_bytes: " /proc/$PID/io gives you the amount of data a process has written. You could write a script which starts cp in the background, sleeps for e.g. 1/10th second, stops the background cp process (kill -STOP $PID), checks the amount which has been written (and read? about the same value in this case), calculates for how long cp must pause in order to take the average transfer rate down to the intended value, sleeps for that time, wakes up cp (kill -CONT $PID), and so on.

Answer 5

Your problem is probably not with your computer, per se, it's probably fine. But that USB flash transition layer has a processor of its own that has to map out all of your writes to compensate for what could be as much as a 90% faulty flash chip, who knows? You flood it then you flood your buffers then you flood the whole bus, then you're stuck, man - after all, that's where all your stuff is. It may sound counter-intuitive but what you really need is blocking I/O - you need to let the FTL set the pace and then just keep up.

(On hacking FTL microcontrollers: http://www.bunniestudios.com/blog/?p=3554)

All of the above answers should work so this is more a "me too!" than anything else: I've totally been there, man. I solved my own issues with rsync's --bwlimit arg (2.5mbs seemed to be the sweet spot for a single, error-free run - anything more and I'd wind up with write-protect errors). rsync was especially suited to my purpose because I was working with entire filesystems - so there were a lot of files - and simply running rsync a second time would fix all of the first run's problems (which was necessary when I'd get impatient and try to ramp past 2.5mbs).

Still, I guess that's not quite as practical for a single file. In your case you could just pipe to dd set to raw-write - you can handle any input that way, but only one target file at a time (though that single file could be an entire block device, of course).

## OBTAIN OPTIMAL IO VALUE FOR TARGET HOST DEV ##
## IT'S IMPORTANT THAT YOUR "bs" VALUE IS A MULTIPLE ##
## OF YOUR TARGET DEV'S SECTOR SIZE (USUALLY 512b) ##
% bs=$(blockdev --getoptio /local/target/dev)

## START LISTENING; PIPE OUT ON INPUT ##
% nc -l -p $PORT | lz4 |\ 
## PIPE THROUGH DECOMPRESSOR TO DD ## 
>    dd bs=$bs of=/mnt/local/target.file \
## AND BE SURE DD'S FLAGS DECLARE RAW IO ##
>        conv=fsync oflag=direct,sync,nocache

## OUR RECEIVER'S WAITING; DIAL REMOTE TO BEGIN ##
% ssh [email protected] <<-REMOTECMD
## JUST REVERSED; NO RAW IO FLAGS NEEDED HERE, THOUGH ## 
>    dd if=/remote/source.file bs=$bs |\
>    lz4 -9 | nc local.target.domain $PORT
> REMOTECMD  

You might find netcat to be a little faster than ssh for the data transport if you give it a shot. Anyway, the other ideas were already taken, so why not?

[EDIT]: I noticed the mentions of lftp, scp, and ssh in the other post and thought we were talking about a remote copy. Local's a lot easier:

% bs=$(blockdev --getoptio /local/target/dev)
% dd if=/src/fi.le bs=$bs iflag=fullblock of=/tgt/fi.le \
>    conv=fsync oflag=direct,sync,nocache

[EDIT2]: Credit where it's due: just noticed ptman beat me to this by like five hours in the comments.

Definitely you could tune $bs for performance here with a multiplier - but some filesystems might require it to be a multiple of the target fs's sectorsize so keep that in mind.

Answer 6

Lower the dirty page limit. The default limit is insane.

Create /etc/sysctl.d/99-sysctl.conf with:

vm.dirty_background_ratio = 3
vm.dirty_ratio = 10

Then run sysctl -p or reboot.

What's happening is that data is being read faster than it can be written to the destination disk. When linux copies files, what it does is read them into RAM, then mark the pages as dirty for writing to the destination. Dirty pages cannot be swapped out. So if the source disk is faster than the destination disk and you're copying more data than you have free RAM, the copy operation will eat up all available RAM (or at least whatever the dirty page limit is, which could be more than the available RAM) and cause starvation as the dirty pages cannot be swapped out and clean pages get used and marked dirty as they are freed.

Note that his will not completely solve the problem...what linux really needs is some way to arbitrate creation of dirty pages so a large transfer taking place does not eat up all available RAM/all allowed dirty pages.

Answer 7

The problem is that the copy is filling up your memory with blocks "in flight," crowding out "useful" data. A known (and very hard to fix) bug in the Linux kernel handling of I/O to slow devices (USB in this case).

Perhaps you can try to parcel out the copying, e.g. by a script like the following (proof-of-concept sketch, totally untested!):

while true do
  dd if=infile of=outfile bs=4096 count=... seek=... skip=...
  sleep 5
done

adjusting seek and skip by count each round. Need to tune count so it doesn't fill up (too much) memory, and 5 to allow it to drain.

Answer 8

This problem has nothing to do with errors or faults in hardware or software, it's just your kernel trying to be nice to you and give your prompt back and copy in the background (it uses an in-kernel cache: more RAM, more cache, but you can limit it by writing somewhere in /proc - not reccommended though). Flash drives are too slow and while the kernel writes on it, other IO operations can't be performed fast enough. ionice mentioned a several times in other answers is ok. But have you tried just mounting the drive with -o sync to avoid OS buffering? It's probably the simplest solution out there.

Answer 9

I had the same issue with two internal disks i.e. severe system freezes during large file copy.

vm.dirty_ratio sets to reduce the maximum amount of memory allocated to dirty pages.

Tried (based on calculations of read/write speeds) to set various values for -

vm.dirty_ratio = 20
vm.dirty_background_ratio = 10
vm.dirty_background_bytes = 0
vm.dirty_bytes = 0
vm.swappiness=1
vm.dirty_background_ratio = 1
vm.dirty_ratio = 2

A satisfactory solution to prevent freezes during file copy (especially files larger than 3 GB) could not be obtained.

On Ubuntu 19 using gnome-disk-utility 3.34.0 (Disks) I noted a GUI utility under "Drive Settings", called "Write Cache". Enabling "Enable Write Cache" solved the copy delay between two internal drives. The copy speed is reduced by 50% but at least it solved the freezes.

"Enable Write Cache" was set for both drives.

System is however still unsatisfactory for copying to or from external USB.