Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Is there a method of slowing down the copy process on Linux?

I have a big file, say 10GB, and I'd like to copy it to another directory, but I don't want to copy it with full speed. Let's say I'd like to copy it with the speed of 1mb/s, not faster. I'd like to use a standard Linux cp command.

Is this possible? (If yes, how?)

Edit: so, I'll add more context to what I'm trying to achieve.

I have a problem on the ArchLinux system when copying large files over USB (to a pendrive, usb disk, etc). After filling up the usb buffer cache, my system stops responding (even the mouse stops; it moves only sporadically). The copy operation is still ongoing, but it takes 100% resources of the box. When the copy operation finishes, everything goes back to normal -- everything is perfectly responsive again.

Maybe it's a hardware error, I don't know, but I do know I have two machines with this problem (both are on ArchLinux, one is a desktop box, second is a laptop).

Easiest and fastest "solution" to this (I agree it's not the 'real' solution, just an ugly 'hack') would be to prevent this buffer from filling up by copying the file with an average write speed of the USB drive, for me that would be enough.

share|improve this question

migrated from serverfault.com Mar 1 at 18:20

This question came from our site for professional system and network administrators.

2  
What are you trying to accomplish? Why do you want to slow down a file operation? –  Michael Hampton Mar 1 at 14:41
7  
If you are seeking to limit disk-to-disk copy speed in an effort to be "nice" to other I/O-bound processes in the system, you are probably better off taking advantage of the kernel's ability to tune I/O scheduling instead. Specifically, ionice can be used to ensure that your disk-to-disk copy process is scheduled I/O at a lower priority than regular processes. –  Steven Monday Mar 1 at 15:13
3  
This is a classic XY problem question. You should instead ask about why your desktop becomes unresponsive when you copy files to a USB device. –  Michael Hampton Mar 1 at 18:50
3  
Linux actually has ridiculously large I/O buffers these days. RAM sizes have grown faster that mass storage speeds. Maybe you could perform the copy using dd(1) and sync so that it would actually be synced periodically instead of being buffered? And pipe viewer (pv) has a rate limiting option. Something like cat file | pv -L 3k > outfile. Neither are the same as using cp(1), though. –  ptman Mar 1 at 18:56
    
@MichaelHampton, there are several unresolved topics on this issue on ArchLinux's forum, so I figured I'll try to cope with it in a different way, just to make it work. –  antonone Mar 1 at 22:02

7 Answers 7

up vote 11 down vote accepted

You can throttle a pipe

tar -cf - . | throttle -M 1 | tar -C /your/usb -xvf -

-b -k -m limits are in bits
-B -K -M are bytes

share|improve this answer
    
If cp can't be slowed down, then using a custom command is the only option I guess. –  antonone Mar 2 at 9:33

I would assume you are trying not to disrupt other activity. Recent versions of Linux include ionice which does allow you to control the scheduling of IO.

Besides allowing various priorities, there is an additional option to limit IO to times when the disk is otherwise idle. The command man ionice will display the documentation.

Try copying the file using a command like:

ionice -c 3 cp largefile /new/directory

If the two directories are on the same device you may find linking the file does what you want. If you are copying for backup purposes, do not use this option. ln is extremely fast as the file itself does not get copied. Try:

ln largefile /new/directory

Or if you just want to access it from a directory on a different device try:

ln -s largefile /new/directory
share|improve this answer

Instead of cp -a /foo /bar you can also use rsync and limit the bandwidth as you need

From the rsync manpage: --bwlimit=KBPS limit I/O bandwidth; KBytes per second

and the final command:

rsync -av --bwlimit=1 /foo /bar
share|improve this answer

If the ionice solution is not enough (whyever) and you really want to limit I/O to an absolute value there are several possibilities:

  1. the probably easiest: ssh. It has a built-in bandwidth limit. You would use e.g. tar (instead of cp) or scp (if that's good enough; I don't know how it handles symlinks and hard links) or rsync. These commands can pipe their data over ssh. In case of tar you write to /dev/stdout (or -) and pipe that into the ssh client which executes another tar on the "remote" side.

  2. elegant but not in the vanilla kernel (AFAIK): The device mapper target ioband. This, of course, works only if you can umount either the source or target volume.

  3. some self-written fun: grep "^write_bytes: " /proc/$PID/io gives you the amount of data a process has written. You could write a script which starts cp in the background, sleeps for e.g. 1/10th second, stops the background cp process (kill -STOP $PID), checks the amount which has been written (and read? about the same value in this case), calculates for how long cp must pause in order to take the average transfer rate down to the intended value, sleeps for that time, wakes up cp (kill -CONT $PID), and so on.

BTW: I've reached my daily limit. Defer your upvotes until 00:00 UTC, please... ;-) (accepting this answer would be OK, though)

share|improve this answer
    
Yes, normally i'm just using lftp to connect to localhost via scp, and limit the bandwich from there. –  antonone Mar 1 at 22:06
1  
Congrats on 10K, just pushed you over. –  slm Mar 2 at 15:39

Your problem is probably not with your computer, per se, it's probably fine. But that USB flash transition layer has a processor of its own that has to map out all of your writes to compensate for what could be as much as a 90% faulty flash chip, who knows? You flood it then you flood your buffers then you flood the whole bus, then you're stuck, man - after all, that's where all your stuff is. It may sound counter-intuitive but what you really need is blocking I/O - you need to let the FTL set the pace and then just keep up.

(On hacking FTL microcontrollers: http://www.bunniestudios.com/blog/?p=3554)

All of the above answers should work so this is more a "me too!" than anything else: I've totally been there, man. I solved my own issues with rsync's --bwlimit arg (2.5mbs seemed to be the sweet spot for a single, error-free run - anything more and I'd wind up with write-protect errors). rsync was especially suited to my purpose because I was working with entire filesystems - so there were a lot of files - and simply running rsync a second time would fix all of the first run's problems (which was necessary when I'd get impatient and try to ramp past 2.5mbs).

Still, I guess that's not quite as practical for a single file. In your case you could just pipe to dd set to raw-write - you can handle any input that way, but only one target file at a time (though that single file could be an entire block device, of course).

## OBTAIN OPTIMAL IO VALUE FOR TARGET HOST DEV ##
## IT'S IMPORTANT THAT YOUR "bs" VALUE IS A MULTIPLE ##
## OF YOUR TARGET DEV'S SECTOR SIZE (USUALLY 512b) ##
% bs=$(blockdev --getoptio /local/target/dev)

## START LISTENING; PIPE OUT ON INPUT ##
% nc -l -p $PORT | lz4 |\ 
## PIPE THROUGH DECOMPRESSOR TO DD ## 
>    dd bs=$bs of=/mnt/local/target.file \
## AND BE SURE DD'S FLAGS DECLARE RAW IO ##
>        conv=fsync oflag=direct,sync,nocache

## OUR RECEIVER'S WAITING; DIAL REMOTE TO BEGIN ##
% ssh user@remote.host <<-REMOTECMD
## JUST REVERSED; NO RAW IO FLAGS NEEDED HERE, THOUGH ## 
>    dd if=/remote/source.file bs=$bs |\
>    lz4 -9 | nc local.target.domain $PORT
> REMOTECMD  

You might find netcat to be a little faster than ssh for the data transport if you give it a shot. Anyway, the other ideas were already taken, so why not?

[EDIT]: I noticed the mentions of lftp, scp, and ssh in the other post and thought we were talking about a remote copy. Local's a lot easier:

% bs=$(blockdev --getoptio /local/target/dev)
% dd if=/src/fi.le bs=$bs iflag=fullblock of=/tgt/fi.le \
>    conv=fsync oflag=direct,sync,nocache

[EDIT2]: Credit where it's due: just noticed ptman beat me to this by like five hours in the comments.

Definitely you could tune $bs for performance here with a multiplier - but some filesystems might require it to be a multiple of the target fs's sectorsize so keep that in mind.

share|improve this answer

The problem is that the copy is filling up your memory with blocks "in flight," crowding out "useful" data. A known (and very hard to fix) bug in the Linux kernel handling of I/O to slow devices (USB in this case).

Perhaps you can try to parcel out the copying, e.g. by a script like the following (proof-of-concept sketch, totally untested!):

while true do
  dd if=infile of=outfile bs=4096 count=... seek=... skip=...
  sleep 5
done

adjusting seek and skip by count each round. Need to tune count so it doesn't fill up (too much) memory, and 5 to allow it to drain.

share|improve this answer

This problem has nothing to do with errors or faults in hardware or software, it's just your kernel trying to be nice to you and give your prompt back and copy in the background (it uses an in-kernel cache: more RAM, more cache, but you can limit it by writing somewhere in /proc - not reccommended though). Flash drives are too slow and while the kernel writes on it, other IO operations can't be performed fast enough. ionice mentioned a several times in other answers is ok. But have you tried just mounting the drive with -o sync to avoid OS buffering? It's probably the simplest solution out there.

share|improve this answer
    
After enabling -o sync, my Internet is faster than write speed to this USB drive. What I don't understand is why kernel doesn't track how quickly cache pages are getting flushed, and schedule future flushes based on that. It's like it always goes full-speed, even if this poor drive can't keep up with the speed. But that's a topic for another question I guess. –  antonone Mar 4 at 11:07

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.