Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have three ways of running mv. First:

mv /db/dbfile1 /db/dbfile2 /db/dbfile3  /usb_storage/

and second:

mv /db/dbfile1  /usb_storage/ &
mv /db/dbfile2  /usb_storage/ &
mv /db/dbfile3  /usb_storage/ &

third (I think first and this is same ?)

mv /db/dbfile1  /usb_storage/ 
mv /db/dbfile2  /usb_storage/ 
mv /db/dbfile3  /usb_storage/ 

Which of these methods is better? Why?

share|improve this question
1  
the head is moving busily when parallel moving files, this could decrease HDD life, right? –  LiuYan 刘研 Mar 21 '13 at 5:24
1  
I don't think moving multiple files to a single partition target simultaneously would do any good. Since all will be writing to the same storage device there will be no real parallelism. –  ghm1014 Mar 21 '13 at 13:44

2 Answers 2

If there was any merit to moving files in parallel, there's a good chance mv would already be doing that by itself, or a similar tool would exist. ;)

There shouldn't be any point in running mv in parallel on the same source/target disk, since you're usually limited by storage media speeds anyhow. If /usb_storage/ is a USB stick, it won't make the stick go any faster.

mv /db/dbfile1 /db/dbfile2 /db/dbfile3 /usb_storage/

Should be the best method since it does what you want, blocks until it's finished (for your alternative to be roughly equivalent, you'd have to add a wait at the end). Writing files sequentially instead of in parallel also helps to prevent unnecessary file fragmentation.

Also, in case /usb_storage/ isn't large enough to hold all the files you're moving onto it, instead of ending up with three incomplete / no copies when working in parallel, you might get at least one complete file with the sequential solution.

share|improve this answer
    
I have updated one more thing , Please check. –  Rahul Patil Mar 21 '13 at 5:34
    
Your third option has the downside of calling mv thrice instead of once. If possible, you always use one command only. It makes little difference with mv and three files, but it's good practice on a larger scale. Imagine a find with thousands of files - if a single (or few) command(s) can handle them (exec +) that's loads better than starting the same command thousands of times (exec ;), because (re)starting commands itself is work too and it will have more CPU/io overhead than need be. –  frostschutz Mar 21 '13 at 8:44
2  
@frostschutz regarding this statement: "If there was any merit to moving files in parallel, there's a good chance mv would already be doing that by itself, or a similar tool would exist." That's not necessarily true: a tool does exist for this sort of thing (GNU parallel), the user is transferring between volumes, programmers aren't always concerned with optimizing for particular use cases (leaving that to admins), parallelizing every little tool can be considered overkill, etc. –  Joel Davis Mar 21 '13 at 13:24
    
parallel has a different purpose altogether, and mv is something that may take hours or even days. If there was any way to do it faster, there would be a tool for it. If you wish to mention any other software, it should be rsync, which is more powerful in some corner cases (target already has some of the files, hard links, ignore bad timestamps for windows filesystems, etc.), and gives you more control over when to delete the source files. OTOH rsync may not be able to replace mv for moves within the same filesystem, so you have to know what you want. –  frostschutz Mar 21 '13 at 18:32

There are potential performance gains to running mv in parallel: while one process is waiting on a read to return, if another file is in cache, it can go straight to performing another I/O request. Additionally, the IO scheduler can reorder the sector requests for all the files you're trying to move so that the read/write heads on the HDD (obviously not so much on the USB Key) can move around as little or possible.

That said, the performance benefits of running in parallel are going to be pretty minimal unless you have a lot of filesystem fragmentation (in order to benefit from avoiding read/write heads going all over the place).

If you're concerned with performance, though, I would make sure your USB key's scheduler is "noop" so that it doesn't spend time reordering requests when it doesn't matter for a one-time shot on media without read/write heads. If your regular HDD scheduler is cfq you might also try to ionice magic to prioritize your mv operations over anything else going on with the system.

share|improve this answer
    
You can't beat reading a single file sequentially (assuming it's not fragmented). With multiple readers, no matter how much reordering you do, the HDD head still has to jump around more than when reading sequentially. At the same time even with a single process, there's actually more than one process involved in the writing; file system access is cached so the real writes happen in the background. There is no benefit to parallel mv, only overhead, when operating on single source and target disk. For separate disks, you may go parallel to utilize all disks at the same time. –  frostschutz Mar 21 '13 at 18:43
    
If it's sequential then that's how the IO scheduler will order the requests. Assuming you're not parallelizing it to the point of CPU resource contention, there's only performance to be gained, the question is whether it's enough to make it worthwhile. Performing multiple reads shouldn't cause the drive head to spin around because both the ATA controller on the HDD and the IO Scheduler at the OS level should catch and reorder that stuff. The mv binary itself is pretty minimal so I don't think we're going to see much overhead. –  Joel Davis Mar 21 '13 at 18:48
    
(cont.) The point I'm making above is that some of the parallel instances of mv could catch files in cache, which will basically go straight to being a write on the USB key, leaving an opening in the HDD queue for I/O requests associated with the other legs of the operation. This is going to be minor but you may crank out a few extra milliseconds this way. –  Joel Davis Mar 21 '13 at 18:51
    
You will only see I/O overhead, unless you cheat by requiring a file to be already cached, and assuming it would no longer be cached when reading several other files first. But in a general question you can not make assumptions such as these, it's way too unlikely a case. –  frostschutz Mar 21 '13 at 19:21
    
How can there be I/O overhead for the whole process? All reads were going to have to happen anyways, you're just putting them together all at once and letting the I/O scheduler basically plan out a reduced number of swipes with the write head (rather than re-treading when mv moves onto a file that was adjacent to a previously read file). You're not increasing the number of requests, you're just giving more of them up front. There aren't any assumptions here, there are several mechanisms designed to reorder requests and that plan can only be harmed by withholding information from them. –  Joel Davis Mar 21 '13 at 19:35

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.