Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I need to copy a very large directory (talking in terabytes here) and want to monitor the progress.

I found that pv is a nice utility, but how can I use it for copying directories recursively? (pv src > dst doesn't work because they are directories)

PS: Running on OS X Mountain Lion, pv was installed from Mac Ports

share|improve this question

4 Answers 4

up vote 13 down vote accepted

Why not use rsync --progress [SRC] [DST]

Do check the man rsync page because it has a lot of very useful options. -a for archive is a good start, but it depends on your exact requirements.

Copying through a pipe will unnecessarily slow down the copy process, especially if it is files based.

share|improve this answer
    
Brilliant, thanks :) –  Florian Peschka Nov 18 '12 at 18:18
    
Why would copying through a pipe slow down the process? The buffering provided by pipes and the multi-tasking associated with pipes is likely to help matters I would say. (and rsync is multi-processes and uses pipes internally IIRC). What's a copy-process that is not file-based? –  Stéphane Chazelas Nov 18 '12 at 19:03
    
Bash is convenient, but not fast. Especially when piping one command to the other. –  jippie Nov 18 '12 at 19:25
    
@StephaneChazelas: Because a pipe's buffer is only 8KB. A program like rsync might want to use a much larger internal buffer. –  Zan Lynx Nov 18 '12 at 21:31

You could use tar or pax or cpio:

mkdir -p dst &&
  (cd src && tar cf - .) | pv -trb | (cd dst && tar xpf -)
share|improve this answer
1  
Additionally, you may add a v to the tar command to see the file names. –  peterph Jan 21 at 15:17

Tar.

tar -cf - /var/log/ | pv | tar -C . -x

Example:

# tar -cf - /var/log/ | pv | tar -C . -x
tar: Removing leading `/' from member names
58MB 0:00:05 [ 2.2MB/s] [                   <=> 
share|improve this answer
find source-dir -print0 | xargs -0 -I % pv % > dest-dir/%
share|improve this answer
2  
Please explain tour solutipon, you‘ll want OP to be able to checo it (and learn) –  vonbrand Jan 27 at 1:41

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.