4

Command I have is:

time find . -type f -print -exec cp {} /dev/null \;

This command finds all files in current folder and subfolders, print the name of each file and copy each of them to /dev/null. At the end it shows how much time it took to copy all the files.

What I need is to count (show) all copied bytes at the end (so I would be able to compute the read speed //cache doesn't matter//) and/or show each file's size beside it's name.

If there would be possibility to show progress for each file (pv) - that would be great!

For this purpose I'm using Cygwin and it's bash shell, but script should also work on real Linux systems.

EDIT: The idea is to read the files, not to copy them (rsync).

Improve this question
1
  • I was afraid of rsync answers before the first answer. I've made an edit for that. I've already read other similar questions but didn't find even the direction that would solve the problem.
    – pbies
    May 31, 2014 at 20:05

2 Answers 2

Reset to default
6

Not sure I understand your question fully, but what about:

find . -type f -exec pv -N {} {} \; > /dev/null

Gives an output like:

  ./file1:  575kB 0:00:00 [1.71GB/s] [=======================>] 100%
  ./file2: 15.2GB 0:00:07 [2.22GB/s] [==>                      ] 15% ETA 0:00:38
Improve this answer
2
  • Very nice! Still an approach. What about the sum of all copied bytes? Can we intermediately sum each file to a variable?
    – pbies
    May 31, 2014 at 21:23
  • The answer is time find . -type f -exec pv -N {} {} \; | wc -c
    – pbies
    May 31, 2014 at 23:45
0

Rather then use cp and find for this task you might want to think about using rsync instead.

Example

$ time rsync -avvz -O --stats --checksum --human-readable \
    --acls --itemize-changes --progress \
    --out-format='[%t] [%i] (Last Modified: %M) (bytes: %-10l) %-100n' \
    "<fromDir>" "<toDir>" | tee /path/to/log.txt

Example

This will generate a report that looks like this.

the command

$ time rsync -avvz -O --stats --checksum --human-readable --acls --itemize-changes --progress --out-format='[%t] [%i] (Last Modified: %M) (bytes: %-10l) %-100n' "How_to_Write_Shared_Libraries" "/home/saml/newdir/." | tee ~/rsync.txt

details of each file transferred

sending incremental file list
delta-transmission disabled for local transfer or --whole-file
[2014/05/31 15:12:34] [cd+++++++++] (Last Modified: 2014/02/21-15:42:44) (bytes: 4096      ) How_to_Write_Shared_Libraries/                                                                      
[2014/05/31 15:12:34] [>f+++++++++] (Last Modified: 2013/12/06-19:59:22) (bytes: 766590    ) How_to_Write_Shared_Libraries/dsohowto.pdf                                                          
     766.59K 100%   20.00MB/s    0:00:00 (xfer#1, to-check=1/3)
[2014/05/31 15:12:34] [>f+++++++++] (Last Modified: 2014/02/21-15:42:44) (bytes: 44        ) How_to_Write_Shared_Libraries/url.txt                                                               
          44 100%    1.23kB/s    0:00:00 (xfer#2, to-check=0/3)
total: matches=0  hash_hits=0  false_alarms=0 data=766634

stats about the transfer as a whole

rsync[5923] (sender) heap statistics:
  arena:        1073152   (bytes from sbrk)
  ordblks:            5   (chunks not in use)
  smblks:             1
  hblks:              2   (chunks from mmap)
  hblkhd:        401408   (bytes from mmap)
  allmem:       1474560   (bytes from sbrk + mmap)
  usmblks:            0
  fsmblks:           96
  uordblks:      410512   (bytes used)
  fordblks:      662640   (bytes free)
  keepcost:      396928   (bytes in releasable chunk)

rsync[5926] (server receiver) heap statistics:
  arena:         286720   (bytes from sbrk)
  ordblks:            2   (chunks not in use)
  smblks:             5
  hblks:              3   (chunks from mmap)
  hblkhd:        667648   (bytes from mmap)
  allmem:        954368   (bytes from sbrk + mmap)
  usmblks:            0
  fsmblks:          384
  uordblks:      180208   (bytes used)
  fordblks:      106512   (bytes free)
  keepcost:      102336   (bytes in releasable chunk)

rsync[5925] (server generator) heap statistics:
  arena:         135168   (bytes from sbrk)
  ordblks:            2   (chunks not in use)
  smblks:             6
  hblks:              2   (chunks from mmap)
  hblkhd:        401408   (bytes from mmap)
  allmem:        536576   (bytes from sbrk + mmap)
  usmblks:            0
  fsmblks:          464
  uordblks:       88688   (bytes used)
  fordblks:       46480   (bytes free)
  keepcost:       32800   (bytes in releasable chunk)

summary stats of the transfer

Number of files: 3
Number of files transferred: 2
Total file size: 766.63K bytes
Total transferred file size: 766.63K bytes
Literal data: 766.63K bytes
Matched data: 0 bytes
File list size: 143
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 667.27K
Total bytes received: 54

sent 667.27K bytes  received 54 bytes  1.33M bytes/sec
total size is 766.63K  speedup is 1.15

time it took

real    0m0.092s
user    0m0.053s
sys     0m0.008s
Improve this answer
1
  • If I set other folder as a target, the copying process is very slow (20-50MB/s, but this is SSD, so it should go up to 200-300MB/s). I can't set target to /dev/null (see EDIT - no copy, just read).
    – pbies
    May 31, 2014 at 19:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .