8

I have 1000000 4-20 kb files in a dir. I need to copy that dir. But it seems I have to do a seek for each files so this takes quite a while.

Is there a way in which I can speed this up?

I am currently thinking that if I could get the disk blocks that these files occupy, I could sort those, merge the blocks that were close (given that sequential read is often faster than seeking) and read these blocks, so that they were in RAM cache (I have 32 GB RAM) before doing the copy.

But for that to work I need a way to identify which blocks the files are on.

I am using EXT4 on a magnetic device (i.e. not SSD).

Edit:

This ought to work but it does not:

ls |
parallel -IOO --pipe "sudo parallel -j100 hdparm --fibmap {}'|tail -n +5'" |
sort -nk 2 | 
perl -ane 'if($u+10000 < $F[1]) { print "$l ",($u-$l),"\n"; $l=$F[1] } $u=$F[2]' |
sudo parallel --colsep ' ' dd if=/dev/sda1 skip={1} bs=512 count={2} '| cat >/dev/null'

When testing it on a big file it does not cache the file.

Edit2:

Here are some benchmarks. Cache was flushed (echo 3 >/proc/sys/vm/drop_caches) between each run. Measurements done with iostats -dkx 5.

rsync -Hav foo/ bar/: 1800 KB/s
cp -a foo/ bar/: 3600 KB/s
cat sort-by-inode | parallel -j1 -X cp foo/{} bar/: 5000 KB/s
cat sort-by-inode | shuf | parallel -j1 -X cp foo/{} bar/: 3000 KB/s
cat sort-by-inode | shuf | parallel -j10 -X cp foo/{} bar/: 7000 KB/s
cat sort-by-inode | parallel -j10 -X cp foo/{} bar/: 8000 KB/s
cat sort-by-inode | parallel -j100 -X cp foo/{} bar/: 9000 KB/s
cat sort-by-inode | parallel -j500 -X cp foo/{} bar/: 10000 KB/s

So what can we learn from that?

It seems sorting by inode is a good idea. But it seems parallelizing multiple cp boosts performance even further. It is worth stressing that the source foo/ is a magnetic disk, so this attacks the myth that parallelizing I/O to a single spindle will not speed up I/O: Parallelizing clearly and consistently speeds up the copying here.

  • Does your directory contain only files? Is your target location on the same filesystem? – Joseph R. Apr 13 '14 at 10:18
  • For exact command you are using for copying? Something like cp -r /mnt/dir1 /mnt/dirdest or something like cp /mnt/dir1/* /mnt/dirdest? – maxschlepzig Apr 13 '14 at 10:26
  • Only files but not the same filesystem: cp -r /mnt/dir1 /mnt2/dirdest – Ole Tange Apr 13 '14 at 11:44
  • btw, I don't think it helps if you execute hard disk accesses in parallel when you want to minimize disk seeks. – maxschlepzig Apr 13 '14 at 13:20
  • I wrote a program that orders files by their extent number (more likely to be the occurrence on disk) here: github.com/nh2/diskorder – nh2 Feb 19 '17 at 11:09
8

Assuming that

  • entries returned by readdir are not sorted by inode number
  • reading files in inode order reduces the number of seek operations
  • the content of most files are in the initial 8k allocation (an ext4 optimization) which also should yield less seek operations

you can try to speed up copying via copying files in inode order.

That means using something like this:

$ cd /mnt/src
$ ls -U -i | sort -k1,1 -n | cut -d' ' -f2- > ~/clist
$ xargs cp -t /mnt2/dst < ~/clist
  • @mikeserv, what do you mean? ls -U is not enough because it does not sort by inode numbers ... and why should I want -1? – maxschlepzig Apr 13 '14 at 16:19
  • @mikeserv, 'in directory order' is not the same as inode order! If this would be the case you would not have to use a different word for this. What you find strange is not relevant. I've even tested it on a ext4 filesystem. And there the directory order is indeed different from the inode order. -1 just lists 'one file per line' - it does not help with newlines in filenames. For that you can use find -print0/xargs -O. – maxschlepzig Apr 13 '14 at 16:36
  • @mikeserv, what are you talking about? Counter example: mkdir tmp; cd tmp; touch foo"<RETURN>"bar; ls prints 'foo?bar'. A ls -1 also prints 'foo?bar'. A ls -1 | wc -l prints '2'. A find -ls prints the filename as './foo\nbar'. A cp -i ls -1` x` fails with 'cp: target ‘x’ is not a directory'. – maxschlepzig Apr 13 '14 at 17:04
  • Damn - you're teaching me left and right! -q does what I thought -1 would! Again, my apologies - not to mention thanks. – mikeserv Apr 13 '14 at 17:13
3

tar handles hardlinks on its own.

cd "$srcdir" ; tar --hard-dereference -cf - ./* |
    tar -C"${tgtdir}" -vxf -

That way you only have the two tar processes and you don't need to keep invoking cp over and over again.

1

On a similar vein to @maxschlepzig's answer, you can parse the output of filefrag to sort files in the order that their first fragments appear on disk:

find . -maxdepth 1 -type f |
  xargs -d'\n' filefrag -v |
  sed -n '
    /^   0:        0../ {
      s/^.\{28\}\([0-9][0-9]*\).*/\1/
      h
      }
    / found$/ {
      s/:[^:]*$//
      H
      g
      s/\n/ /p
      }' |
    sort -nk 1,1 |
    cut -d' ' -f 2- |
    cpio -p dest_dir

MMV with the above sed script, so be sure to test thoroughly.

Otherwise, whatever you do, filefrag (part of e2fsprogs) will be much faster to use than hdparm as it can take multiple file arguments. Just the overhead of running hdparm 1,000,000 times is going to add a lot of overhead.

Also it probably wouldn't be so difficult to write a perl script (or C program), to a FIEMAP ioctl for each file, create a sorted array of the blocks that should be copied and the files the belong to and then to copy everything in order by reading the size of each block from the corresponding file (be careful not to run out of files descriptors though).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.