Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 4 down vote favorite
1

I have to copy a directory structure to a new host as part of an application migration. The directory structure is as follows:

{10 - ff} / {00 - ff} / {00 - ff} (non-inclusive) / uniquefilename

So there are millions of files. The third level of the tree is not fully populated, and not every third level directory has a file in it.

a du takes a LONG time to complete for even a second level folder. Using tar | nc would take 10 days. I can get a 1 day outage, but not 10.

share|improve this question
How do you make your regular backups? Maybe the backup is using the exact fast mechanism you're looking for. – jippie Jun 27 '12 at 9:10
What filesystem is this on? What kind of storage (disk partition, RAID volume, etc.)? On what OS? – Gilles Jun 27 '12 at 22:51
By the way, this is the kind of situation for which it's really convenient to have your data on a RAID-1 (mirror) volume, even if there's just one disk underneath: temporarily make the second storage medium a second component of the RAID-1 array, then remove the original component. – Gilles Jun 27 '12 at 22:56

3 Answers

up vote 6 down vote

Use rsync for this.

  1. Prior to when you have to flip over to using the new server, do an rsync of the whole directory from one host to the other. From the sound of it, this will take a long time to complete.
  2. Once it completes, run rsync again to catch any files modified during the first rsync.
  3. Keep doing periodic rsync runs until you want to perform the cutover. This is just to keep the volume of data that has to be transferred low.
  4. Take an outage and stop all writes to the old server.
  5. Perform one last rsync
  6. Start up the services on the new server
share|improve this answer
IMHO rsync may be very slow creating indexes before the first sync. I experienced this with large FS-based media database. – Aurelien Jun 27 '12 at 13:24
Indeed, I have tested rsync in "-n" mode. It took 15 minutes to prep the file list for one of the 254 top-level directories. That multiplies out to almost 3 days, just to traverse and prepare for any file copies. That is a bit long for an outage, unfortunately. – Quijybo Jun 27 '12 at 13:27
up vote 2 down vote

If this directory structure is the bulk of the data on it's device/partition, ignore the directory structure, if that is what is causing the problem.

Instead, unmount the device (or remount read-only) and use dd(1) with netcat to copy the filesystem blocks across to a partition of the same size on the new host.

If there's extraneous data outside the hierarchy you want, delete it after migration.

You could also consider pulling the drive and plugging it into the new host if the network between the hosts may be a bottleneck.

share|improve this answer
That is a great idea - there is unfortunately no backup of this volume...yet. I'd be loath to unplug the drive. And anyway, we want this dataset off one spindle and onto SAN. Time to read up on dd! – Quijybo Jun 27 '12 at 13:29
@Quijybo Don't waste your time on dd, it isn't useful here. You can use something like nc -l -p 999 >/dev/sdc1 on the receiver side and nc new-host.example.com 999 </dev/sdb1 on the sender side if your network is physically secure, or ssh root@new-host.example.com tee /dev/sdc1 </dev/sdb1 from the sender side (assuming the volume is on /dev/sdb1 on the old host and /dev/sdc1 on the new host). – Gilles Jun 27 '12 at 22:55
up vote 0 down vote

For a one shot copy, and if you can have source and target mounted on the same system, you can do 

find [source top dir] | cpio -dump [target mountpoint]

It will copy and preserve file attributes. There should be compression parameter in cpio tool too.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.