I'm copying terabytes worth of data from an OLD OLD OLD *nix system. I could spend the money to buy new hardware to plug the drives into but I want to avoid spending money.

The system is crappy. It freezes and crashes a lot. So I have to restart the copy process every time. And it takes a long time for it to "catch up" because it has to "skip" the files that it already copied.

I could use rsync but the system is so slow that every time I start an rsync that initial inventory scan/compare takes a while.

So then I thought I could generate a one-time-list of all the files that need to be copied and save them to a file. Then I could have a script that goes through the file and copies that file then deletes that line. If the script/copy fails in the middle I could just continue from where it left off. I don't know how to handle/process the part that would delete the line from the queue but while trying to figure that out I realized there must be some publicly available tool or script that already does what I need.

So the goal is that it would go through some input queue of files to copy and copy them and, as it finishes a copy, it would remove it from the queue. I don't need it for this particular use-case but it should be able to accommodate new entries being added to the queue. And, of course, the queue should be file based so it can be continued from in case of a system error/issue.

Anyone know of one?


While it is recommended to use a "professional", proven tool to fulfill this task reliably, here's a quick-and-dirty on-the-fly approach to a solution. Deleting a line in a file might fail due to system problems, so keeping the successfully copied files' names in a log file seems a bit safer. Create a "todo" file containing the files to copy, e.g. by redirecting ls results, and an empty "log" file. Then run

comm -2 -3 todo log |
while read FN
  do echo cp "$FN" "new/path/to/$FN" && echo "$FN" >> log || echo "$FN" >&2
  done 2>>errlog

Any time this is run, the files already copied and logged will be skipped. You may want to sort the "todo" files to avoid comm's warning "file is not in sorted order".

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.