2

I have a large amount of files I need to copy to Box.com using DAV2fs. I've found I'm only able to copy a couple gigs at a time, and then I have to wait for a couple minutes while box catches up or it errors out.

So what I did was I made a list of all the files/directories using "find ./ > outfile.txt"

I want to iterate though the list of files, and after say 100 copies(or whatever) wait 10 minutes. How would I do that? without using cp -r, when the outfile.txt looks like:

/dirctory1
/directory1/file.txt
/directory1/file2.txt

cp omits the directory so file and file2 never get copied. If I do cp -r then it will do directory1 and all it's contents so my file count will be off.

EDIT:

To clarify, I'm interested in the cp portion. Specifically how to get cp to create a directory without doing -r as -r will throw off my count.

5

Try this. (Not tested.)

destination=     # assign 
i=0
while read line; do
    cp --parents "$line" "$destination"
    ((i++))
    [ $i -eq 100 ] && sleep 600 && i=0
done < outfile.txt
# or: done < <(find ... and so on) # to avoid creating the temp file
| improve this answer | |
  • I may have not been thorough enough with my question. My question was about the cp command with the file count. cp will not copy directories, so anything that is not in the root directory will not get copied. But if I do cp -r, as soon as it encounters a directory it will copy all of it's contents and will not increase the count properly. – hemlock Dec 8 '16 at 23:28
  • @hemlock Use the option --parents. – user147505 Dec 8 '16 at 23:50
1

Change the "echo cp" to be your real cp and the following should do it. It'll sleep for 10 minutes after every 100 files. It assumes that none of your filenames have/had newline characters in them, such that each line in your outfile.txt corresponds to a filename.

#!/usr/bin/env bash
count=0
while IFS= read -r file
do
  echo cp "$file" dest/
  count=$((++count))
  [[ $(($count % 100)) == 0 ]] && sleep $((10 * 60))
done < outfile.txt
| improve this answer | |
  • I may have not been thorough enough with my question. My question was about the cp command with the file count. cp will not copy directories, so anything that is not in the root directory will not get copied. But if I do cp -r, as soon as it encounters a directory it will copy all of it's contents and will not increase the count properly. – hemlock Dec 8 '16 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.