0

I have 110 large files inside a directory. But I cannot copy them at once since the system gets timed out after 50 min. I want to copy files 1 to 10, next 11 to 20, next 21 to 30, and so on. The order does not matter. I want to copy 10 of the files each time I write the cp one line command. How can I write cp to copy file number 11 to 20 (this is not the head or the tail)?

P.S. my laptop is macbook

The file names have blank spaces in them.

1
rsync -ai /path/to/source/directory/ /path/to/destination/directory

This uses rsync to copy all the files from the source directory to the destination directory. If the copying is interrupted, you may restart it by simply giving the same command again.

The -a option (--archive) preserves the meta-data of the files (owner, permissions, timestamps, etc.) This is used in subsequent runs to determine what files need to be transferred, and what files don't need to be transferred.

The -i option (--itemize-changes) causes rsync to output information about each transfer.


In a comment you mention that you may want to do this against an AWS S3 bucket.

In that case, you may use

aws s3 sync /path/to/source/directory/ s3://bucket/path/to/destination/directory/

This would presumably also be restartable if your machine removes the connection for whatever reason.

See also

Note: I'm not a user of AWS, so I have no way of testing this.

  • Thank you very much @ Kusalananda. It works very well. – fary001 May 12 at 15:48
1

Using GNU find, xargs, and cp, this would copy 10 files at a time:

d="destination_directory..."
s="source_directory..."
find "$s" -type f -print0 | xargs -0rn 10 cp -n -t "$d"
  • Thank you very much @agc. Question: let us assume I run it and it grabs 10 files to copy and it finishes copying them during the 50 min time window that the system has. Then, after I reconnect and run it again for the second time, how it would realize which 10 files have been copied to avoid? – fary001 May 11 at 0:44
  • @fary001, by using the -n (or --no-clobber) option, which prevents cp from overwriting an existing file. See revised answer. – agc May 11 at 1:02
  • Also, it gives an error: "illegal option --r" – fary001 May 11 at 1:08
  • Thanks for the answer to my first question. Does --n also work for aws s3 along with cp command? – fary001 May 11 at 1:09
  • @fary001 Re "gives an error": Odd. The only -r is in xargs -0rn ..., so please run xargs --version | head -1 and post the version number. – agc May 11 at 1:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.