2

I am trying to speed up the process of copying hundreds of file between 2 S3 buckets in my account.

Bucket1
dt=2017-01-01/ 
   file1, file2, file3 ... file100
..
..
dt=2017-01-31/

Each file is a ~300-400 MB and even 1 GB in some cases.

Now, I need to copy them over to a second bucket, Bucket 2 with the same structure.

I tried using the aws cli command, but that is pretty slow - aws s3 cp s3://Bucket1/ s3://Bucket2/ --recursive

Next, I tried launching parallel processes using a script with & -

while read line
do
aws s3 cp s3://Bucket1/${line} s3://Bucket2/${line} --recursive &
done < date-list

Contents of date-list
dt=2017-01-01/
dt=2017-01-02/
..
..

Even this is not too much of an improvement and I don't see all the folders getting copied. After 2017-01-05, it kinda stops. Wonder why.

Looking up for possible solutions, I came upon this blog by AWS where they seem to be using xargs https://aws.amazon.com/blogs/apn/getting-the-most-out-of-the-amazon-s3-cli/

So I modified the script to use xargs instead -

while read line
do
echo ${line} | xargs -n1 -P100 -I {} aws s3 cp s3://Bucket1/{} s3://Bucket2/{} --recursive
done < date-list

Is there a way I can trigger the parallel copy of all days of a particular month? And then, maybe later, of the entire year. Right now, it is copying day by day and it still is slow considering the whole year+ data that needs to be copied.

Would really appreciate any pointers.

1

According to Amazon, you are on the right track with parallel calls through the AWS CLI. Amazon definitely recommends it as a performance optimization technique.

Have you tried setting max concurrent requests in your configurations or when you call the script?

Here is my one liner:

$ seq 0 93 | xargs -I {} date -d "2017-01-01 {} days" +%Y-%m-%d | xargs -I {} -P31 aws s3 cp s3://source_bucket/dt={}/ s3://dest_bucket/dt={}/ --recursive --no-progress

You can set the total number of days using the seq command (93 for ~ 3 months) and set the number of parallel calls (~30 for a month at a time) using the -P flag.

I use the --no-progress option instead of the --quiet option since I can then pipe the output of the command to a log file of successful transfers.

  • Thanks for the pointer! I will surely try what you recommend and update – Abi Jan 18 '18 at 8:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.