xargs with aws cli (amazon s3 copy)

Question

I am trying to speed up the process of copying hundreds of file between 2 S3 buckets in my account.

Bucket1
dt=2017-01-01/ 
   file1, file2, file3 ... file100
..
..
dt=2017-01-31/

Each file is a ~300-400 MB and even 1 GB in some cases.

Now, I need to copy them over to a second bucket, Bucket 2 with the same structure.

I tried using the aws cli command, but that is pretty slow - aws s3 cp s3://Bucket1/ s3://Bucket2/ --recursive

Next, I tried launching parallel processes using a script with & -

while read line
do
aws s3 cp s3://Bucket1/${line} s3://Bucket2/${line} --recursive &
done < date-list

Contents of date-list
dt=2017-01-01/
dt=2017-01-02/
..
..

Even this is not too much of an improvement and I don't see all the folders getting copied. After 2017-01-05, it kinda stops. Wonder why.

Looking up for possible solutions, I came upon this blog by AWS where they seem to be using xargs https://aws.amazon.com/blogs/apn/getting-the-most-out-of-the-amazon-s3-cli/

So I modified the script to use xargs instead -

while read line
do
echo ${line} | xargs -n1 -P100 -I {} aws s3 cp s3://Bucket1/{} s3://Bucket2/{} --recursive
done < date-list

Is there a way I can trigger the parallel copy of all days of a particular month? And then, maybe later, of the entire year. Right now, it is copying day by day and it still is slow considering the whole year+ data that needs to be copied.

Would really appreciate any pointers.

Answer 1

According to Amazon, you are on the right track with parallel calls through the AWS CLI. Amazon definitely recommends it as a performance optimization technique.

Have you tried setting max concurrent requests in your configurations or when you call the script?

Here is my one liner:

$ seq 0 93 | xargs -I {} date -d "2017-01-01 {} days" +%Y-%m-%d | xargs -I {} -P31 aws s3 cp s3://source_bucket/dt={}/ s3://dest_bucket/dt={}/ --recursive --no-progress

You can set the total number of days using the seq command (93 for ~ 3 months) and set the number of parallel calls (~30 for a month at a time) using the -P flag.

I use the --no-progress option instead of the --quiet option since I can then pipe the output of the command to a log file of successful transfers.