I am trying to speed up the process of copying hundreds of file between 2 S3 buckets in my account.
Bucket1
dt=2017-01-01/
file1, file2, file3 ... file100
..
..
dt=2017-01-31/
Each file is a ~300-400 MB and even 1 GB in some cases.
Now, I need to copy them over to a second bucket, Bucket 2 with the same structure.
I tried using the aws cli command, but that is pretty slow -
aws s3 cp s3://Bucket1/ s3://Bucket2/ --recursive
Next, I tried launching parallel processes using a script with & -
while read line
do
aws s3 cp s3://Bucket1/${line} s3://Bucket2/${line} --recursive &
done < date-list
Contents of date-list
dt=2017-01-01/
dt=2017-01-02/
..
..
Even this is not too much of an improvement and I don't see all the folders getting copied. After 2017-01-05, it kinda stops. Wonder why.
Looking up for possible solutions, I came upon this blog by AWS where they seem to be using xargs https://aws.amazon.com/blogs/apn/getting-the-most-out-of-the-amazon-s3-cli/
So I modified the script to use xargs instead -
while read line
do
echo ${line} | xargs -n1 -P100 -I {} aws s3 cp s3://Bucket1/{} s3://Bucket2/{} --recursive
done < date-list
Is there a way I can trigger the parallel copy of all days of a particular month? And then, maybe later, of the entire year. Right now, it is copying day by day and it still is slow considering the whole year+ data that needs to be copied.
Would really appreciate any pointers.