3

I am trying to write a bash script to get the total size of sub folders in a S3 bucket.

My bucketpath s3://path1/path2/subfolders

Inside the path2 folder i have many sub-folder like

2019_06
2019_07
2019_08
2019_09
2019_10
2019_11
2019_12

I need to get the size of each subfolder in a bash script.

I wrote a script like

**

#!/bin/bash

    FILES=$(mktemp)

    aws s3 ls "s3://path1/path2/"  >> "$FILES"

   cat $FILES
echo
for file in $FILES
do    
  if [ ! -e "$file" ]      
  then 
s3cmd du -r  s3://path1/path2/$file
   echo "$file"; echo
   continue               
 fi 

echo
done

**

The output of cat $tmpfile is as below

2019_06
2019_07
2019_08
2019_09
2019_10
2019_11
2019_12

But am getting error. While passing the variable into the for loop. Ideally my aim is like for each iteration when for loop runs inside do .....The command should be like

s3cmd du -r s3://path1/path2/2019_06

s3cmd du -r s3://path1/path2/2019_07

s3cmd du -r s3://path1/path2/2019_08

etc...

So that i can get the total size of the folder

Kindly help!

2 Answers 2

1

Instead of making so many requests I would list all the objects in the bucket recursively and then add-up all the sizes locally from the output.

For start: aws s3 ls --recursive s3://path1/ > all-files.log

And then process all-files.log locally. Much easier :)

1
  • As suggested i have used like aws s3 ls "s3://path1/path2/" >> "$FILES" Can u pls guide me how to use $FILES in for loop to get the size of the folder
    – clarie
    Commented Apr 29, 2020 at 7:33
0

In you original script on first step you use $FILES as temp file name where you store S3 files names. But on your final step you expect the file list to be in the array $FILES.

We could fix just this error but I recommend to rewrite the script so it just process ls result without using temp files. That simplifies things a lot.

Here is working script that you can even add as function in your ~/.bashrc:

function s3du {
    readonly folder_to_scan=${1:?"The argument 's3://bucket/folder_to_scan/' must be specified."}

     for subfolder in $(aws s3 ls "${folder_to_scan}" | grep PRE | awk '{print $2}'); do 
        echo "${folder_to_scan}${subfolder}:" 
        aws s3 ls "${folder_to_scan}${subfolder}" --recursive \
            --human-readable \
            --summarize \ 
            | tail -n2 
    done
}

Use it like this s3du s3://my-bucket/my-folder/

1
  • Welcome to Unix & Linux! Your answer is a bit terse and could be improved by explaining the issue, i.e., the underlying cause of the problem and why your solution correctly would resolve it for the user in the question.
    – Kusalananda
    Commented Dec 21, 2022 at 21:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .