1

am trying to write a bash script to get the total size of sub folders in a S3 bucket.

My bucketpath s3://path1/path2/subfolders

Inside the path2 folder i have many sub-folder like

2019_06
2019_07
2019_08
2019_09
2019_10
2019_11
2019_12

I need to get the size of each subfolder in a bash script.

I wrote a script like

#!/bin/bash

FILES=$(mktemp)

aws s3 ls "s3://path1/path2/"  >> "$FILES"

cat $FILES
echo
for file in $FILES
do    
  if [ ! -e "$file" ]      
  then 
    s3cmd du -r  s3://path1/path2/$file
    echo "$file"; echo
    continue               
  fi 

  echo
done

The output of cat $tmpfile is as below

2019_06
2019_07
2019_08
2019_09
2019_10
2019_11
2019_12

But am getting error. While passing the variable into the for loop. Ideally my aim is like for each iteration when for loop runs inside do .....The command should be like

s3cmd du -r s3://path1/path2/2019_06

s3cmd du -r s3://path1/path2/2019_07

s3cmd du -r s3://path1/path2/2019_08

etc...

So that i can get the total size of the folder

Kindly help!

Update I have edited the code as suggested

#!/bin/bash

FILES=$(mktemp)

aws s3 ls "s3://path1/path2/"  >> "$FILES"

for file in `cat $FILES`
do
  if [ -n "$file" ]
  echo $file
done
1
aws s3 ls "s3://path1/path2/" | while read file
do    
    # do something with $file
done
0

First of all, if you want to check if a file exist no need for exclamation mark ! since[ -e FILE ] will return True if FILE exists.

But the problem is your bash script cannot check if 2019_06 existed because these files are in S3. Lines in $FILES are just strings.

You can check with [ -n STRING ] which means True if the length of "STRING" is non-zero.

for file in `cat $FILES`
do    
   if [ -n "$file" ]
   then 
      echo $file
      s3cmd du -r  s3://path1/path2/$file
   fi
done
5
  • I have edited the code...can u pls review it..Its not working
    – clarie
    Apr 29 '20 at 11:51
  • Could you be more specific? In what way does it not work? Also, the added if statement is lacking the then ... fi control structure. In general, and in particular if you are new to shell scripting, have a look at shellcheck for debugging scripts; it is also available stand-alone on many linux distributions via the package manager.
    – AdminBee
    Apr 29 '20 at 11:59
  • My comment was actually directed at @clarie, maybe I should have paid more attention when formulating ...
    – AdminBee
    Apr 29 '20 at 12:10
  • @clarie Please check the answer, I edited.
    – binarysta
    Apr 29 '20 at 12:12
  • @binarysta Thanks !!! it works😊😊😊
    – clarie
    Apr 29 '20 at 12:32
0

Use of globbing can be problematic is some corner cases. The following makes use of find which is less susceptible to these problems.

find "s3://path1/path2/" -type f | while read -r file
do
  echo $file
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.