0

am trying to write a bash script to get the total size of sub folders in a S3 bucket.

My bucketpath s3://path1/path2/subfolders

Inside the path2 folder i have many sub-folder like

2019_06
2019_07
2019_08
2019_09
2019_10
2019_11
2019_12

I need to get the size of each subfolder in a bash script.

I wrote a script like

#!/bin/bash

FILES=$(mktemp)

aws s3 ls "s3://path1/path2/"  >> "$FILES"

cat $FILES
echo
for file in $FILES
do    
  if [ ! -e "$file" ]      
  then 
    s3cmd du -r  s3://path1/path2/$file
    echo "$file"; echo
    continue               
  fi 

  echo
done

The output of cat $tmpfile is as below

2019_06
2019_07
2019_08
2019_09
2019_10
2019_11
2019_12

But am getting error. While passing the variable into the for loop. Ideally my aim is like for each iteration when for loop runs inside do .....The command should be like

s3cmd du -r s3://path1/path2/2019_06

s3cmd du -r s3://path1/path2/2019_07

s3cmd du -r s3://path1/path2/2019_08

etc...

So that i can get the total size of the folder

Kindly help!

Update I have edited the code as suggested

#!/bin/bash

FILES=$(mktemp)

aws s3 ls "s3://path1/path2/"  >> "$FILES"

for file in `cat $FILES`
do
  if [ -n "$file" ]
  echo $file
done
0

First of all, if you want to check if a file exist no need for exclamation mark ! since[ -e FILE ] will return True if FILE exists.

But the problem is your bash script cannot check if 2019_06 existed because these files are in S3. Lines in $FILES are just strings.

You can check with [ -n STRING ] which means True if the length of "STRING" is non-zero.

for file in `cat $FILES`
do    
   if [ -n "$file" ]
   then 
      echo $file
      s3cmd du -r  s3://path1/path2/$file
   fi
done
| improve this answer | |
  • I have edited the code...can u pls review it..Its not working – clarie Apr 29 at 11:51
  • Could you be more specific? In what way does it not work? Also, the added if statement is lacking the then ... fi control structure. In general, and in particular if you are new to shell scripting, have a look at shellcheck for debugging scripts; it is also available stand-alone on many linux distributions via the package manager. – AdminBee Apr 29 at 11:59
  • My comment was actually directed at @clarie, maybe I should have paid more attention when formulating ... – AdminBee Apr 29 at 12:10
  • @clarie Please check the answer, I edited. – binarysta Apr 29 at 12:12
  • @binarysta Thanks !!! it works๐Ÿ˜Š๐Ÿ˜Š๐Ÿ˜Š – clarie Apr 29 at 12:32
0
aws s3 ls "s3://path1/path2/" | while read file
do    
    # do something with $file
done
| improve this answer | |

Your Answer

By clicking โ€œPost Your Answerโ€, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.