Use substrings of filename as parameters in for loop that builds aws command

Question

I'm trying to upload all files in a directory to s3 using dates in the filenames as parameters to create the s3 locations. Here's what I have so far.

for file in /home/ec2-user/clickparts/t*; do 
    year="${file:9:4}"
    month="${file:14:2}"
    day="${file:17:2}"
    aws s3 cp "$file" s3://mybucket/json/clicks/clickpartition/$year/$month/$day/
done

Below is the output for the file "the_date=2017-05-04"

upload: ./the_date=2017-05-04 to s3://mybucket/json/clicks/clickpartition/-use//c/ic//the_date=2017-05-04

I want to put the file in

s3://mybucket/json/clicks/clickpartition/2017/05/04/the_date=2017-05-04

Answer 1

Given a file "the_date=2017-05-04", your for loop will set the file variable to /home/ec2-user/clickparts/the_date=2017-05-04. If you take 4 characters from the 9th character, you get -use, which is what you see where your year variable is used.

One way to fix this is to take account of the number of characters in your path, and add the number of characters (in this case 26) to each of the start numbers when setting your year month and day variables.

Another way might be to change to the appropriate directory before the for loop (and change back after it finishes), then your for loop becomes for file in t*; do, which would set your file variable to what I believe you are expecting.

Answer 2

You can also run it through basename filename="$(basename -- "$file")"