-1

I'm trying to upload all files in a directory to s3 using dates in the filenames as parameters to create the s3 locations. Here's what I have so far.

for file in /home/ec2-user/clickparts/t*; do 
    year="${file:9:4}"
    month="${file:14:2}"
    day="${file:17:2}"
    aws s3 cp "$file" s3://mybucket/json/clicks/clickpartition/$year/$month/$day/
done

Below is the output for the file "the_date=2017-05-04"

upload: ./the_date=2017-05-04 to s3://mybucket/json/clicks/clickpartition/-use//c/ic//the_date=2017-05-04

I want to put the file in

s3://mybucket/json/clicks/clickpartition/2017/05/04/the_date=2017-05-04

2
  • 2
    So, does it work? Does it fail? How does it fail? What are you expecting to happen and what actually happens? Any error messages? What do your file names look like? We can't know if your commands make sense if you don't tell us anything about them! Please edit your question and clarify. – terdon Mar 12 '20 at 18:59
  • @terdon Edited for clarity of these questions. – RagePwn Mar 12 '20 at 19:05
1

Given a file "the_date=2017-05-04", your for loop will set the file variable to /home/ec2-user/clickparts/the_date=2017-05-04. If you take 4 characters from the 9th character, you get -use, which is what you see where your year variable is used.

One way to fix this is to take account of the number of characters in your path, and add the number of characters (in this case 26) to each of the start numbers when setting your year month and day variables.

Another way might be to change to the appropriate directory before the for loop (and change back after it finishes), then your for loop becomes for file in t*; do, which would set your file variable to what I believe you are expecting.

1
  • Also, the question is tagged with "loop-device" instead of "loop". I've tried editing the tag, but it doesn't let me, because I wasn't changing any text in the Question. ("It looks like nothing has changed") Is there something I'm missing, or is someone else able to change it? – mdmay74 Mar 12 '20 at 21:51
1

You can also run it through basename filename="$(basename -- "$file")"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.