4

For some reason I cannot pass the bash variable $FOLDER as a python argument on the following code. The python script downloads some files from amazon s3.

Bash script:

#!/bin/bash
FOLDER=$(./aws get $BUCKET"/location.txt")
python /data/pythonscript.py $FOLDER
#

The output of the $FOLDER is a regenerated date eg. 2014/07/31/14-16-34 which is used as a path.

Here is the python script:

#!/usr/bin/python

import boto, sys
from boto.s3.connection import S3Connection

access_key = 'accesskey'
secret_key = 'secretkey'
bucket_name = 'a name'
folder_path =  str(sys.argv[1]) if len(sys.argv) > 1 else ''

print("Forwarded folder path " + folder_path)

conn = S3Connection(access_key, secret_key)
bucket = conn.get_bucket(bucket_name)

print("Bucket Location:" + bucket.get_location())

for key in bucket.list(prefix=folder_path, delimiter=''):
        if '.' in key.name:
                file_name = key.name[len(folder_path)+1:]
                print("Downloading file " + file_name)
                key.get_contents_to_filename('/data/temp/' + file_name)

When I execute the bash script without changing the python /data/pythonscript.py $FOLDER line, I get the following output:

Forwarded folder path 2014/07/31/14-16-34 
Buckect Location: 

But when I change it to python /data/pythonscript.py 2014/07/31/14-16-34 , everything works:

Forwarded folder path 2014/07/31/14-16-34
Bucket Location: 
Downloading 2014/07/31/14-16-34/FacetedSearch.zip 
Downloading file FacetedSearch.zip
Downloading 2014/07/31/14-16-34/Location.zip
Downloading file Location.zip
Downloading 2014/07/31/14-16-34/LocationPage.zip
Downloading file LocationPage.zip
6
  • What about this doesn't work? Does the python script not get called with $FOLDER expanded? – drs Jul 31 '14 at 14:20
  • Python is getting the $FOLDER argument but rest of the code doesn't work as expected. But if I change the /data/pythonscript.py $FOLDER to /data/pythonscript.py 2014/07/31/14-16-34 in bash script, it works. Python file also works from command line independently. – Bris Jul 31 '14 at 14:29
  • You've framed your question nicer than a lot of new users, but "rest of the code doesn't work as expected" is still pretty vague. What happens? Is there an error? What is the value of folder_path when using the bash variable $FOLDER as the argument? – drs Jul 31 '14 at 14:42
  • Man, is s3download.py the same script, you called pythonscript.py earlier? – Boris Burkov Jul 31 '14 at 15:42
  • Just a minor complaint, but you mistyped bucket: Buckect Location in line 16. – polym Jul 31 '14 at 15:49
7

Perhaps the aws bash command is returning non-printable characters that you don't see with print(). Try removing them with tr:

FOLDER=$(./aws get $BUCKET"/location.txt" | tr -cd "[:print:]")
0
2

Try to put $FOLDER inside double quotes:

python script.py "$FOLDER"

Change the first line of your script to

#!/usr/bin/env python

This should work.

And be more clear on your questions.

As @drs said, your question is still incomplete:

  • Your examples are still vague and with flaws, like what is the correct name of your script, is it pythonscript or s3download?
  • Is it "work as expected" supposed to have a blank value for bucket.get_location()?
1
  • 1
    I tried your suggestions but unfortunately, I am still getting the same result. The script is named pythonscript.py and the bucket.get_location() indeed returns a blank value. But apart from that blank value the script manages to download the files from amazon s3 when I use the mentioned method. – Bris Jul 31 '14 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.