Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm in a bit of an interesting situation where I have a Python script that can theoretically be run by a variety of users with a variety of environments (and PATHs) and on a variety of Linux systems. I want this script to be executable on as many of these as possible without artificial restrictions. Here are some known setups:

  • Python 2.6 is the system Python version, so python, python2, and python2.6 all exist in /usr/bin (and are equivalent).
  • Python 2.6 is the system Python version, as above, but Python 2.7 is installed alongside it as python2.7.
  • Python 2.4 is the system Python version, which my script does not support. In /usr/bin we have python, python2, and python2.4 which are equivalent, and python2.5, which the script supports.

I want to run the same executable python script on all three of these. It would be nice if it tried to use /usr/bin/python2.7 first, if it exists, then fall back to /usr/bin/python2.6, then fall back to /usr/bin/python2.5, then simply error out if none of those were present. I'm not too hung up on it using the most recent 2.x possible, though, as long as it's able to find one of the correct interpreters if present.

My first inclination was to change the shebang line from:

#!/usr/bin/python

to

#!/usr/bin/python2.[5-7]

since this works fine in bash. But running the script gives:

/usr/bin/python2.[5-7]: bad interpreter: No such file or directory

Okay, so I try the following, which also works in bash:

#!/bin/bash -c /usr/bin/python2.[5-7]

But again, this fails with:

/bin/bash: - : invalid option

Okay, obviously I could just write a separate shell script that finds the correct interpreter and runs the python script using whatever interpreter it found. I'd just find it a hassle to distribute two files where one should suffice as long as it's run with the most up-to-date python 2 interpreter installed. Asking people to invoke the interpreter explicitly (e.g., $ python2.5 script.py) is not an option. Relying on the user's PATH being set up a certain way is also not an option.

Edit:

Version checking within the Python script is not going to work since I'm using the "with" statement which exists as of Python 2.6 (and can be used in 2.5 with from __future__ import with_statement). This causes the script to fail immediately with a user-unfriendly SyntaxError, and prevents me from ever having an opportunity to check the version first and emit an appropriate error.

Example: (try this with a Python interpreter less than 2.6)

#!/usr/bin/env python

import sys

print "You'll never see this!"
sys.exit()

with open('/dev/null', 'w') as out:
    out.write('something')
share|improve this question
    
Not really what you want, therefor comment. But you can use import sys; sys.version_info() to check if the user has the required python version. –  Bernhard Feb 26 '13 at 20:47
1  
@Bernhard Yes this is true, but by then it's too late to do anything about it. For the third situation I listed above running the script directly (i.e., ./script.py) would cause python2.4 to execute it, which would cause your code to detect that it was the wrong version (and quit, presumably). But there's a perfectly good python2.5 that could have been used as the interpreter instead! –  user108471 Feb 26 '13 at 20:52
2  
Use a wrapper script to figure out if there's an appropriate python and exec it if so, otherwise print an error. –  Kevin Feb 26 '13 at 20:54
1  
So do it first thing in the same file. –  Kevin Feb 26 '13 at 21:04
8  
@user108471 : You are assuming the shebang line is handled by bash. It isn't, it's a system call (execve). The arguments are string literals, no globbing, no regexps. That's it. Even if the first arg is "/bin/bash" and the second options ("-c ...") those options are not parsed by a shell. They are handed untreated to the bash executable, which is why you get those errors. Plus, the shebang only works if it is at the beginining. So you are out of luck here, I'm afraid (short of a script which finds a python interpreter and feeds it a HERE doc, which sounds like an awful mess). –  goldilocks Feb 26 '13 at 21:26

5 Answers 5

I'm not expert, but I believe that you shouldn't specify exact python version to use and leave that choice to system/user.

Also you should use that instead of hardcoding path to python in script:

#!/usr/bin/env python

or

#!/usr/bin/env python3 (or python2)

It is recommended by Python doc in all versions:

A good choice is usually

#!/usr/bin/env python

which searches for the Python interpreter in the whole PATH. However, some Unices may not have the env command, so you may need to hardcode /usr/bin/python as the interpreter path.

In various distributions Python may be installed in different places, so env will search for it in PATH. It should be available in all major Linux distributions and from what I see in FreeBSD.

Script should be executed with that version of Python which is in your PATH and which is chosen by your distribution*.

If your script is compatible with all version of Python except 2.4 you should just check inside of it if it is run in Python 2.4 and print some info and exit.

More to read

  • Here you can find examples in what places Python might be installed in different systems.
  • Here you can find some advantages and disadvantages for using env.
  • Here you can find examples of PATH manipulation and different results.

Footnote

*In Gentoo there is tool called eselect. Using it you may set default versions of different applications (including Python) as default:

$ eselect python list
Available Python interpreters:
  [1]   python2.6
  [2]   python2.7 *
  [3]   python3.2
$ sudo eselect python set 1
$ eselect python list
Available Python interpreters:
  [1]   python2.6 *
  [2]   python2.7
  [3]   python3.2
share|improve this answer
1  
I appreciate that what I want to do is counter to what would be considered good practice. What you've posted is entirely reasonable, but at the same time it is not what I am asking for. I do not want my users to have to explicitly point my script at the appropriate version of Python when it is perfectly possible to detect the appropriate version of Python in all situations I care about. –  user108471 Feb 26 '13 at 21:56
1  
Please see my update for why I cannot "just check inside of it if it is run in Python 2.4 and print some info and exit." –  user108471 Feb 27 '13 at 18:24
    
You are right. I just found this question on SO and now I can see that there is no option to do that if you want to have just one file... –  pbm Feb 27 '13 at 21:41
up vote 7 down vote accepted

Based on some ideas from a few comments, I managed to cobble together a truly ugly hack that seems to work. The script becomes a bash script wraps a Python script and passes it to a Python interpreter via a "here document".

At the beginning:

#!/bin/bash

''':'
vers=( /usr/bin/python2.[5-7] )
latest="${vers[$((${#vers[@]} - 1))]}"
if !(ls $latest &>/dev/null); then
    echo "ERROR: Python versions < 2.5 not supported"
    exit 1
fi
cat <<'# EOF' | exec $latest - "$@"
''' #'''

The Python code goes here. Then at the very end:

# EOF

When the user runs the script, the most recent Python version between 2.5 and 2.7 is used to interpret the rest of the script as a here document.

An explanation on some shenanigans:

The triple-quote stuff I've added also allows this same script to be imported as a Python module (which I use for testing purposes). When imported by Python, everything between the first and second triple-single-quote is interpreted as a module-level string, and the third triple-single-quote is commented out. The rest is ordinary Python.

When run directly (as a bash script now), the first two single-quotes become an empty string, and the third single-quote forms another string with the fourth single-quote, containing only a colon. This string is interpreted by Bash as a no-op. Everything else is Bash syntax for globbing the Python binaries in /usr/bin, selecting the last one, and running exec, passing the rest of the file as a here document. The here document starts with a Python triple-single-quote containing only a hash/pound/octothorpe sign. The rest of the script then is interpreted as normal until the line reading '# EOF' terminates the here document.

I feel like this is perverse, so I hope somebody has a better solution.

share|improve this answer
    
Maybe it's not so nasty after all ;) +1 –  goldilocks Feb 27 '13 at 3:19
    
The disadvantage of this is that it will mess up syntax coloring on most editors –  Lie Ryan Feb 27 '13 at 5:28
    
@LieRyan That depends. My script uses a .py filename extension, which most text editors prefer when choosing which syntax to use for coloring. Hypothetically, if I were to rename this without the .py extension, I could use a modeline to hint the correct syntax (for Vim users at least) with something like: # ft=python. –  user108471 Feb 27 '13 at 16:45

As your requirements state a known list of binaries, you could do it in Python with the following. It wouldn't work past a single digit minor/major version of Python but I don't see that happening any time soon.

Runs the highest version located on disk from the ordered, increasing list of versioned pythons, if the version tagged on the binary is higher than the current version of python executing. The "ordered increasing list of versions" being the important bit for this code.

#!/usr/bin/env python
import os, sys

pythons = [ '/usr/bin/python2.3','/usr/bin/python2.4', '/usr/bin/python2.5', '/usr/bin/python2.6', '/usr/bin/python2.7' ]
py = list(filter( os.path.isfile, pythons ))
if py:
  py = py.pop()
  thepy = int( py[-3:-2] + py[-1:] )
  mypy  = int( ''.join( map(str, sys.version_info[0:2]) ) )
  if thepy > mypy:
    print("moving versions to "+py)
    args = sys.argv
    args.insert( 0, sys.argv[0] )
    os.execv( py, args )

print("do normal stuff")

Apologies for my scratchy python

share|improve this answer
    
wouldn't it continue running after execv'ing? so normal stuff would be executed twice? –  Janus Troelsen Feb 26 '13 at 23:38
1  
execv replaces the currently executing program with a newly loaded program image –  mtm Feb 26 '13 at 23:45
    
This looks like a great solution that feels way less ugly than what I came up with. I'll have to give this a try to see if it works for this purpose. –  user108471 Feb 27 '13 at 1:33
    
This suggestion very nearly works for what I need. The only flaw is something I didn't originally mention: to support Python 2.5, I use from __future__ import with_statement, which must be the first thing in the Python script. I don't suppose you happen to know a way to perform that action when starting the new interpreter? –  user108471 Feb 27 '13 at 16:41
    
are you sure it needs to be the very first thing? or just before you try and use any withs like a normal import?. Does an extra if mypy == 25: from __future__ import with_statement work just before 'normal stuff'? You probably don't need the if, if you don't support 2.4. –  mtm Feb 27 '13 at 17:21

The shebang line can only specify a fixed path to an interpreter. There's the #!/usr/bin/env trick to look up the interpreter in the PATH but that's it. If you want more sophistication, you'll need to write some wrapper shell code.

The most obvious solution is to write a wrapper script. Call the python script foo.real and make a wrapper script foo:

#!/bin/sh
if type python2 >/dev/null 2>/dev/null; then
  exec python2 "$0.real" "$@"
else
  exec python "$0.real" "$@"
fi

If you want to put everything in one file, you can often make it a polyglot that starts with a #!/bin/sh line (so will be executed by the shell) but is also a valid script in another language. Depending on the language, a polyglot may be impossible (if #! causes a syntax error, for example). In Python, it isn't very difficult.

#!/bin/sh
''':'
if type python2 >/dev/null 2>/dev/null; then
  exec python2 "$0.real" "$@"
else
  exec python "$0.real" "$@"
fi
'''
# real Python script starts here
def …

(The whole text between ''' and ''' is a Python string at toplevel, which has no effect. For the shell, the second line is ''':' which after stripping quotes is the no-op command :.)

share|improve this answer
    
The second solution is nice as it does not require adding # EOF at the end like in this answer. Your approach basically is the same as outlined here. –  sschuberth Dec 6 '13 at 14:21

You can write a small bash script which checks for the available phython executable and calls it with the script as parameter. You can then make this script the shebang line target:

#!/my/python/search/script

And this script simply does (after the search):

"$python_path" "$1"

I was unsure whether the kernel would accept this script indirection but I checked and it works.

Edit 1

To make this embarrassing unperceptiveness a good proposal finally:

It is possible to combine both scripts in one file. You just write the python script as a here document in the bash script (if you change the python script you just need to copy the scripts together again). Either you create a temporary file in e.g. /tmp or (if python supports that, I don't know) you provide the script as input to the interpreter:

# do the search here and then
# either
cat >"tmpfile" <<"EOF" # quoting EOF is important so that bash leaves the python code alone
# here is the python script
EOF
"$python_path" "tmpfile"
# or
"$python_path" <<"EOF"
# here is the python script
EOF
share|improve this answer
    
This is more or less the solution the was already stated in the last paragraph of the qeustion! –  Bernhard Feb 26 '13 at 21:10
    
Clever, but it requires this magic script to be installed somewhere on every system. –  user108471 Feb 26 '13 at 21:12
    
@Bernhard Oooops, caught. In the future I'll read to the end. As compensation I am going to improve it to a one file solution. –  Hauke Laging Feb 26 '13 at 21:26
    
@user108471 Magic script can contain something like this: $(ls /usr/bin/python?.? | tail -n1 ) but I did not succeed to use this cleverly in a shebang. –  Bernhard Feb 26 '13 at 21:26
    
@Bernhard You want to do the search within the shebang line? IIRC the kernel does not care about quoting in the shebang line. Otherwise (of if this has changed meanwhile) one could do something like `#!/bin/bash -c do_search_here_without_whitespace...;exec $python "$1" But how do that without whitespace? –  Hauke Laging Feb 26 '13 at 21:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.