3

I'm working on a python script which takes care to migrate a mysql database with a certain schema / structure, into a postgresql database with a different structure.

During the development phase I was working inside a Virtual Machine(CentOS7) with all my environment correctly set up.

Currently I'm in the testing phase, and I'm trying to run the script in a real server for the first time, but I'm already facing troubles caused by different environment (python version, or python-modules version incompatibility).

Since I will have to execute this script into many servers(All of them will be GNU/Linux servers, most of them CentOS, some Debian), I'm looking for a way to integrate python, and all the python-modules (dependencies) directly into my script, a sort of portable version of python, if you know what I mean.

E.g. I would like to integrate into my script package the following elements/binaries:

  • Python 2.7.5
  • mysql-connector-python-2.1.3-1
  • python-psycopg2
  • 2
    Define "portable". Please edit your question and explain what systems this should work on. Will they all be Linux? Will they all be CentOS? All CentOS7? – terdon Apr 4 '16 at 9:46
  • If all servers are running CentOS, you may want to look at CentOS's "The Software Collections." wiki.centos.org/AdditionalResources/Repositories/SCL More specifically for SCL and Python27: softwarecollections.org/en/scls/rhscl/python27 – Yuusou Apr 4 '16 at 10:04
  • Thank you for the hint, I've updated the question, all the servers will be GNU/Linux, most of them will be CentOS, some of them Debian, or Debian based – lese Apr 4 '16 at 10:25
4

virtualenv is probably what you are looking for. See http://docs.python-guide.org/en/latest/dev/virtualenvs/:

A Virtual Environment is a tool to keep the dependencies required by different projects in separate places, by creating virtual Python environments for them. It solves the “Project X depends on version 1.x but, Project Y needs 4.x” dilemma, and keeps your global site-packages directory clean and manageable.

Install virtualenv via pip:

$ pip install virtualenv

Basic Usage

  1. Create a virtual environment for a project:

    $ cd  my_project_folder 
    $ virtualenv venv 
    

    virtualenv venv will create a folder in the current directory which will contain the Python executable files, and a copy of the pip library which you can use to install other packages. The name of the virtual environment (in this case, it was venv) can be anything; omitting the name will place the files in the current directory instead.

    This creates a copy of Python in whichever directory you ran the command in, placing it in a folder named venv.

    You can also use a Python interpreter of your choice.

    $ virtualenv -p /usr/bin/python2.7 venv 
    

    This will use the Python interpreter in /usr/bin/python2.7

  2. To begin using the virtual environment, it needs to be activated:

    $ source venv/bin/activate 
    

    The name of the current virtual environment will now appear on the left of the prompt (e.g. (venv)Your-Computer:your_project UserName$) to let you know that it’s active. From now on, any package that you install using pip will be placed in the venv folder, isolated from the global Python installation.

    Install packages as usual, for example:

    $ pip install requests 
    

    If you are done working in the virtual environment for the moment, you can deactivate it:

    $ deactivate
    

If you want to move your environment:

You can make a list of installed packages inside the virtualenv:

    $ pip freeze > requirements.txt

And install them on the destination virtualenv using:

    $ pip install -r requirements.txt

From my experience, virtualenvs can be created and managed for both python2 and python3 (on my system, I have both virtualenv and virtualenv3)

Note that virtualenv does not itself provide the python interpreter. It allows you to create isolated environments where a python interpreter is already available.

IMHO, bundling python binaries into your script would not only make your package much bigger, it would in fact make your script less portable as the binaries would be compiled for the specific OS and glibc. If someone wanted to use the script on a different (linux) OS/architecture it would not be possible unless you provided a package for that version.

  • virtualenv is a great tool, of course, but it doesn’t really add anything that aids portability across servers, which is what the original question asked. Wouldn’t it be better to point the asker at pip freeze, or for the more complicated case (dependencies on external libraries, etc.) at containerisation tools like Docker? – wjv Apr 13 '16 at 6:37
  • hmm, interesting. i'm unaware of pip freeze, I will need take a look. Docker adds another whole layer of complexity and additional dependencies which have to be managed. For the script that OP is writing (migrate mysql -> postres), it may be too much overhead to introduce docker, in my opinion, considering how to setup the connectivity from the container to mysql and postgress DBs (which themselves may not be containerized). YMMV. – nagu Apr 13 '16 at 7:22
  • The python binary included in a virtualenv from an installed system is dynamically linked to the libpython shared object: ldd $(which python) | grep python returns something similar to libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007fe071dc9000) – Bryce Guinta Feb 18 '17 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.