There are three general categories of modules you're dealing with:
- Those supporting programs installed for all users with the OS package system. (This may even include tools and libraries used by people programming in Python; see below.) For these you use the OS packages where you can, and
pip installs to the system directories where necessary.
- Those supporting programs installed by a particular user only for her own use, and also for certain aspects of her "day-to-day" use of Python as a scripting language. For these she uses
pip --user, perhaps pyenv or pythonz, and similar tools and tactics.
- Those supporting development and use of a specific application. For these you use
virtualenv (or a similar tool).
Each level here may also be getting support from a previous level. For example, our user in (2) may be relying on a Python interpreter installed via OS packages.
Going into this in a bit more detail:
System Programs and Packages
Programs written in Python that you want to "just run" are easy: just use the OS install tools and let them bring in whatever they need; this is no different from a non-Python program. This is likely to bring in Python tools/libraries (such as the Python interpreter itself!) that users on your machine may start to rely on; this isn't a problem so long as they understand the dependency and, ideally, know alternative means to handle it on hosts that don't provide those dependencies.
A common and simple example of such a dependency is some of my personal scripts in
~/.local/bin/ that start with
#!/usr/bin/env python. These will work fine (so long as they run under Python 2) on RH/CentOS 7 and most (but not all) Ubuntu installs; they will not under a basic Debian install or on Windows. Much as I dislike my personal environment having much in the way of dependencies on OS packages (I work on a number of different OSes), something like this I find fairly acceptable; my backup plan on the rare hosts that don't have a system Python and can't get one is to go with a User system as described below.
People using a system python interpreter are also usually dependent on the system
pip3. That's about where I usually draw the line on my system dependencies; everything from
virtualenv forward I deal with myself. (For example, my standard activate script relies on whatever
pip is in the path, but downloads its own copy of
virtualenv to bootstrap the virtual environment it's creating.
That said, there are probably circumstances where it's perfectly reasonable to make more of a development environment available. You might have Python interfaces into complex packages (such as a DBMS) where you want to use the system version of that and you feel it's best you also let the system choose the particular Python library code you'll use to talk to it. Or you may be deploying a lot of hosts with a basic development environment for a Python class, and find it easiest to automate with standard system packages.
User "Day-to-day" Programs and Packages
Users may have Python libraries or programs that don't fit well into a virtual environment because they're wanted to help create virtual environments in the first place (e.g., virtualenvwrapper) or they're things you commonly use from the command line even while doing non-Python work. Even if they do have the capability to install system versions of these, they may feel more comfortable installing their own (e.g., because they want to be using the latest version of the tool and its dependencies).
pip --user is what people will be using for this, though certain dependencies, such as the Python interpreter itself, require a bit more than that. pyenv and pythonz are useful for building personal interpreters (whether installed in
~/.local/bin to be the default interpreter or otherwise), and of course one can always just build "by hand" from source if the dev libraries are available.
I try to keep the bare minimum set of things installed here: virtualenvwrapper (because I use it constantly) and perhaps the latest version of pip. I try to avoid dependencies outside the standard library or on Python 3 for personal scripts I write to be used across many hosts. (Though we'll see how long I can hold out with that as I move more and more of these personal scripts to Python.)
Separate Application Development and Runtime Environments
This is the usual virtualenv thing. For development you should almost always be using a virtualenv to ensure that you're not using system dependencies, or often more than one to test against different Python versions.
These virtual environments are also good for applications with a lot of dependencies where you want to avoid polluting your user environment. For example I usually set up a virtualenv for running Jupyter notebooks and the like.