What is the correct way to upgrade pip in stretch debian?

Question

I have gone through some notes that it is not a good idea to do pip upgrade using sudo command. My question is if I don't give sudo I get permission errors. How can I resolve this? Also, what is the reason sudo is not suggested in order to upgrade pip?

$python -m pip install --upgrade pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
  Rolling back uninstall of pip
  Moving to /home/abc/.local/bin/pip
   from /tmp/pip-uninstall-V4F8Pj/pip
  Moving to /home/abc/.local/bin/pip2
   from /tmp/pip-uninstall-V4F8Pj/pip2
  Moving to /home/abc/.local/bin/pip2.7
   from /tmp/pip-uninstall-V4F8Pj/pip2.7
  Moving to /home/abc/.local/lib/python2.7/site-packages/pip-19.2.3.dist-info/
   from /home/abc/.local/lib/python2.7/site-packages/~ip-19.2.3.dist-info
  Moving to /home/abc/.local/lib/python2.7/site-packages/pip/
   from /home/abc/.local/lib/python2.7/site-packages/~ip
ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/pip-20.0.2.dist-info/top_level.txt'
Consider using the `--user` option or check the permissions.

WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Answer 1

Never upgrade the OS provided version of tools outside of the package management system, because if there's a new package released it will overwrite your changes.

So sudo pip install --upgrade pip is a bad thing. The OS package system believes it controls the files, and you've overridden them. Odd behaviour may result, including installing of an older version than you've previously installed!

If you want a newer version then you can install it in the user profile

% pip install --upgrade --user pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl (1.4MB)
    100% |################################| 1.4MB 615kB/s 
Installing collected packages: pip
Successfully installed pip-20.0.2

This will install the latest version in $HOME/.local/bin

% ls -l .local/bin/pip                                           
-rwxr-xr-x 1 sweh sweh 223 Feb 16 21:49 .local/bin/pip

If you have $HOME/.local/bin on your PATH then you'll always pick up user pip installed programs.

Most of the time, however, you don't need to upgrade pip.

Answer 2

Adding to Stephens answer, you also shouldn't be installing packages globally using pip (i.e., sudo pip install foo). Like upgrading pip, this interferes with distribution-provided files, leading to potential conflicts.

For Python packages on Debian, there are two viable strategies, which you should not mix. Pick one. You can switch from one to the other as the situation demands.

1) Use Debian-provided packages only

That is, anything you can apt install. Need PIL? apt install python-pil or apt install python3-pil. This ensures all the files are correctly managed by Debian, and you get free security support along with the rest of Debian's packages. Well, until July-ish anyway, when Stretch will start to lose security support.

This means you'll sometimes have to settle for a bit older version of a package. Sometimes you might not be able to find the package you need at all. Perhaps this can be supplemented with packages installed using pip --user as Stephen describes, I haven't tried this.

2) Use virtualenv

You can set up a Python Virtual Environment (sudo apt install virtualenv; virtualenv venv; source venv/bin/activate). You can install packages (or upgrade pip) within this virtualenv, and all such installs will be completely contained in that virtualenv. This also means that you can use different versions of packages for different projects, and anything that's on PyPi is available. rm -r venv/ will get rid of the entire Virtual Environment, including any packages. No conflicts with OS files.

This approach makes it a lot more explicit what your project needs, and makes managing that easier. On the other hand, the burden of updating those packages for security fixes becomes yours. It's a bit heavy-handed for small one-off projects, but can be quite useful in specific cases.