4

I ran this command on Ubuntu 14.04 Linux while trying to install a package:

export PYTHONPATH=/usr/local/lib/python2.7/ && sudo python setup.py install

The package I was trying to install is http://github.com/ewels/MultiQC

Then lots of things python related have gone wrong since this. For example, virtualenv doesn't work anymore, and this package doesn't work either:

http://wiki.dnanexus.com/images/files/dx-toolkit-v0.240.1-ubuntu-14.04-amd64.tar.gz

source Downloads/dx-toolkit/environment
dx login
Traceback (most recent call last):
  File "/home/avilella/Downloads/dx-toolkit/bin/dx", line 7, in <module>
    from dxpy.scripts.dx import main
  File "/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages/dxpy/__init__.py", line 143, in <module>
    from . import exceptions
  File "/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages/dxpy/exceptions.py", line 211, in <module>
    network_exceptions = (requests.packages.urllib3.exceptions.ProtocolError,
AttributeError: 'module' object has no attribute 'packages'

How do I revert this to the stage before running this command-line? Even starting from a fresh bash terminal, it now won't find any of my other python libraries. Even with a freshly created Linux user on the same computer, the errors are the same.

Related question wasn't answered: Fixing a broken PYTHONPATH environment variable

[EDIT]

I don't know how to revert to the list of folders before, but I presume it was a combination of /usr/lib/python2.7 and /usr/local/lib/python2.7

Trying virtualenv also fails now:

virtualenv
Fatal Python error: Py_Initialize: Unable to get the locale encoding
  File "/usr/lib/python2.7/encodings/__init__.py", line 123
    raise CodecRegistryError,\
                            ^
SyntaxError: invalid syntax
Aborted (core dumped)

Interactive python works to the extend I tried below:

avilella@ubuntu14:~$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print "foo"
foo
>>> import urllib3
>>> 

I have now gone through the folder and tried both pip install and apt-get install python-${package} with this:

ls /usr/local/lib/python2.7/dist-packages/*/ | grep -v egg | grep '\:' | tr ':' ' ' | grep -v dist-info | xargs -n1 basename  | while read i; do sudo pip install $i; done

Also:

ls /usr/local/lib/python2.7/dist-packages/*/ | grep -v egg | grep '\:' | tr ':' ' ' | grep -v dist-info | xargs -n1 basename  | while read i; do sudo apt-get install -y python-${i}; done

But still same issues.

Package requests is there:

ls -l /usr/lib/python2.7/dist-packages/requests/
total 304
-rw-r--r-- 1 root root 14594 Mar 26  2015 adapters.py
-rw-rw-r-- 1 root root 13750 Sep  4 16:05 adapters.pyc
-rw-r--r-- 1 root root  4333 Sep 24  2013 api.py
-rw-rw-r-- 1 root root  5261 Sep  4 16:05 api.pyc
-rw-r--r-- 1 root root  6173 Dec  5  2013 auth.py
-rw-rw-r-- 1 root root  7206 Sep  4 16:05 auth.pyc
-rw-r--r-- 1 root root   527 Mar 26  2015 certs.py
-rw-rw-r-- 1 root root   736 Sep  4 16:05 certs.pyc
-rw-r--r-- 1 root root  2531 Mar 26  2015 compat.py
-rw-rw-r-- 1 root root  2541 Sep  4 16:05 compat.pyc
-rw-r--r-- 1 root root 16686 Jan  8  2014 cookies.py
-rw-rw-r-- 1 root root 20167 Sep  4 16:05 cookies.pyc
-rw-r--r-- 1 root root  1599 Mar 26  2015 exceptions.py
-rw-rw-r-- 1 root root  3863 Sep  4 16:05 exceptions.pyc
-rw-r--r-- 1 root root   820 Sep 24  2013 hooks.py
-rw-rw-r-- 1 root root  1055 Sep  4 16:05 hooks.pyc
-rw-r--r-- 1 root root  1846 Mar 26  2015 __init__.py
-rw-rw-r-- 1 root root  2501 Sep  4 16:05 __init__.pyc
-rw-r--r-- 1 root root 25332 Mar 26  2015 models.py
-rw-rw-r-- 1 root root 23051 Sep  4 16:05 models.pyc
-rw-r--r-- 1 root root 22042 Mar 26  2015 sessions.py
-rw-rw-r-- 1 root root 18049 Sep  4 16:05 sessions.pyc
-rw-r--r-- 1 root root  3136 Sep 24  2013 status_codes.py
-rw-rw-r-- 1 root root  4439 Sep  4 16:05 status_codes.pyc
-rw-r--r-- 1 root root  3575 Sep 24  2013 structures.py
-rw-rw-r-- 1 root root  6359 Sep  4 16:05 structures.pyc
-rw-r--r-- 1 root root 19642 Mar 26  2015 utils.py
-rw-rw-r-- 1 root root 19386 Sep  4 16:05 utils.pyc

I checked a working version of a different VM I have, where everything python related works, and the requests folder contains exactly the same files.

Any mentions to urllib in the non-working Linux (below) are the same as in the working one (further below), with 2 files changing their date between one an the other:

[       4096 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3
[       3346 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/_collections.pyc
[      22000 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.pyc
[       4096 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/contrib
[       6015 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.pyc
[       6511 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/fields.pyc
[       3162 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/filepost.pyc
[       2074 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/__init__.pyc
[       4096 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/packages
[       9076 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.pyc
[       5850 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/request.pyc
[       9879 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/response.pyc
[      19662 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/util.pyc
[      28639 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py
[       5961 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/fields.py
[       2486 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/filepost.py
[      10159 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/response.py
[      20575 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/util.py
[       1703 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/__init__.py
[       3274 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.py
[       2898 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/_collections.py
[       9009 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.py
[       5874 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/request.py
[      17079 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3-1.7.1.egg-info
[       7142 Oct  5 14:47]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.pyc
[       8385 Oct 26  2016]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.py
[      46549 Oct  5 14:47]  /usr/lib/python2.7/urllib2.pyc
[      50626 Oct  5 14:47]  /usr/lib/python2.7/urllib.pyc
[      51788 Oct 26  2016]  /usr/lib/python2.7/urllib2.py
[      59371 Oct 26  2016]  /usr/lib/python2.7/urllib.py

The working one below:

[       4096 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3
[       3346 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/_collections.pyc
[      22000 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.pyc
[       4096 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/contrib
[       6015 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.pyc
[       6511 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/fields.pyc
[       3162 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/filepost.pyc
[       2074 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/__init__.pyc
[       4096 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/packages
[       9076 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.pyc
[       5850 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/request.pyc
[       9879 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/response.pyc
[      19662 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/util.pyc
[      28639 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py
[       5961 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/fields.py
[       2486 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/filepost.py
[      10159 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/response.py
[      20575 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/util.py
[       1703 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/__init__.py
[       3274 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.py
[       2898 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/_collections.py
[       9009 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.py
[       5874 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/request.py
[      17079 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3-1.7.1.egg-info
[      46549 Jul 12 11:47]  /usr/lib/python2.7/urllib2.pyc
[      50626 Jul 12 11:47]  /usr/lib/python2.7/urllib.pyc
[       7142 Jul 12 11:47]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.pyc
[       8385 Oct 26  2016]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.py
[      51788 Oct 26  2016]  /usr/lib/python2.7/urllib2.py
[      59371 Oct 26  2016]  /usr/lib/python2.7/urllib.py

Another example tool that doesn't work anymore is tosheets https://github.com/kren1/tosheets:

tosheets
Traceback (most recent call last):
  File "/usr/local/bin/tosheets", line 7, in <module>
    from tosheets.tosheets import main
  File "/usr/local/lib/python2.7/dist-packages/tosheets/tosheets.py", line 27, in <module>
    from apiclient import discovery
ImportError: cannot import name discovery

After doing unset PYTHON, then:

>>> import sys
>>> sys.path
['', '/usr/local/lib/python2.7/dist-packages/multiqc-1.4dev-py2.7.egg', '/usr/lib/python2.7/dist-packages', '/usr/lib/pymodules/python2.7', '/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages', '/home/avilella/Downloads/dx-toolkit/lib/python', '/home/avilella', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/avilella/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']

unset PYTHON, the python3:

>>> import sys
>>> sys.path
['', '/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages', '/home/avilella/Downloads/dx-toolkit/lib/python', '/home/avilella', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages']

The contents of $PYTHONPATH before doing any source'ing are empty.

The contents of Downloads/dx-toolkit/environment are below:

# -*- Mode: shell-script -*-                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
# Copyright (C) 2013-2016 DNAnexus, Inc.                                                                                                                                                                                                                                                                                                                                                                                                                                                       
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
# This file is part of dx-toolkit (DNAnexus platform client libraries).                                                                                                                                                                                                                                                                                                                                                                                                                        
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
#   Licensed under the Apache License, Version 2.0 (the "License"); you may not                                                                                                                                                                                                                                                                                                                                                                                                                
#   use this file except in compliance with the License. You may obtain a copy                                                                                                                                                                                                                                                                                                                                                                                                                 
#   of the License at                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
#       http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                                                                                                                                                                                                                                                                                                                             
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
#   Unless required by applicable law or agreed to in writing, software                                                                                                                                                                                                                                                                                                                                                                                                                        
#   distributed under the License is distributed on an "AS IS" BASIS, WITHOUT                                                                                                                                                                                                                                                                                                                                                                                                                  
#   WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the                                                                                                                                                                                                                                                                                                                                                                                                                   
#   License for the specific language governing permissions and limitations                                                                                                                                                                                                                                                                                                                                                                                                                    
#   under the License.                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
# Source this file in a bash shell to initialize DNAnexus environment                                                                                                                                                                                                                                                                                                                                                                                                                          
# variables:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
# $ source environment                                                                                                                                                                                                                                                                                                                                                                                                                                                                         

# This line is compatible with both bash and csh/tcsh. If running in csh, it sources the file "environment.csh" and quits.                                                                                                                                                                                                                                                                                                                                                                     

test ! -z "$version" && echo "$shell" | grep -q csh && set SOURCE=`echo $_ | cut -f 2 -d " "` && set SOURCE=`dirname "$SOURCE"` && source "$SOURCE/environment.csh" && exit

# Resolve the location of this file                                                                                                                                                                                                                                                                                                                                                                                                                                                            
SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done

# Get RHEL version                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
if [ -e /etc/redhat-release ] ; then
  RHEL_MAJOR_VERSION=$(grep -o "Red Hat Enterprise Linux .* release [0-9]\+" /etc/redhat-release | sed -e "s/Red Hat Enterprise Linux .* release //")
fi

export DNANEXUS_HOME="$( cd -P "$( dirname "$SOURCE" )" && pwd )"

# Detect system installation of dx-toolkit                                                                                                                                                                                                                                                                                                                                                                                                                                                     
if [ "$DNANEXUS_HOME" == "/etc/profile.d" ]; then

  export DNANEXUS_HOME="/usr/share/dnanexus"
  # Private Python packages. We really ought not pollute PYTHONPATH with these though.                                                                                                                                                                                                                                                                                                                                                                                                         
  export PYTHONPATH="/usr/share/dnanexus/lib/python2.7/site-packages:$PYTHONPATH"
  export CLASSPATH="/usr/share/java/dnanexus-api-0.1.0.jar:${CLASSPATH}"

else

  export PATH="${DNANEXUS_HOME}/bin:$PATH"
  export CLASSPATH="${DNANEXUS_HOME}/lib/java/*:${CLASSPATH}"

  if [ "$RHEL_MAJOR_VERSION" == "7" ]; then
    export PYTHONPATH="${DNANEXUS_HOME}/share/dnanexus/lib/python2.7/site-packages:${DNANEXUS_HOME}/lib64/python2.7/site-packages:$PYTHONPATH"
  else
    export PYTHONPATH="${DNANEXUS_HOME}/share/dnanexus/lib/python2.7/site-packages:${DNANEXUS_HOME}/lib/python:$PYTHONPATH"
  fi

fi

# Note: The default I/O stream encoding in Python 2.7 (as configured on ubuntu) is ascii, not UTF-8 or the system locale                                                                                                                                                                                                                                                                                                                                                                       
# encoding. We reset it here to avoid having to set it for every I/O operation explicitly.                                                                                                                                                                                                                                                                                                                                                                                                     
export PYTHONIOENCODING=UTF-8

eval "$(register-python-argcomplete dx|sed 's/-o default//')"

# Clean up old session files                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
(shopt -s nullglob;
 if [[ $HOME != "" ]]; then
     for session_dir in "${HOME}/.dnanexus_config/sessions/"*; do
         if ! ps -p $(basename "$session_dir") &> /dev/null; then
             rm -rf "$session_dir"
         fi
     done
 fi
)

if [[ -z "$DX_SECURITY_CONTEXT" && -n "$DX_AUTH_TOKEN" ]]; then
    export DX_SECURITY_CONTEXT="{\"auth_token_type\":\"Bearer\",\"auth_token\":\"$DX_AUTH_TOKEN\"}"
fi

I have a similar Ubuntu 14.04 VM working, would it be possible to rsync certain folders to it so that the non-working Ubuntu 14.04 is back to normal?

I seem to be stuck at this point, not knowing how to proceed any further.

22
  • 1
    You would need to know what the PYTHONPATH variable was before the change. Nov 30, 2017 at 14:35
  • I don't know how to revert to the list of folders before, but I presume it was a combination of /usr/lib/python2.7 and /usr/local/lib/python2.7. Where can I find examples? What do I do when I know?
    – 719016
    Nov 30, 2017 at 14:39
  • export PYTHON='/usr/lib/python2.7:/usr/local/lib/python2.7' Nov 30, 2017 at 14:41
  • Hi @RamanSailopal, is it PYTHON or PYTHONPATH?
    – 719016
    Nov 30, 2017 at 14:45
  • 1
    Normally you don't need a PYTHONPATH (I don't have one). Looks more like you have broken the requests package. Have you got a directory /usr/lib/python2.7/dist-packages/requests/packages? Otherwise what are the contents of /usr/lib/python2.7/dist-packages/requests/. You may just need to reinstall requests. Btw, on Ubuntu, you can install most popular python packages with apt-get install python-{package}
    – xenoid
    Nov 30, 2017 at 14:58

2 Answers 2

1

The develop mode of setup.py will link your local code path to PYTHONPATH.

If you install the local package by python setup.py develop , which will alter the system PYTHONPATH and drive you crazy to delete it from environment variables.

I find a correct way to uninstall them and restore your default PYTHONPATH from here. You need use the --uninstall or -u option to develop, i.e:

python setup.py develop --uninstall

PS: if you cannot uninstall the package at the first time, please try to reinstall and do uninstall again.

0

I found the only solution is to reinstall the OS from scratch.

3
  • that's brutal :( Dec 8, 2017 at 9:02
  • 1
    Next time exterminate all the python packages, exterminate also the python packages in your home and you will win.
    – peterh
    Dec 8, 2017 at 9:47
  • What do you mean exterminate?
    – 719016
    Dec 8, 2017 at 18:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .