4

I ran this command on Ubuntu 14.04 Linux while trying to install a package:

export PYTHONPATH=/usr/local/lib/python2.7/ && sudo python setup.py install

The package I was trying to install is http://github.com/ewels/MultiQC

Then lots of things python related have gone wrong since this. For example, virtualenv doesn't work anymore, and this package doesn't work either:

http://wiki.dnanexus.com/images/files/dx-toolkit-v0.240.1-ubuntu-14.04-amd64.tar.gz

source Downloads/dx-toolkit/environment
dx login
Traceback (most recent call last):
  File "/home/avilella/Downloads/dx-toolkit/bin/dx", line 7, in <module>
    from dxpy.scripts.dx import main
  File "/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages/dxpy/__init__.py", line 143, in <module>
    from . import exceptions
  File "/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages/dxpy/exceptions.py", line 211, in <module>
    network_exceptions = (requests.packages.urllib3.exceptions.ProtocolError,
AttributeError: 'module' object has no attribute 'packages'

How do I revert this to the stage before running this command-line? Even starting from a fresh bash terminal, it now won't find any of my other python libraries. Even with a freshly created Linux user on the same computer, the errors are the same.

Related question wasn't answered: Fixing a broken PYTHONPATH environment variable

[EDIT]

I don't know how to revert to the list of folders before, but I presume it was a combination of /usr/lib/python2.7 and /usr/local/lib/python2.7

Trying virtualenv also fails now:

virtualenv
Fatal Python error: Py_Initialize: Unable to get the locale encoding
  File "/usr/lib/python2.7/encodings/__init__.py", line 123
    raise CodecRegistryError,\
                            ^
SyntaxError: invalid syntax
Aborted (core dumped)

Interactive python works to the extend I tried below:

avilella@ubuntu14:~$ python
Python 2.7.6 (default, Oct 26 2016, 20:30:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print "foo"
foo
>>> import urllib3
>>> 

I have now gone through the folder and tried both pip install and apt-get install python-${package} with this:

ls /usr/local/lib/python2.7/dist-packages/*/ | grep -v egg | grep '\:' | tr ':' ' ' | grep -v dist-info | xargs -n1 basename  | while read i; do sudo pip install $i; done

Also:

ls /usr/local/lib/python2.7/dist-packages/*/ | grep -v egg | grep '\:' | tr ':' ' ' | grep -v dist-info | xargs -n1 basename  | while read i; do sudo apt-get install -y python-${i}; done

But still same issues.

Package requests is there:

ls -l /usr/lib/python2.7/dist-packages/requests/
total 304
-rw-r--r-- 1 root root 14594 Mar 26  2015 adapters.py
-rw-rw-r-- 1 root root 13750 Sep  4 16:05 adapters.pyc
-rw-r--r-- 1 root root  4333 Sep 24  2013 api.py
-rw-rw-r-- 1 root root  5261 Sep  4 16:05 api.pyc
-rw-r--r-- 1 root root  6173 Dec  5  2013 auth.py
-rw-rw-r-- 1 root root  7206 Sep  4 16:05 auth.pyc
-rw-r--r-- 1 root root   527 Mar 26  2015 certs.py
-rw-rw-r-- 1 root root   736 Sep  4 16:05 certs.pyc
-rw-r--r-- 1 root root  2531 Mar 26  2015 compat.py
-rw-rw-r-- 1 root root  2541 Sep  4 16:05 compat.pyc
-rw-r--r-- 1 root root 16686 Jan  8  2014 cookies.py
-rw-rw-r-- 1 root root 20167 Sep  4 16:05 cookies.pyc
-rw-r--r-- 1 root root  1599 Mar 26  2015 exceptions.py
-rw-rw-r-- 1 root root  3863 Sep  4 16:05 exceptions.pyc
-rw-r--r-- 1 root root   820 Sep 24  2013 hooks.py
-rw-rw-r-- 1 root root  1055 Sep  4 16:05 hooks.pyc
-rw-r--r-- 1 root root  1846 Mar 26  2015 __init__.py
-rw-rw-r-- 1 root root  2501 Sep  4 16:05 __init__.pyc
-rw-r--r-- 1 root root 25332 Mar 26  2015 models.py
-rw-rw-r-- 1 root root 23051 Sep  4 16:05 models.pyc
-rw-r--r-- 1 root root 22042 Mar 26  2015 sessions.py
-rw-rw-r-- 1 root root 18049 Sep  4 16:05 sessions.pyc
-rw-r--r-- 1 root root  3136 Sep 24  2013 status_codes.py
-rw-rw-r-- 1 root root  4439 Sep  4 16:05 status_codes.pyc
-rw-r--r-- 1 root root  3575 Sep 24  2013 structures.py
-rw-rw-r-- 1 root root  6359 Sep  4 16:05 structures.pyc
-rw-r--r-- 1 root root 19642 Mar 26  2015 utils.py
-rw-rw-r-- 1 root root 19386 Sep  4 16:05 utils.pyc

I checked a working version of a different VM I have, where everything python related works, and the requests folder contains exactly the same files.

Any mentions to urllib in the non-working Linux (below) are the same as in the working one (further below), with 2 files changing their date between one an the other:

[       4096 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3
[       3346 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/_collections.pyc
[      22000 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.pyc
[       4096 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/contrib
[       6015 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.pyc
[       6511 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/fields.pyc
[       3162 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/filepost.pyc
[       2074 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/__init__.pyc
[       4096 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/packages
[       9076 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.pyc
[       5850 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/request.pyc
[       9879 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/response.pyc
[      19662 Sep  4 16:05]  /usr/lib/python2.7/dist-packages/urllib3/util.pyc
[      28639 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py
[       5961 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/fields.py
[       2486 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/filepost.py
[      10159 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/response.py
[      20575 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/util.py
[       1703 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/__init__.py
[       3274 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.py
[       2898 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/_collections.py
[       9009 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.py
[       5874 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/request.py
[      17079 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3-1.7.1.egg-info
[       7142 Oct  5 14:47]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.pyc
[       8385 Oct 26  2016]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.py
[      46549 Oct  5 14:47]  /usr/lib/python2.7/urllib2.pyc
[      50626 Oct  5 14:47]  /usr/lib/python2.7/urllib.pyc
[      51788 Oct 26  2016]  /usr/lib/python2.7/urllib2.py
[      59371 Oct 26  2016]  /usr/lib/python2.7/urllib.py

The working one below:

[       4096 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3
[       3346 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/_collections.pyc
[      22000 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.pyc
[       4096 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/contrib
[       6015 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.pyc
[       6511 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/fields.pyc
[       3162 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/filepost.pyc
[       2074 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/__init__.pyc
[       4096 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/packages
[       9076 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.pyc
[       5850 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/request.pyc
[       9879 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/response.pyc
[      19662 Jul 10  9:06]  /usr/lib/python2.7/dist-packages/urllib3/util.pyc
[      28639 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py
[       5961 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/fields.py
[       2486 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/filepost.py
[      10159 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/response.py
[      20575 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3/util.py
[       1703 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/__init__.py
[       3274 Sep 25  2013]  /usr/lib/python2.7/dist-packages/urllib3/exceptions.py
[       2898 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/_collections.py
[       9009 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/poolmanager.py
[       5874 Aug 14  2013]  /usr/lib/python2.7/dist-packages/urllib3/request.py
[      17079 Oct 12  2015]  /usr/lib/python2.7/dist-packages/urllib3-1.7.1.egg-info
[      46549 Jul 12 11:47]  /usr/lib/python2.7/urllib2.pyc
[      50626 Jul 12 11:47]  /usr/lib/python2.7/urllib.pyc
[       7142 Jul 12 11:47]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.pyc
[       8385 Oct 26  2016]  /usr/lib/python2.7/lib2to3/fixes/fix_urllib.py
[      51788 Oct 26  2016]  /usr/lib/python2.7/urllib2.py
[      59371 Oct 26  2016]  /usr/lib/python2.7/urllib.py

Another example tool that doesn't work anymore is tosheets https://github.com/kren1/tosheets:

tosheets
Traceback (most recent call last):
  File "/usr/local/bin/tosheets", line 7, in <module>
    from tosheets.tosheets import main
  File "/usr/local/lib/python2.7/dist-packages/tosheets/tosheets.py", line 27, in <module>
    from apiclient import discovery
ImportError: cannot import name discovery

After doing unset PYTHON, then:

>>> import sys
>>> sys.path
['', '/usr/local/lib/python2.7/dist-packages/multiqc-1.4dev-py2.7.egg', '/usr/lib/python2.7/dist-packages', '/usr/lib/pymodules/python2.7', '/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages', '/home/avilella/Downloads/dx-toolkit/lib/python', '/home/avilella', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/home/avilella/.local/lib/python2.7/site-packages', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client']

unset PYTHON, the python3:

>>> import sys
>>> sys.path
['', '/home/avilella/Downloads/dx-toolkit/share/dnanexus/lib/python2.7/site-packages', '/home/avilella/Downloads/dx-toolkit/lib/python', '/home/avilella', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages']

The contents of $PYTHONPATH before doing any source'ing are empty.

The contents of Downloads/dx-toolkit/environment are below:

# -*- Mode: shell-script -*-                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
# Copyright (C) 2013-2016 DNAnexus, Inc.                                                                                                                                                                                                                                                                                                                                                                                                                                                       
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
# This file is part of dx-toolkit (DNAnexus platform client libraries).                                                                                                                                                                                                                                                                                                                                                                                                                        
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
#   Licensed under the Apache License, Version 2.0 (the "License"); you may not                                                                                                                                                                                                                                                                                                                                                                                                                
#   use this file except in compliance with the License. You may obtain a copy                                                                                                                                                                                                                                                                                                                                                                                                                 
#   of the License at                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
#       http://www.apache.org/licenses/LICENSE-2.0                                                                                                                                                                                                                                                                                                                                                                                                                                             
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
#   Unless required by applicable law or agreed to in writing, software                                                                                                                                                                                                                                                                                                                                                                                                                        
#   distributed under the License is distributed on an "AS IS" BASIS, WITHOUT                                                                                                                                                                                                                                                                                                                                                                                                                  
#   WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the                                                                                                                                                                                                                                                                                                                                                                                                                   
#   License for the specific language governing permissions and limitations                                                                                                                                                                                                                                                                                                                                                                                                                    
#   under the License.                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
#                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
# Source this file in a bash shell to initialize DNAnexus environment                                                                                                                                                                                                                                                                                                                                                                                                                          
# variables:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
# $ source environment                                                                                                                                                                                                                                                                                                                                                                                                                                                                         

# This line is compatible with both bash and csh/tcsh. If running in csh, it sources the file "environment.csh" and quits.                                                                                                                                                                                                                                                                                                                                                                     

test ! -z "$version" && echo "$shell" | grep -q csh && set SOURCE=`echo $_ | cut -f 2 -d " "` && set SOURCE=`dirname "$SOURCE"` && source "$SOURCE/environment.csh" && exit

# Resolve the location of this file                                                                                                                                                                                                                                                                                                                                                                                                                                                            
SOURCE="${BASH_SOURCE[0]}"
while [ -h "$SOURCE" ] ; do SOURCE="$(readlink "$SOURCE")"; done

# Get RHEL version                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
if [ -e /etc/redhat-release ] ; then
  RHEL_MAJOR_VERSION=$(grep -o "Red Hat Enterprise Linux .* release [0-9]\+" /etc/redhat-release | sed -e "s/Red Hat Enterprise Linux .* release //")
fi

export DNANEXUS_HOME="$( cd -P "$( dirname "$SOURCE" )" && pwd )"

# Detect system installation of dx-toolkit                                                                                                                                                                                                                                                                                                                                                                                                                                                     
if [ "$DNANEXUS_HOME" == "/etc/profile.d" ]; then

  export DNANEXUS_HOME="/usr/share/dnanexus"
  # Private Python packages. We really ought not pollute PYTHONPATH with these though.                                                                                                                                                                                                                                                                                                                                                                                                         
  export PYTHONPATH="/usr/share/dnanexus/lib/python2.7/site-packages:$PYTHONPATH"
  export CLASSPATH="/usr/share/java/dnanexus-api-0.1.0.jar:${CLASSPATH}"

else

  export PATH="${DNANEXUS_HOME}/bin:$PATH"
  export CLASSPATH="${DNANEXUS_HOME}/lib/java/*:${CLASSPATH}"

  if [ "$RHEL_MAJOR_VERSION" == "7" ]; then
    export PYTHONPATH="${DNANEXUS_HOME}/share/dnanexus/lib/python2.7/site-packages:${DNANEXUS_HOME}/lib64/python2.7/site-packages:$PYTHONPATH"
  else
    export PYTHONPATH="${DNANEXUS_HOME}/share/dnanexus/lib/python2.7/site-packages:${DNANEXUS_HOME}/lib/python:$PYTHONPATH"
  fi

fi

# Note: The default I/O stream encoding in Python 2.7 (as configured on ubuntu) is ascii, not UTF-8 or the system locale                                                                                                                                                                                                                                                                                                                                                                       
# encoding. We reset it here to avoid having to set it for every I/O operation explicitly.                                                                                                                                                                                                                                                                                                                                                                                                     
export PYTHONIOENCODING=UTF-8

eval "$(register-python-argcomplete dx|sed 's/-o default//')"

# Clean up old session files                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
(shopt -s nullglob;
 if [[ $HOME != "" ]]; then
     for session_dir in "${HOME}/.dnanexus_config/sessions/"*; do
         if ! ps -p $(basename "$session_dir") &> /dev/null; then
             rm -rf "$session_dir"
         fi
     done
 fi
)

if [[ -z "$DX_SECURITY_CONTEXT" && -n "$DX_AUTH_TOKEN" ]]; then
    export DX_SECURITY_CONTEXT="{\"auth_token_type\":\"Bearer\",\"auth_token\":\"$DX_AUTH_TOKEN\"}"
fi

I have a similar Ubuntu 14.04 VM working, would it be possible to rsync certain folders to it so that the non-working Ubuntu 14.04 is back to normal?

I seem to be stuck at this point, not knowing how to proceed any further.

  • 1
    You would need to know what the PYTHONPATH variable was before the change. – Raman Sailopal Nov 30 '17 at 14:35
  • I don't know how to revert to the list of folders before, but I presume it was a combination of /usr/lib/python2.7 and /usr/local/lib/python2.7. Where can I find examples? What do I do when I know? – 719016 Nov 30 '17 at 14:39
  • export PYTHON='/usr/lib/python2.7:/usr/local/lib/python2.7' – Raman Sailopal Nov 30 '17 at 14:41
  • Hi @RamanSailopal, is it PYTHON or PYTHONPATH? – 719016 Nov 30 '17 at 14:45
  • 1
    Normally you don't need a PYTHONPATH (I don't have one). Looks more like you have broken the requests package. Have you got a directory /usr/lib/python2.7/dist-packages/requests/packages? Otherwise what are the contents of /usr/lib/python2.7/dist-packages/requests/. You may just need to reinstall requests. Btw, on Ubuntu, you can install most popular python packages with apt-get install python-{package} – xenoid Nov 30 '17 at 14:58
0

I found the only solution is to reinstall the OS from scratch.

  • that's brutal :( – rajaganesh87 Dec 8 '17 at 9:02
  • 1
    Next time exterminate all the python packages, exterminate also the python packages in your home and you will win. – peterh Dec 8 '17 at 9:47
  • What do you mean exterminate? – 719016 Dec 8 '17 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.