It seems I haven't removed the older version of GDAL. How should I do that?
make uninstall is a reasonable option, if you have kept the sources, which is good practice if you are doing local installs.
A simple alternative is to remove all files relating to this library, both old and new and then reinstall the new one.
However, I have to wonder why you are installing from source when the package is available on Debian, and therefore on Ubuntu. If you are on an older release/version of Ubuntu, and there are no suitable PPAs available, note that backporting from a newer release/version of Ubuntu is possible, and much of the time not difficult.
The listed build dependencies of GDAL 1.11 in Debian are
Build-Depends: debhelper (>= 9), dh-autoreconf, zlib1g-dev, libnetcdf-dev (>=
1:4.0.0), netcdf-bin, libjasper-dev, libpng-dev,
libjpeg-dev, libgif-dev, libwebp-dev, libhdf4-alt-dev [!ppc64],
libhdf5-dev (>= 1.8.8), libpcre3-dev, libpq-dev, libxerces-c-dev,
unixodbc-dev (>= 2.2.11), doxygen, d-shlibs, libgeos-dev,
libmysqlclient-dev, dh-python, python-all-dev (>= 2.6.6-3~),
python-numpy, python3-all-dev, python3-numpy, libcurl4-gnutls-dev |
libcurl-ssl-dev, libsqlite3-dev, libogdi3.2-dev, chrpath, swig, patch,
libexpat1-dev, libproj-dev, libdap-dev, libxml2-dev, libspatialite-dev
(>= 2.4.0~rc2-4), libepsilon-dev (>= 0.9.1-1~),
libpoppler-private-dev, liblzma-dev, libopenjpeg-dev,
libarmadillo-dev, libfreexl-dev (>= 1.0.0), libkml-dev (>= 1.3.0~),
liburiparser-dev, default-jdk, ant
which doesn't look too bad. To be clear, the idea here is to rebuild Debian/Ubuntu sources on your release. If you want tips or more details, I can elaborate.
Here are the versions of GDAL available in Debian.
$ apt-cache policy libgdal1-dev
1 http://debian.lcs.mit.edu/debian/ experimental/main amd64 Packages
50 http://debian.lcs.mit.edu/debian/ testing/main amd64 Packages
50 http://debian.lcs.mit.edu/debian/ unstable/main amd64 Packages
500 http://debian.lcs.mit.edu/debian/ wheezy/main amd64 Packages
500 http://debian.lcs.mit.edu/debian/ squeeze/main amd64 Packages