How can I install more recent versions of software than what Debian provides?

Question

Suppose I want a more recent version of software than is available for my current version of an operating system, what can I do?

Cases to consider:

1. There are semiofficial/official sources of additional packages available for that version of the OS. E.g. backports.org for Debian or PPAs for Ubuntu.
2. There are no more recent versions of the package available for that version of the OS, but there are more recent versions available for more recent versions of the OS. This is the standard case for backporting.
3. There are no packaged versions of more recent versions of the software available. Options available are to package the more recent version.

---

^{Per Let's compile a list of canonical Q&As this is intended as a place to put canonical answers for the following. Answers should probably be made community wiki.}

Answer 1

(If you have questions/comments about this answer, please add a comment. Or, if you have sufficient rep, you can ping me in chat.)

Directly installing binary packages from a newer version of Debian - not the answer.

Suppose you are running some version of a Debian-based distribution. You want a more recent version of a package than is available to you. The first thing that every beginner tries to do it to install the binary package directly on your version of Debian. This may or not work, depending on what version you are running, and how much newer the package is. In general, this procedure will not work well.

Consider for example the case where one is trying to install a binary package from testing/unstable directly on stable. This will most likely not go well, unless testing/unstable happen to very close to stable at that moment. The reason has to do with the nature of a Linux-based binary distribution like Debian. Such operating systems depend heavily on shared libraries, and these dependencies are often very tightly version-dependent; often much more so than necessary. Debian currently does not have a good way of making version dependencies "tight" - a shorthand way of saying that the version dependency is exactly as restrictive as necessary.

What does this mean for the user? Suppose for example that you are trying to install say slrn from Debian unstable to Debian stable. What would this look like?

# apt-get install slrn/unstable
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Selected version '1.0.1-10' (Debian:testing [amd64]) for 'slrn'
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 slrn : Depends: libc6 (>= 2.15) but 2.13-38+deb7u1 is to be installed
E: Unable to correct problems, you have held broken packages.

Despite the error produced by apt, there are no broken packages here. So, what went wrong? The problem is that the version of libc6 that the unstable slrn was compiled against is different (and has a higher version number) than the one available on Debian stable. (libc6 is the GNU C library. The C library is central to any Unix-like operating system, and GNU C library is the version that Linux-based operating systems generally use.)

Therefore the unstable slrn requires a higher numbered version of libc6 than is available for stable. Note that because a package has been compiled against a higher version of library does not necessarily require a higher version of that library, but it is often the case.

The syntax

apt-get install slrn/unstable

means: use the unstable slrn but for all other packages only use the versions from stable. To be more precise, it uses priority numbers. See man apt_preferences for details.

One can also do

apt-get install -t unstable slrn

This is much more likely to work, but you generally don't want to do it. Why?

This means: temporarily treat all packages in unstable on an equal footing with the packages in stable. Therefore this will pull in the unstable slrn's dependencies from unstable if they are of a higher version number, and they generally will be. This will generally include the GNU C library for reasons already explained. Now, this approach will generally "succeed", in that the dependencies will be satisfied by definition (unstable's slrn has dependencies which are satisfied in unstable), but you end up with a mixture of packages that suddenly are being forced to run with versions of libraries different from what they were built for. This will probably not end well.

The answer is... BACKPORTS!

So, what is the correct way to do this? It is to rebuild the Debian sources of more recent versions on your system, popularly known as "backporting". Consider the following cases:

There are semiofficial/official sources of additional packages available for that version of Debian.

The first place to look is Debian Backports, which is the official site for Debian backports.

For a concrete example:

Add the appropriate backports line for your release and update to find the new packages then install something from backports explicitly (because backports are deactivated by default).

echo "deb http://ftp.debian.org/debian stretch-backports main" | sudo tee /etc/apt/sources.list.d/stretch-backports.list
sudo apt-get update
sudo apt-get install -t stretch-backports git

This will get the latest stable version of git which has useful newer features than the stable one included with stretch (e.g. 'include' which allows you to combine multiple config files or change your username for ~/work/projects/ vs ~/personal/projects/).

Another place to look at is the various PPAs by Ubuntu maintainers. You can do a search for "packagename PPA".

There are no more recent versions of the package available for that version of the OS, but there are more recent versions available for more recent versions/releases of the OS. This is the standard case for backporting.

Backporting means that you rebuild the Debian sources from a later version of Debian on the version you are running. This procedure may be easy or involved and difficult depending on the package. Here is an outline of how to do this.

A Brief Backporting Tutorial for Beginners

For concreteness I will assume you are running the current Debian stable, currently wheezy. I'll use the package slrn as an example.

First, note that all the Debian packaging files live in the debian/ subdirectory of the source directory.

The first step is to check whether a more recent version is available. You can do this using apt-cache policy.

apt-cache policy slrn

slrn:
  Installed: 1.0.0~pre18-1.3
  Candidate: 1.0.0~pre18-1.3
  Version table:
     1.0.1-10 0
         50 http://debian.lcs.mit.edu/debian/ testing/main amd64 Packages
         50 http://debian.lcs.mit.edu/debian/ unstable/main amd64 Packages
 *** 1.0.0~pre18-1.3 0
        500 http://debian.lcs.mit.edu/debian/ wheezy/main amd64 Packages
        100 /var/lib/dpkg/status
     1.0.0~pre18-1.1 0
        500 http://debian.lcs.mit.edu/debian/ squeeze/main amd64 Packages

We would like to backport 1.0.1-10.

STEP 1:

NB: Make sure that the deb-src lines for the source version you want to download appears in your /etc/apt/sources.list. For example, if you want to download the unstable version of slrn, you need the deb-src line for unstable, or it won't work. Note that you don't need the corresponding deb lines to download the sources, though apt-cache policy uses that information, so if you don't have the corresponding deb lines, then apt-cache policy won't show you the relevant version(s). If you do have the deb lines, don't forget to pin the newer versions using an entry in /etc/apt/preferences or similar. An entry in /etc/apt/preferences like this (for unstable) will work, for example.

Package: *
Pin: release a=unstable
Pin-Priority: 50

If you add lines in /etc/apt/sources.list, don't forget to run apt-get update afterwards.

Download the sources for slrn. A good place is /usr/local/src/slrn.

apt-get source slrn=1.0.1-10

STEP 2:

Change the version number slightly, so as to distinguish your backport from the upstream version. Run dch --bpo, which will automatically add an entry to the debian/changelog file with an appropriate version number, for example

slrn (1.0.1-10~bpo10+1) UNRELEASED; urgency=low

  * Backport to buster.

 -- User <user@domain>  Sun, 02 Feb 2014 23:54:13 +0530

STEP 3:

Attempt to build the sources. If the packages required for the build are not available, then the attempt will fail. Change directory into the source directory. Use debuild from the devtools package.

cd slrn-1.0.1/
debuild -uc -us

If the build dependencies are satisfied, then the sources will build and produce some debs at the level above the source directory; in this case /usr/local/src/slrn.

STEP 4:

Suppose the build dependencies are not satisfied. Then you need to try to install the build dependencies. This may or may not work, as the dependencies may not be available for your version, or if available, may not be available in the right version.

NB: It is unfortunately not uncommon for Debian packages to require versions of build dependencies that are higher than necessary. There is no automated way in Debian to check this, and often package maintainers don't care as long as it works on the corresponding version/release. Therefore, take a skeptical attitude to dependency versions, and use common sense. For example, widely used packages like Python and the GNU tools will not depend on very specific versions of their dependencies, regardless what the Debian packager lists.

In any case, you can try to install them doing

apt-get build-dep slrn=1.0.1-10

If this succeeds, then try building the package again (STEP 2). If it fails, then further work is needed. Note that debuild looks at the Build Dependencies in the debian/control file, and you can change these if necessary. So let us talk about that now. Here are the Build Dependencies for slrn.

Build-Depends: debhelper (>=9), libslang2-dev, libuu-dev,
 exim4 | mail-transport-agent, libgnutls-openssl-dev, po-debconf, autoconf,
 libcanlock2-dev, autotools-dev, dpkg-dev (>= 1.16.0), chrpath, dh-autoreconf, inn2-inews

An alternative to using apt-get build-dep is to install these manually, by doing

apt-get install debhelper libslang2-dev ...

If you start changing these values in the control file, then you should switch to a manual installation, as then apt-get build-dep will no longer be doing the right thing.

There are no packaged versions of more recent versions of the software available. Options available are to package the more recent version.

In many cases, one can reuse the packaging from earlier versions of the software in conjunction with newer sources. This approach can run into problems, notably patches that applied to earlier versions of the software may not apply here, so one may need to resync them with the sources. The 3.0 (quilt) source format which is now becoming standard uses quilt, and patches are located in the debian/patches directory.

However, a detailed discussion of these issues is out of scope for this post.

Answer 2

One way that always works, not only in Debian, is compiling the necessary software yourself. (I've been doing this for years now on Debian, both when I needed a newer version that was available, and when the software was not provided at all).

I store locally compiled packages in /use/local using stow, which allows me to store all files related to a package in a subdirectory tree, and then to make symlinks to that tree. That makes managing compiled packages easy: Installed files don't collide with Debian-provided files, and I can remove a package with one command.

The steps to compile and install a package, say some_software, are usually a variant of the following:

1. Download .tar file etc. into /usr/local/src/.

2. Create a file /usr/local/packages/some_software that describes where I downloaded the software, what it does, what version it is, and that contains notes what I had to do to make it compile (see below).

3. Unpack contents of .tar file into /usr/local/tmp/some_software.

4. Alternatively, if compiling from a repository, check out the repository in a suitable subdirectory (e.g. /usr/local/git/some_software), and compile there,

5. cd into this directory, look at README, INSTALL etc.

6. In most cases, there's an autotools script to configure the package. Call with ./configure --prefix /usr/local/stow/some_software-version so files get installed in this subdirectory. Otherwise, read Makefile and figure out how to set the path for installed files.

7. Compile with make.

8. Install with make install.

9. cd /usr/local/stow, then stow some_software-version

10. Test if it works.

Often compiling doesn't work in the first attempt: The package may need libraries and the corresponding include files, so one has to install the correct ...-dev packages from Debian. Or gcc may complain about stuff that a different version of gcc which the author of the package used doesn't complain about. So sometimes one has to read code, and fix these, but that doesn't happen too often.