15

General question:

Could some explain what the command apt-get update does and when I really should use it?


Remarks

Please give a detailed answer. Not just a copy of the man page, unless your version is really detailed (I put one definition from man page below).

apt-get update: Used to re-synchronize the package index files from their sources. The indexes of available packages are fetched from the location(s) specified in /etc/apt/sources.list(5). An update should always be performed before an upgrade or dist-upgrade.


Sub-questions:

  • Where is stored the package index? On a database? On a file?
  • What happens if I do apt-get install without updating the cache? Is there a chance that the remote package would not exist anymore and that the link would be broken?
  • Is there some agreed politic about deb repositories? For example, should a repository only contains the last version of a package, or on the contrary should it contains all versions available for a specific distribution release?

Context

I ask my question because I am studying the Docker framework. One of its feature is the Dockerfile, which allows you to build a sort of OS image by executing some instruction from this file. One property of this image is that it should always be the same, whatever the context is (time of build, etc).

I'm afraid that if I launch apt-get update command at different time, the result would be different and so my images would be different.

  • I think this post could serve as a wiki article for how to ask a high level question. Very useful. – Zerodf Jan 6 '18 at 13:19
12

apt-get update downloads the list of available packages.

The list of packages can change over time. New packages are added, and old packages are removed. Thus if you have a really old cache, and you try to do an apt-get install, it might try to download a package which no longer exists.
How long an old package is kept in a repository is up to the repo maintainer (your distribution). As such, if you're using something like docker, where the cache might be very out of date, you should always run apt-get update before installing any packages.

The reason for removing and adding packages is mostly bug fixes & security updates. Though if you're using 3rd party repos like PPA, anything goes.

When using something like docker for containerization in a corporate environment, you should build the container once, and then move that container through your various release environments (development, staging, production), and not rebuild the container each time. This will make sure you don't get a different container that hasn't been tested.

To answer your question of where the cache files live, /var/lib/apt/lists.

  • Great answer! Thank you! I want to react for the paragraph "(...) not rebuild the container each time. This will make sure you don't get a different container that hasn't been tested." I have read that a best practice is to never use apt-get upgrade. One of the reason would be: "It also produces inconsistent images because you no longer have one source of truth of how your application should run and what versions of dependencies are included in the image." Isn't it the same problem with apt-get update then? And Dockerfile aren't supposed to guarantee the image ? – Pierre-Jean May 19 '14 at 19:56
  • 2
    Kinda. apt-get update will only affect newly installed packages. Existing packages will only be upgraded if needed by the new packages (this should be minimal). With apt-get upgrade you upgrade all packages, including the existing ones, resulting in a vastly different image. While this can result in a different result each time you build from the dockerfile, I don't personally think this is a serious issue if you go through a multi-environment release. I think this is more of an issue if you distribute the dockerfile to other people and have them build it. – Patrick May 19 '14 at 21:58
0

Could some explain what the command apt-get update does and when I really should use it?

apt-get update downloads updated indexes from the distribution's package repositories, listing all available packages and their precise versions.

Common distributions like Ubuntu and Debian are usually conservative and backward compatible in their package offerings, so the versions won't change very much over time; they will change due to security updates or bug fixes. For example, mysql could be upgraded from 5.7.18 to 5.7.19 but not to 6.x.

Where is stored the package index? On a database? On a file?

It's usually stored in one or more files inside /var/lib/apt. In the context of Docker, these files are inside the image. When building the Dockerfile, they get stored in the new layers of filesystem that get created and persisted as the newly built image.

What happens if I do apt-get install without updating the cache?

You may try to download package versions that do not exist anymore. This is quite common on virtual machines, but it is possible inside containers too if the distribution repositories have released new packages after the base image was built. There may be no coordination between the distribution maintainers and the Dockerfile maintainers, which are downstream from the distribution and may be larger in number. There is only one Debian repository but thousands of jessie-based container images and Dockerfile.

Moreover, some upstream images like the ubuntu one remove the downloaded index to make the image smaller and avoid outdated files there. So it's expected an updated index should be downloaded when building on top of a base image, not for each version of a base image to ship with the latest index.

Is there a chance that the remote package would not exist anymore and that the link would be broken?

Definitely, because the versions stored in the index are very precise like 5.7.19 (simplification; they are more similar to 5.7.19-0ubuntu1).

Is there some agreed politic about deb repositories? For example, should a repository only contains the last version of a package, or on the contrary should it contains all versions available for a specific distribution release?

It's common for old minor versions to be removed quickly once an update is available; I assume this is to save space on the servers as binaries can weigh several tens of megabytes, multiplied by all the versions and architectures supported. So it is usually impossible to pin, say, mysql-5.7.18 in the subsequent apt-get install; as soon as mysql-5.7.19 is released in the distribution the previous one will get removed.

To be fair to Docker, this non-determinism of apt-get update is an issue that is brought over as part of the package management of each distribution. You would have the same problem trying to build repeatably an EC2 or Vagrant virtual machine.

Some system administrators use services such as Aptly to mirror the original repositories and be able to pin a particular version, but you run the risk of missing out on security updates unless you have a frequently-run separate process for testing the updates and changing what you are pinning.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.