4

We've got a Squid web cache set up that is used for caching package downloads, so that all the machines here don't have to independently redownload everything. The installer prompts for a mirror and proxy. As long as every machine uses the exact same mirror (here, http.us.debian.org) and the proxy then it works.

That's somewhat annoying, as it involves going to 'enter information manually' and typing it in each time (as the installer would do ftp.us.debian.org, which Squid doesn't realize is identical).

The installer defaults to just mirror, is there a way to make that work? So I can be lazy and just hit enter a bunch?

  • using just mirror as the hostname will work - IF there is dns set up, and if you use the search option in resolv.conf. This way it should append the search domain to mirror giving you mirror.searchdomain.com at which point regular dns kicks in. Of course, the big final IF is if there is actually a mirror at that hostname... :) 300gb will get you wheezy, jessie, and stretch for both amd64 and i386, can be run off a Pi with an externally powered drive, and easy to set up. – ivanivan Aug 3 '17 at 2:14
  • one thing to note is that squid takes a fair bit of tuning (esp. max file size and retention rules) to be a good cache for large files like .deb packages. apt-cacher-ng as suggested by @RuiFRibeiro is much better at caching .debs. I just run my own mirror with debmirror and point all my debian boxes at that. – cas Aug 3 '17 at 8:13
  • @cas indeed, there is a lot of config for squid. Though squid-deb-proxy will probably do it all for you (and if not, file a bug). – derobert Aug 3 '17 at 17:45
7

For having a local deb cache to server my Debian server farm, I actually prefer to use apt-cacher-ng (caching proxy server for software repositories)

It is a proxy specially APT/deb aware, quite customisable and can cache your deb files for quite a while (configurable).

You install it with:

apt-get install apt-cacher-ng

And by default it caches repositories/debs into /home/apt-cacher-ng. Under this directory, it creates a directory per repository used in your Debian servers, then distros used, much similar to mirror structures. As an added bonus, is also much easier to fetch manually a deb from cache from here, than from a Squid server.

To use it in all your servers, add to the directory /etc/apt/apt.conf.d a file 02proxy with the contents:

Acquire::http { Proxy "http://your_proxy_APT_server:3142"; };

After you add that file, the Debian package manager will proxy all the configured repositories via the configured http APT proxy.

It also got an interesting statics page for consulting it´s activity.

You might also need to open 3142/TCP in your firewalls to allow the servers to talk with your new proxy APT server.

The advantage of such setup is that besides downloading only one deb copy for a bucketload of servers, and saving bandwidth and the public repositories usage is that allows you to update internal servers that do not need to have Internet access (example: DHCP servers).

As documented in Appendix B of the Official Install Guide, you can have your DHCP server give out a preseed file, by adding something like this to it's config:

if substring (option vendor-class-identifier, 0, 3) = "d-i" {
    filename "http://host/preseed.cfg";
}

Then using these preseed options, you can configure the mirror and proxy automatically:

d-i mirror/protocol string http
d-i mirror/country string manual
d-i mirror/http/hostname string http.us.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string http://your_proxy_APT_server:3128/

See also: How to set up Apt caching server on Ubuntu or Debian

  • 1
    I'd second this suggestion, and also add the caution against using approx, which used to be a very nice, easy to set up tool, but the devs I think lost interest, or moved on in life, and it started getting flakier and flakier. apt-cacher-ng I think has a more reliable set of maintainers. But you definitely want to use something like this, it speeds things up massively, I used to use it a lot. It also saves Debian's server bandwidth as well, or whoever mirrors it, which is nice to do. – Lizardx Aug 2 '17 at 19:20
  • This sort of defeats the point of being lazy—it's not really that much different than entering the Squid proxy in manually. (Well, apt-cacher-ng is a bit better than Squid at caching the packages, but Squid is already running here and has many gigabytes of packages already cached. Also used to be used for CentOS as well.) – derobert Aug 2 '17 at 20:07
  • 1
    @derobert Packer /preseeding? I have not installed a Debian machine from scratch for ages at work now... all vms. I also replicated the 02proxy between all the machines I had at the time. – Rui F Ribeiro Aug 2 '17 at 22:06
  • @RuiFRibeiro preseeding would probably make for a good answer. I'm mostly doing VMs too, but unfortunately not just of one release... Mostly Wheezy or later though. – derobert Aug 2 '17 at 22:44
  • @derobert I migrated 90% to 9 by now, a couple of web servers in Jessie (PHP) and the odd machine in 7. – Rui F Ribeiro Aug 3 '17 at 5:47
2

edit: The preseeding approach I edited in to Rui F Ribeiro's answer works with Squid—or any other proxy—as well, and (as long as you're using DHCP) is likely a better approach.


First, mirror will be completed with the system domain name, so add a DNS entry for mirror.yourdomain.com. Point that to a server running Apache; in this example, the same server also runs Squid.

Make sure Apache has both mod_proxy and mod_proxy_http enabled (e.g., a2enmod proxy; a2enmod proxy_http on Debian). Then add a new virtual host:

<VirtualHost *:80>
    ServerName mirror.yourdomain.com
    ServerAlias mirror

    ProxyPass /debian http://http.us.debian.org/debian
    ProxyRemote http http://localhost:3128/
    ProxyVia off
</VirtualHost>

After restarting Apache, package downloads from the Debian mirror "mirror" should go through the cache.

  • I think proxypass does not cache files. – Rui F Ribeiro Aug 2 '17 at 18:53
  • @RuiFRibeiro right, in Apache. But it's passing it to Squid, which is where the cache lives – derobert Aug 2 '17 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.