2

With no previous experience I have just set up a squid proxy. Its only purpose is to prevent that all the Linux systems (openSUSE) download their updates from the Internet.

A proxy can easily be configured for the openSUSE update tool. tcpdump shows that the other systems access the proxy and the proxy accesses the repositories.

What I don't understand: After a distro upgrade the first online update is supposed to be 400 MiB to 700 MiB (that's what YaST says before starting the update). But the squid storage directory contains only about 80 MiB data.

The access.log shows both hits and misses. Even if for some strange reasons the Linux systems would access different servers (Is there some kind of DNS-intransparent redirection?) then I would expect the proxy storage to be at least as big as the biggest download.

The repositories are http, not https thus there should not be a problem for a proxy.

problem found

This link in this answer provides the information why squid does not work well here:

packages are downloaded using segmented downloading spread over multiple mirrors, which together makes it impossible for squid to do much caching.

2

I found this tutorial: How to cache openSUSE repositories with Squid.

excerpt

How to make your local Squid web cache work with openSUSE repositories and the openSUSE network installation process. In effect, how to run a fully autonomous, local on-demand repository mirror. Even with a high-speed ADSL internet connection, savings of up to 60% are easily achieved.

Below are the step by step details from that article.

Step by step

  1. Squid

    The Squid web-proxy is the key element in this setup, so a working Squid installation is prerequisite. Setting up Squid is not as complicated as it may appear, but you'll have to consult squid documentation, it's outside the scope of this article. Whether you prefer directing access using environment variables http_proxy et al, or if you run a transparent proxy (like I do), is not really important.

    Note: the setup here works for Squid 2.7, I don't think the storeurl_rewrite feature has been implemented in Squid 3.x yet.

  2. jesred

    jesred is the URL rewriter. It's fairly mature, but fully functional. (original webpage). I had to make a couple of changes to make it fully compatible with squid 2.7:

    For the moment, it does not come packaged, you'll have to build it from scratch:

    tar xzvf <tarball>
    cd jesred-1.3
    make
    

    Installation: when you're done, copy the binary jesred into /usr/local/bin or whatever your preferred location for your own binaries is.

    The config file for jesred: /etc/squid/jesred.conf

    allow = /etc/squid/redirector.acl
    rules = /etc/squid/opensuse-redirect.rules
    redirect_log = /var/log/squid/redirect.log
    rewrite_log = /var/log/squid/rewrite.log
    

    Using /etc/squid/redirector.acl you can control which clients' requests the rewriter should process, but I find this is actually easier to control with Squid's ACL and storeurl_access directive, so I enable for all clients:

    # rewrite all URLs from
    0.0.0.0/0
    
  3. /etc/squid/squid.conf

    Configuration: add the following lines to /etc/squid/squid.conf storeurl_rewrite_program /usr/bin/jesred storeurl_rewrite_children 5

    acl metalink req_mime_type application/metalink4+xml
    storeurl_access deny metalink 
    
    storeurl_access allow localnet
    storeurl_access allow localhost
    
    acl localhost src 127.0.0.0/8
    acl localnet src 192.168.0.0/16
    
  4. fetcher206 logfile

    Amend /etc/squid/squid./conf as follows:

    logformat f206 %{%Y-%m-%dT%H:%M:%S}tl %Ss/%03Hs %rm %ru %mt
    access_log /var/log/squid/fetch206.log f206
    

    This log will be read by fetcher206.

    To prevent it growing too big, add the following to /etc/logrotate.d/:

    /var/log/squid/fetch206.log {
       compress
       dateext
       maxage 365
       rotate 5
       size=+4M
       notifempty
       missingok
       create 640 squid root
       sharedscripts
       postrotate
        /etc/init.d/squid reload
       endscript
    }
    
  5. squid delay pool

    This is an optional step - depending on your available downstream bandwidth, you may want to restrict what is used by fetcher206 for retrieving the repository files. This prevents

    • slowing down the current installation and
    • abuse of the internet connection

    delay_pools 1
    delay_class 1 1
    delay_access 1 allow localhost
    delay_parameters 1 1000000/1000000
    

    Add the above to /etc/squid/squid.conf - it defines one delay_pool, only accessible from localhost (which is where fetcher206 will be running wget) with a maximum bandwidth of 1MByte/sec.

    If you have other http/proxy traffic originating from localhost, you could just add another 127.0.0.x address, and use that specifically for fetcher206.

  6. mirror database

    We need a current list of the available openSUSE mirrors. This can be retrieved from mirrors.opensuse.org. For the time being, I use XSL to parse the HTML page, but I hope to move to a suitably formatted list direct from MirrorBrain.

    mkdir -p /var/lib/fetcher206
    cp tarball/Makefile.mirrors /var/lib/fetcher206/Makefile
    cp tarball/extract* /var/lib/fetcher206/
    make -C /var/lib/fetcher206
    cp tarball/opensuse_mirrors.cron /etc/cron.d/opensuse_mirrors
    
  7. reload squid

    When you've come this far, it's time to reload squid with

    squid -k reconfigure
    
  8. fetcher206

    fetcher206 is, for the time being, a PHP script. Install it by simply copying it into /usr/bin. It has a few hard-coded options, such as number of wgets to run concurrently, name of logfile etc.

    fetcher206 does not yet have a systemd service unit, nor an LSB init-script. For the time being, you simply start it with:

    startproc -s -q /usr/bin/fetcher206
    

References

  • I just realize that this explains the cache misses but not the small amount of cached data. – Hauke Laging Apr 22 '14 at 3:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.