6

I would like to create an RPM repository for Fedora packages on my local network. Due to storage limitations, I want the repository to be empty initially and download packages once they are accessed.

Background

I work a lot with local VMs. Anytime I create a new VM and install Fedora, a lot of packages are downloaded from the internet, and most of the downloaded packages are the same. To speed up the process I would like the RPMs to be cached on a server located on the same network.

Similar questions have been answered with a combination of createrepo & reposync. I do not like the reposync part, because I don't want to clone the whole repository up front when I need only some of the packages.

Ideal Solution

I would like the server on my local network to act as an RPM repository for my Fedora installations. It should pass-through the metadata from whatever is configured in /etc/yum.repo.d/*. The server should deliver the requested RPM if it is present in the local cache, or else download it and then deliver it.

A less ambitious approach would be to configure a single RPM repository instead of https://mirrors.fedoraproject.org/... and just use an http proxy.

Update: 02 Nov. 2015

I already have an nginx running on the network, so I played around with a combination of proxy_pass and proxy_cache. It kinda works, but IMHO it has more drawbacks than benefits:

  • a separate configuration for every repo configured in /etc/yum.repo.d/*.
  • can't use metadata from https://mirrors.fedoraproject.org/ because of alternate mirrors.

I dropped the nginx thing and installed squid, as suggested in comments. squid works great for me. With the store_id_program configuration, I am even able to use the alternate mirrors and still hit the cache, no matter where the RPM came from originally.

Improve this question
6
  • 3
    The only thought in my head while reading your Q was "use squid" -- and then you said it
    – Jeff Schaller ♦
    Oct 31, 2015 at 21:08
  • yes, squid is the answer. note that some .rpms are quite large (e.g. kernels can be 30-40MB) so you'll have to increase squid's maximum_object_size from default 4MB to, say, 64MB. You might also want a refresh_pattern .rpm ... rule to keep rpm files longer than the default.
    – cas
    Oct 31, 2015 at 23:31
  • btw, you can also use a squid url-rewriter to match the directory pattern of distro rpm repos and rewrite them to use just the one nearby mirror.
    – cas
    Oct 31, 2015 at 23:33
  • and this LWN article from 2009 is old but still worth reading lwn.net/Articles/318658
    – cas
    Oct 31, 2015 at 23:36
  • and, finally, according to sotechdesign.com.au/… you can use Debian's apt-cacher-ng to cache yum repositories as well as .deb repos....but you have to do it in a debian vm because it probably isn't ported to fedora/rhel/centos/etc
    – cas
    Oct 31, 2015 at 23:39

4 Answers 4

Reset to default
4

Here you can find fine-tuned squid.conf for rpm caching:

https://github.com/spacewalkproject/spacewalk/blob/master/proxy/installer/squid.conf

You just should modify memory and port setting.

Improve this answer
1
2

I'm not sure if you found a good solution for this but I ended up writing a quick little Go program. It acts as a caching proxy and keeps around RPMs, but proxies upstream to database files so it's always correct. It's configured for CentOS, Fedora EPEL, and Arch Linux for now.

You can see it at https://github.com/yobert/remirror

Improve this answer
1

apt-cacher-ng works great. We are using it for some times and it works fine. And remember to only use http mirrors (and not https ones) when using it to make it more efficient (naturally, it can't see & cache https traffic):

To configure fedora official repositories to only use http mirrors, you can add &protocol=http at the end of the metalink config url, e.g. for fedora repository it'll become:

metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch&protocol=http

https://www.unix-ag.uni-kl.de/~bloch/acng/

A caching proxy. Specialized for package files from Linux distributors, primarily for Debian (and Debian based) distributions but not limited to those. See documentation of Apt-Cacher to learn what it's good for.

Improve this answer
3
  • Not sure why I got negative vote, but I've tried and it works fine. Probably the voter assumed that it only works for .deb, but if (s)he've read the link it'd be clear that it works for rpm repos too.
    – hedayat
    Dec 7, 2018 at 8:47
  • »Attempts to add apt-cacher-ng support ended up in pain and the author lost any motivation in further research on this subject.« unix-ag.uni-kl.de/~bloch/acng/html/… It would be very helpful to get detailed instructions (where do I get the package from? it's not in fedora or EPEL it seems) from people who uses this in "production" at the moment.
    – LiveWireBT
    Oct 23, 2021 at 19:43
  • Currently, we've set up apt-cacher-ng under Ubuntu, but it serves both Ubuntu and Fedora. So, I've no experience in setting up apt-cacher-ng under Fedora, although installing from source should work. But if the settings we use for Fedora is required, I can see and post here. About the author, I've emailed him for better Fedora support but have not heard back.
    – hedayat
    Oct 24, 2021 at 20:10
0

Yum wiki contain page with review of caching solutions

Some of it on topic:

  • Install "IntelligentMirror", and register with MirrorManager. 

    Pros
    Zero configuration, on the client side.
    Post setup it mostly just works and everything should be immediate.
    Only downloads what is required by the users.
    Fully automated server side, once setup/working. 
Cons
    Requires setting up a local Squid + Apache-httpd + IntelligentMirror to serve the data.
    If the server is down then metalinks/MM will route you to an external mirror.
    Only intelligently caches packages, not metadata.

  • (beta) avahi-packages-server.py and yum avahi support (see: http://james.fedorapeople.org/yum/avahi)

    Pros
    Zero configuration client.
    Zero configuration server.
    Only downloads what is required by the users. 
Cons
    Not upstream yet (still beta).
    A client that doesn't run the avahi server side doesn't share it's downloads (so all clients need to run the "server").
    Requires working avahi on the network.
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .