18

The other day I tried installing opencv-git from the AUR with makepkg on Arch Linux. Of course it pulls from the git repository as the name indicates. This pulls 1Gb. I am reading about making a shallow clone with git. When I look at the PKGBUILD file, using grep git PKGBUILD, I see:

pkgname="opencv-git"
makedepends=('git' 'cmake' 'python2-numpy' 'mesa' 'eigen2')
provides=("${pkgname%-git}")
conflicts=("${pkgname%-git}")
source=("${pkgname%-git}::git+http://github.com/Itseez/opencv.git"
    cd "${srcdir}/${pkgname%-git}"
    git describe --long | sed -r 's/([^-]*-g)/r\1/;s/-/./g'
    cd "${srcdir}/${pkgname%-git}"
    cd "${srcdir}/${pkgname%-git}"
    cd "${srcdir}/${pkgname%-git}"
    install -Dm644 "LICENSE" "${pkgdir}/usr/share/licenses/${pkgname%-git}/LICENSE"

Is there a way to modify the recipe or the makepkg command to pull only a shallow clone (the latest version of the source is what I want) and not the full repository to save space and bandwidth? Reading man 5 PKGBUILD doesn't provide the insight I'm looking for. Also looked quickly through the makepkg and pacman manpages - can't seem to find how to do that.

  • "In the end I failed at successfully building the recipe." What did you do exactly, and what went wrong? Provide more details, please. As someone once told me on IRC, unfortunately we left our crystal balls at home. It sounds, reading between the lines, as if the git repos failed to successfully clone, possibly because of network issues? But I'm just guessing. Be explicit, please. – Faheem Mitha Sep 10 '14 at 21:20
  • An educated guess is that you can split the process into two parts. First clone the git repos as a shallow clone or whatever. Then apply the recipe. I'd guess you could replace the network address git+http://github.com/Itseez/opencv.git in the AUR recipe with a local pathname. Have you tried that? If this build system forces you to clone a repos even if it you have it available locally, then it is pretty freaking nutty. – Faheem Mitha Sep 10 '14 at 21:24
  • @FaheemMitha Thank you, I have removed the reference to failing at build - I do not care. I'm looking for an integrated solution that could be based on something like what you describe. I think there might be an option to not download if there is local content... – user44370 Sep 10 '14 at 21:26
  • If your main reason for asking this question is to avoid using unnecessary bandwidth/space, then it would not hurt to say so explicitly. Like I said, try just using the local path - this will probably work by the principle of least surprise. If the option for specifying a shallow clone is not given on the man page, it is possible that functionality is not available. I suggest asking in an appropriate Arch forum, perhaps a mailing list dedicated to that build system. First clarify whether that functionality exists; if not, you could file a wishlist bug. – Faheem Mitha Sep 10 '14 at 21:31
  • 3
    bugs.archlinux.org/task/23065 – jasonwryan Sep 10 '14 at 22:31
13

This can be done by using a custom dlagent. I do not really understand Arch packaging or how the dlagents work, so I only have a hack answer, but it gets the job done.

The idea is to modify the PKGBUILD to use a custom download agent. I modified the source

"${pkgname%-git}::git+http://github.com/Itseez/opencv.git"

into

"${pkgname%-git}::mygit://opencv.git"

and then defined a new dlagent called mygit which does a shallow clone. I did this by adding to the DLAGENTS array in /etc/makepkg.conf the following dlagent:

'mygit::/usr/bin/git clone --depth 1 http://github.com/Itseez/opencv.git'

My guess is you could probably define this download agent somewhere else, but I do not know how. Also notice that the repository that is being cloned is hard coded into the command. Again, this can probably be avoided. Finally, the download location is not what the PKGBUILD expects. To work around this, I simply move the repository after downloading it. I do this by adding

mv "${srcdir}/../mygit:/opencv.git" "${srcdir}/../${pkgname%-git}"

at the beginning of the pkgver function.

I think the cleaner solution would be to figure out what the git+http dlagent is doing and redfine that temporarily. This should avoid all the hack aspects of the solution.

  • Thank you, this works. Yes it would need some work to abstract it to work in other cases than this one. But it's worth investigating and your answer is a valid proof of concept. I've accordingly changed the selected answer to yours. – user44370 Sep 11 '14 at 19:05
11

Personally I modified the makepkg script and it's working like a charm:

# vim `which makepkg` +/clone
...
541         msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git"
542         if ! git clone --mirror "$url" "$dir"; then
543             error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git"
...

Appending "--mirror --single-branch --depth 1" to the "git clone" command:

541         msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git"
542         if ! git clone --mirror --single-branch --depth 1 "$url" "$dir"; then
543             error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git"

Here is a diff view:

--- makepkg ...
+++ makepkg-patched ...
@@ -539,7 +539,7 @@

    if [[ ! -d "$dir" ]] || dir_is_empty "$dir" ; then
        msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git"
-       if ! git clone --mirror "$url" "$dir"; then
+       if ! git clone --mirror --single-branch --depth 1 "$url" "$dir"; then
            error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git"
            plain "$(gettext "Aborting...")"
            exit 1
  • Installing softethervpn-git now downloads 100M only instead of 468M before the hack. – amized May 15 '15 at 15:28
  • Brilliant! Thanks a ton! This should be the default. – ccpizza Oct 8 '15 at 19:11
  • Note: this will fail with PKGBUILDs depending on a tag for instance. See this discussion. However it can probably be fixed by using the fragments there (branch, tag, commit etc.). – BenC Jan 23 '16 at 8:09
  • 7
    Note: now /usr/share/makepkg/source/git.sh should be patched instead – nponeccop Mar 12 '16 at 19:53
6

According to https://bugs.archlinux.org/task/23065 (credit to jasonwryan), adding shallow cloning functionality to AUR PKGBUILD was a wishlist item that was closed on Saturday, 05 March 2011 with the comment:

Reason for closing: Won't implement

This suggests that it is not going to happen unless someone submits a patch.

As I suggested to the poster in comments, what he is trying to do can almost certainly be accomplished by breaking the process up into two steps:

  1. Clone the git repository using shallow clone
  2. Run the PKGBUILD recipe, but point it at the local clone. I am not an Arch User, so don't know whether this is the case, but I'd be very surprised at a package building system that forced users to clone repositories from remote in order to build packages.
  • Thank you. In the comments in the link, there is this discussion about how this modification might have brought unexpected consequences to the script workflow. The issue is how git populates and links missing objects etc. - disk usage would not be a benefit if I understand correctly. Indeed I tried doing a shallow clone in the recipe directory which ended up being 1GB+ the same but that was picked up by makepkg(pkgver complains slightly but still) and works too! – user44370 Sep 11 '14 at 19:26
2

If you make a shallow mirror of the repository in the same directory as the PKGBUILD, you can use makepkg --holdver to prevent makepkg from updating the rest of the repository. This removes the need to modify the PKGBUILD, makepkg.conf, or makepkg itself; however, cloning/updating the repository needs to be done manually.

As an example with cling-git, which would normally clone the entirety of llvm and clang:

$ git clone --mirror --depth=1 --branch=cling-patches http://root.cern.ch/git/llvm.git llvm
$ git clone --mirror --depth=1 --branch=cling-patches http://root.cern.ch/git/clang.git clang
$ git clone --mirror --depth=1 http://root.cern.ch/git/cling.git cling

$ makepkg --holdver

From the makepkg man pages:

--holdver
    When using VCS sources (PKGBUILD(5)) any currently checked out source
    will not be updated to the latest revision.

Note that makepkg will still clone repos that are not already present, which means that I could have omitted manually cloning the cling repository in the above example since it's not so large.

1

If you don't want to modify makepkg scripts.

as outlined here, point DEVELSRCDIR in /etc/yaourtrc or ~/.yaourtrc file to some persistent folder. Then all the repository checkouts (git/svn/...) will happen in this folder. Once repository is cloned, then only quick fetch with newest revisions will be performed instead of full clone each time.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy