50

I found a collection of slackbuilds, some i need there are on GitHub. https://github.com/PhantomX/slackbuilds/ I don't want to get all git.

git clone https://github.com/PhantomX/slackbuilds.git

But only get a slackbuild, for this one.

How to do this? Is it possible?

1
40

You will end up downloading the entire history, so I don't see much benefit in it, but you can checkout specific parts using a "sparse" checkout. Quoting this Stack Overflow post:

The steps to do a sparse clone are as follows:

mkdir <repo>
cd <repo>
git init
git remote add -f origin <url>

I'm going to interrupt here. Since I'm quoting another post, I don't want to edit the quoted parts, but do not use -f with git remote add. It will do a fetch, which will pull in the entire history. Just add the remote without a fetch:

git remote add origin <url>

And then do a shallow fetch like described later.

This creates an empty repository with your remote, and fetches all objects but doesn't check them out. Then do:

git config core.sparseCheckout true

Now you need to define which files/folders you want to actually check out. This is done by listing them in .git/info/sparse-checkout, eg:

echo "some/dir/" >> .git/info/sparse-checkout
echo "another/sub/tree" >> .git/info/sparse-checkout

[...]

You might want to have a look at the extended tutorial and you should probably read the official documentation for sparse checkout.

You might be better off using a shallow clone. Instead of a normal git pull, try:

git pull --depth=1 origin master

I had an occasion to test this again recently, trying to get only the Ubuntu Mono Powerline fonts. The steps above ended up downloading some 11 MB, where the Ubuntu Fonts themselves are ~900 KB:

% git pull --depth=1 origin master
remote: Enumerating objects: 310, done.
remote: Counting objects: 100% (310/310), done.
remote: Compressing objects: 100% (236/236), done.
remote: Total 310 (delta 75), reused 260 (delta 71), pack-reused 0
Receiving objects: 100% (310/310), 10.40 MiB | 3.25 MiB/s, done.
Resolving deltas: 100% (75/75), done.
From https://github.com/powerline/fonts
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
% du -hxd1 .
11M     ./.git
824K    ./UbuntuMono
12M     .

A normal clone took about 20 MB. There's some savings, but not enough.

Using the --filter + checkout method in Ciro Santilli's answer really cuts down the size, but as mentioned there, downloads each blob one by one, which is slow:

% git fetch --depth=1 --filter=blob:none
remote: Enumerating objects: 52, done.
remote: Counting objects: 100% (52/52), done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 52 (delta 1), reused 35 (delta 1), pack-reused 0
Receiving objects: 100% (52/52), 14.55 KiB | 1.32 MiB/s, done.
Resolving deltas: 100% (1/1), done.
From https://github.com/powerline/fonts
 * [new branch]      master     -> origin/master
 * [new branch]      terminus   -> origin/terminus
% git checkout origin/master -- UbuntuMono
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 1.98 KiB | 1.98 MiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 581 bytes | 581.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 121.43 KiB | 609.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 100.66 KiB | 512.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 107.62 KiB | 583.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 112.15 KiB | 791.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 454 bytes | 454.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 468 bytes | 468.00 KiB/s, done.
% du -hxd1 .
692K    ./.git
824K    ./UbuntuMono
1.5M    .

TL;DR: Use all of --filter, sparse checkout and shallow clone to reduce the total download, or only use sparse checkout + shallow clone if you don't care about the total download and just want that one directory however it may be obtained.

5
  • 1
    I am getting error: Sparse checkout leaves no entry on working directory after doing a git pull after making the modification to core.sparseCheckout and .git/info/sparse-checkout – Sandeepan Nath Aug 7 '17 at 12:00
  • Correction: git init; git remote ... just set up things, nothing is downloaded. The whole point of sparse is to get only objects you are interested in. – vonbrand Feb 22 '20 at 19:08
  • @vonbrand that's what the -f option is for: "With -f option, git fetch <name> is run immediately after the remote information is set up." – muru Feb 23 '20 at 16:05
  • 1
    After git remote add -f origin <url> git is still downloading the whole repo. In my case, I don't want to spend time waiting for that, even if it doesn't checkout the entire repo. – Fernando Ortega Apr 28 '20 at 20:01
  • In sparse checkout we define what we want to check out but is there a way to exclude a few directories and check out the rest? – Ram Patra Jun 14 '20 at 11:24
31

git clone --filter from git 2.19 now works on GitHub (tested 2021-01-14, git 2.30.0)

This option was added together with an update to the remote protocol, and it truly prevents objects from being downloaded from the server.

E.g., to clone only objects required for d1 of this minimal test repository: https://github.com/cirosantilli/test-git-partial-clone I can do:

git clone \
  --depth 1  \
  --filter=blob:none  \
  --sparse \
  https://github.com/cirosantilli/test-git-partial-clone \
;
cd test-git-partial-clone
git sparse-checkout init --cone
git sparse-checkout set d1

Here's a less minimal and more realistic version at https://github.com/cirosantilli/test-git-partial-clone-big-small

git clone \
  --depth 1  \
  --filter=blob:none  \
  --sparse \
  https://github.com/cirosantilli/test-git-partial-clone-big-small \
;
cd test-git-partial-clone
git sparse-checkout init --cone
git sparse-checkout set small

That repository contains:

  • a big directory with 10 10MB files
  • a small directory with 1000 files of size one byte

All contents are pseudo-random and therefore incompressible.

Clone times on my 36.4 Mbps internet:

  • full: 24s
  • partial: "instantaneous"

The sparse-checkout part is also needed unfortunately. You can also only download certain files with the much more understandable:

git clone \
  --depth 1  \
  --filter=blob:none  \
  --no-checkout \
  https://github.com/cirosantilli/test-git-partial-clone \
;
cd test-git-partial-clone
git checkout master -- di

but that method for some reason downloads files one by one very slowly, making it unusable unless you have very few files in the directory.

Analysis of the objects in the minimal repository

The clone command obtains only:

  • a single commit object with the tip of the master branch
  • all 4 tree objects of the repository:
    • toplevel directory of commit
    • the the three directories d1, d2, master

Then, the git sparse-checkout set command fetches only the missing blobs (files) from the server:

  • d1/a
  • d1/b

Even better, later on GitHub will likely start supporting:

  --filter=blob:none \
  --filter=tree:0 \

where --filter=tree:0 from Git 2.20 will prevent the unnecessary clone fetch of all tree objects, and allow it to be deferred to checkout. But on my 2020-09-18 test that fails with:

fatal: invalid filter-spec 'combine:blob:none+tree:0'

presumably because the --filter=combine: composite filter (added in Git 2.24, implied by multiple --filter) is not yet implemented.

I observed which objects were fetched with:

git verify-pack -v .git/objects/pack/*.pack

as mentioned at: https://stackoverflow.com/questions/7348698/git-how-to-list-all-objects-in-the-database/18793029#18793029 It does not give me a super clear indication of what each object is exactly, but it does say the type of each object (commit, tree, blob), and since there are so few objects in that minimal repo, I can unambiguously deduce what each object is.

git rev-list --objects --all did produce clearer output with paths for tree/blobs, but it unfortunately fetches some objects when I run it, which makes it hard to determine what was fetched when, let me know if anyone has a better command.

TODO find GitHub announcement that saying when they started supporting it. https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/ from 2020-01-17 already mentions --filter blob:none.

git sparse-checkout

I think this command is meant to manage a settings file that says "I only care about these subtrees" so that future commands will only affect those subtrees. But it is a bit hard to be sure because the current documentation is a bit... sparse ;-)

It does not, by itself, prevent the fetching of blobs.

If this understanding is correct, then this would be a good complement to git clone --filter described above, as it would prevent unintentional fetching of more objects if you intend to do git operations in the partial cloned repo.

When I tried on Git 2.25.1:

git clone \
  --depth 1 \
  --filter=blob:none \
  --no-checkout \
  https://github.com/cirosantilli/test-git-partial-clone \
;
cd test-git-partial-clone
git sparse-checkout init

it didn't work because the init actually fetched all objects.

However, in Git 2.28 it didn't fetch the objects as desired. But then if I do:

git sparse-checkout set d1

d1 is not fetched and checked out, even though this explicitly says it should: https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/#sparse-checkout-and-partial-clones With disclaimer:

Keep an eye out for the partial clone feature to become generally available[1].

[1]: GitHub is still evaluating this feature internally while it’s enabled on a select few repositories (including the example used in this post). As the feature stabilizes and matures, we’ll keep you updated with its progress.

So yeah, it's just too hard to be certain at the moment, thanks in part to the joys of GitHub being closed source. But let's keep an eye on it.

Command breakdown

The server should be configured with:

git config --local uploadpack.allowfilter 1
git config --local uploadpack.allowanysha1inwant 1

Command breakdown:

The format of --filter is documented on man git-rev-list.

Docs on Git tree:

Test it out locally

The following script reproducibly generates the https://github.com/cirosantilli/test-git-partial-clone repository locally, does a local clone, and observes what was cloned:

#!/usr/bin/env bash
set -eu

list-objects() (
  git rev-list --all --objects
  echo "master commit SHA: $(git log -1 --format="%H")"
  echo "mybranch commit SHA: $(git log -1 --format="%H")"
  git ls-tree master
  git ls-tree mybranch | grep mybranch
  git ls-tree master~ | grep root
)

# Reproducibility.
export GIT_COMMITTER_NAME='a'
export GIT_COMMITTER_EMAIL='a'
export GIT_AUTHOR_NAME='a'
export GIT_AUTHOR_EMAIL='a'
export GIT_COMMITTER_DATE='2000-01-01T00:00:00+0000'
export GIT_AUTHOR_DATE='2000-01-01T00:00:00+0000'

rm -rf server_repo local_repo
mkdir server_repo
cd server_repo

# Create repo.
git init --quiet
git config --local uploadpack.allowfilter 1
git config --local uploadpack.allowanysha1inwant 1

# First commit.
# Directories present in all branches.
mkdir d1 d2
printf 'd1/a' > ./d1/a
printf 'd1/b' > ./d1/b
printf 'd2/a' > ./d2/a
printf 'd2/b' > ./d2/b
# Present only in root.
mkdir 'root'
printf 'root' > ./root/root
git add .
git commit -m 'root' --quiet

# Second commit only on master.
git rm --quiet -r ./root
mkdir 'master'
printf 'master' > ./master/master
git add .
git commit -m 'master commit' --quiet

# Second commit only on mybranch.
git checkout -b mybranch --quiet master~
git rm --quiet -r ./root
mkdir 'mybranch'
printf 'mybranch' > ./mybranch/mybranch
git add .
git commit -m 'mybranch commit' --quiet

echo "# List and identify all objects"
list-objects
echo

# Restore master.
git checkout --quiet master
cd ..

# Clone. Don't checkout for now, only .git/ dir.
git clone --depth 1 --quiet --no-checkout --filter=blob:none "file://$(pwd)/server_repo" local_repo
cd local_repo

# List missing objects from master.
echo "# Missing objects after --no-checkout"
git rev-list --all --quiet --objects --missing=print
echo

echo "# Git checkout fails without internet"
mv ../server_repo ../server_repo.off
! git checkout master
echo

echo "# Git checkout fetches the missing directory from internet"
mv ../server_repo.off ../server_repo
git checkout master -- d1/
echo

echo "# Missing objects after checking out d1"
git rev-list --all --quiet --objects --missing=print

GitHub upstream.

Output in Git v2.19.0:

# List and identify all objects
c6fcdfaf2b1462f809aecdad83a186eeec00f9c1
fc5e97944480982cfc180a6d6634699921ee63ec
7251a83be9a03161acde7b71a8fda9be19f47128
62d67bce3c672fe2b9065f372726a11e57bade7e
b64bf435a3e54c5208a1b70b7bcb0fc627463a75 d1
308150e8fddde043f3dbbb8573abb6af1df96e63 d1/a
f70a17f51b7b30fec48a32e4f19ac15e261fd1a4 d1/b
84de03c312dc741d0f2a66df7b2f168d823e122a d2
0975df9b39e23c15f63db194df7f45c76528bccb d2/a
41484c13520fcbb6e7243a26fdb1fc9405c08520 d2/b
7d5230379e4652f1b1da7ed1e78e0b8253e03ba3 master
8b25206ff90e9432f6f1a8600f87a7bd695a24af master/master
ef29f15c9a7c5417944cc09711b6a9ee51b01d89
19f7a4ca4a038aff89d803f017f76d2b66063043 mybranch
1b671b190e293aa091239b8b5e8c149411d00523 mybranch/mybranch
c3760bb1a0ece87cdbaf9a563c77a45e30a4e30e
a0234da53ec608b54813b4271fbf00ba5318b99f root
93ca1422a8da0a9effc465eccbcb17e23015542d root/root
master commit SHA: fc5e97944480982cfc180a6d6634699921ee63ec
mybranch commit SHA: fc5e97944480982cfc180a6d6634699921ee63ec
040000 tree b64bf435a3e54c5208a1b70b7bcb0fc627463a75    d1
040000 tree 84de03c312dc741d0f2a66df7b2f168d823e122a    d2
040000 tree 7d5230379e4652f1b1da7ed1e78e0b8253e03ba3    master
040000 tree 19f7a4ca4a038aff89d803f017f76d2b66063043    mybranch
040000 tree a0234da53ec608b54813b4271fbf00ba5318b99f    root

# Missing objects after --no-checkout
?f70a17f51b7b30fec48a32e4f19ac15e261fd1a4
?8b25206ff90e9432f6f1a8600f87a7bd695a24af
?41484c13520fcbb6e7243a26fdb1fc9405c08520
?0975df9b39e23c15f63db194df7f45c76528bccb
?308150e8fddde043f3dbbb8573abb6af1df96e63

# Git checkout fails without internet
fatal: '/home/ciro/bak/git/test-git-web-interface/other-test-repos/partial-clone.tmp/server_repo' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

# Git checkout fetches the missing directory from internet
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1/1), 45 bytes | 45.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0)
Receiving objects: 100% (1/1), 45 bytes | 45.00 KiB/s, done.

# Missing objects after checking out d1
?8b25206ff90e9432f6f1a8600f87a7bd695a24af
?41484c13520fcbb6e7243a26fdb1fc9405c08520
?0975df9b39e23c15f63db194df7f45c76528bccb

Conclusions: all blobs from outside of d1/ are missing. E.g. 0975df9b39e23c15f63db194df7f45c76528bccb, which is d2/b is not there after checking out d1/a.

Note that root/root and mybranch/mybranch are also missing, but --depth 1 hides that from the list of missing files. If you remove --depth 1, then they show on the list of missing files.

I have a dream

This feature could revolutionize Git.

Imagine having all the code base of your enterprise in a single repo without ugly third-party tools like repo.

Imagine storing huge blobs directly in the repo without any ugly third party extensions.

Imagine if GitHub would allow per file / directory metadata like stars and permissions, so you can store all your personal stuff under a single repo.

Imagine if submodules were treated exactly like regular directories: just request a tree SHA, and a DNS-like mechanism resolves your request, first looking on your local ~/.git, then first to closer servers (your enterprise's mirror / cache) and ending up on GitHub.

10

Try this:

svn export https://github.com/PhantomX/slackbuilds/trunk/${directory}
2
  • This is the simplest way. It does not download all the repo. Note that we must add the /trunk and remove /tree/master. More here – Bernardo Ramos Jan 23 '17 at 1:38
  • Thanks! I just want to clone a folder of the repo and not all the folders in a repo. And this one really works for me. – Edper Apr 18 '17 at 9:32
3

working solution :

you need to turn off the compression:

git config --global core.compression 0

then you need to use shallow clone

git clone --depth=1 <url>

then most important step is to cd into your cloned project

cd <shallow cloned project dir>

now deopen the clone,step by step

git fetch --depth=N, with increasing N

eg.

git fetch --depth=4

then,

git fetch --depth=100

then,

git fetch --depth=500

you can choose how many steps you want by replacing this N,

and finally download all of the remaining revisions using,

git fetch --unshallow 

upvote if it helps you :)

2

I got this working finally thanks to the above. I was editing the file directly, getting errors by accidently including the quotes. Your .git/info/sparse-checkout file should include just the directory, no quotes. Here's an example: This gigantic project: https://github.com/SharePoint/sp-dev-fx-webparts, I wanted just the react-script-editor in the samples directory. https://github.com/SharePoint/sp-dev-fx-webparts/tree/master/samples/react-script-editor

I followed the directions above and got it working when my .git/info/sparse-checkout file had just this in it

samples/react-script-editor

1

This will clone a specific folder and remove all history unrelated to it.

git clone --single-branch -b {branch} git@github.com:{user}/{repo}.git
git filter-branch --subdirectory-filter {path/to/folder} HEAD
git remote remove origin
git remote add origin git@github.com:{user}/{new-repo}.git
git push -u origin master
1
  • This will still clone all history (--depth=1 to avoid), and all other directories (??? to avoid?). – Mussri Jul 14 '20 at 21:39
1

For GitHub repos, you can clone any sub-directories of any GitHub repository (at any reference) using https://github.com/HR/github-clone

1
  • Sadly this is unsuitable for Dockerfile scenarios as it would haul the entirety of Python in as a dependency just to use it once during the image build. – Szczepan Hołyszewski Jul 19 '20 at 22:15
1

I'll answer with specific restrictions I faced.

  • In VSO, SSH was disabled by security, so couldn't use git archive. Only PAT/ CredentialProvider was supported.
  • Our branch had enormous code, so git clone would itself take 15 minutes, at minimum.

Here's what I did:

  1. One-time manual process: in VSO UI, create a new remote branch say 'emptyBranch' which has zero files (all are deleted).
  2. git clone --depth=1 --single-branch --branch emptyBranch <remote> (pretty quick)
  3. git checkout <branchToSyncTo> <directoryToSyncTo> (checks-out only specific folder)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.