57

I found a collection of slackbuilds, some i need there are on GitHub. https://github.com/PhantomX/slackbuilds/ I don't want to get all git.

git clone https://github.com/PhantomX/slackbuilds.git

But only get a slackbuild, for this one.

How to do this? Is it possible?

2

8 Answers 8

44

You will end up downloading the entire history, so I don't see much benefit in it, but you can checkout specific parts using a "sparse" checkout. Quoting this Stack Overflow post:

The steps to do a sparse clone are as follows:

mkdir <repo>
cd <repo>
git init
git remote add -f origin <url>

I'm going to interrupt here. Since I'm quoting another post, I don't want to edit the quoted parts, but do not use -f with git remote add. It will do a fetch, which will pull in the entire history. Just add the remote without a fetch:

git remote add origin <url>

And then do a shallow fetch like described later.

This creates an empty repository with your remote, and fetches all objects but doesn't check them out. Then do:

git config core.sparseCheckout true

Now you need to define which files/folders you want to actually check out. This is done by listing them in .git/info/sparse-checkout, eg:

echo "some/dir/" >> .git/info/sparse-checkout
echo "another/sub/tree" >> .git/info/sparse-checkout

[...]

You might want to have a look at the extended tutorial and you should probably read the official documentation for sparse checkout.

You might be better off using a shallow clone. Instead of a normal git pull, try:

git pull --depth=1 origin master

I had an occasion to test this again recently, trying to get only the Ubuntu Mono Powerline fonts. The steps above ended up downloading some 11 MB, where the Ubuntu Fonts themselves are ~900 KB:

% git pull --depth=1 origin master
remote: Enumerating objects: 310, done.
remote: Counting objects: 100% (310/310), done.
remote: Compressing objects: 100% (236/236), done.
remote: Total 310 (delta 75), reused 260 (delta 71), pack-reused 0
Receiving objects: 100% (310/310), 10.40 MiB | 3.25 MiB/s, done.
Resolving deltas: 100% (75/75), done.
From https://github.com/powerline/fonts
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
% du -hxd1 .
11M     ./.git
824K    ./UbuntuMono
12M     .

A normal clone took about 20 MB. There's some savings, but not enough.

Using the --filter + checkout method in Ciro Santilli's answer really cuts down the size, but as mentioned there, downloads each blob one by one, which is slow:

% git fetch --depth=1 --filter=blob:none
remote: Enumerating objects: 52, done.
remote: Counting objects: 100% (52/52), done.
remote: Compressing objects: 100% (49/49), done.
remote: Total 52 (delta 1), reused 35 (delta 1), pack-reused 0
Receiving objects: 100% (52/52), 14.55 KiB | 1.32 MiB/s, done.
Resolving deltas: 100% (1/1), done.
From https://github.com/powerline/fonts
 * [new branch]      master     -> origin/master
 * [new branch]      terminus   -> origin/terminus
% git checkout origin/master -- UbuntuMono
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 1.98 KiB | 1.98 MiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 581 bytes | 581.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 121.43 KiB | 609.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 100.66 KiB | 512.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 107.62 KiB | 583.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 112.15 KiB | 791.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 454 bytes | 454.00 KiB/s, done.
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), 468 bytes | 468.00 KiB/s, done.
% du -hxd1 .
692K    ./.git
824K    ./UbuntuMono
1.5M    .

TL;DR: Use all of --filter, sparse checkout and shallow clone to reduce the total download, or only use sparse checkout + shallow clone if you don't care about the total download and just want that one directory however it may be obtained.

5
  • 1
    I am getting error: Sparse checkout leaves no entry on working directory after doing a git pull after making the modification to core.sparseCheckout and .git/info/sparse-checkout Commented Aug 7, 2017 at 12:00
  • Correction: git init; git remote ... just set up things, nothing is downloaded. The whole point of sparse is to get only objects you are interested in.
    – vonbrand
    Commented Feb 22, 2020 at 19:08
  • @vonbrand that's what the -f option is for: "With -f option, git fetch <name> is run immediately after the remote information is set up."
    – muru
    Commented Feb 23, 2020 at 16:05
  • 1
    After git remote add -f origin <url> git is still downloading the whole repo. In my case, I don't want to spend time waiting for that, even if it doesn't checkout the entire repo. Commented Apr 28, 2020 at 20:01
  • In sparse checkout we define what we want to check out but is there a way to exclude a few directories and check out the rest?
    – Ram Patra
    Commented Jun 14, 2020 at 11:24
33

git clone --filter + git sparse-checkout downloads only the required files

E.g., to clone only files in subdirectory small/ in this test repository: https://github.com/cirosantilli/test-git-partial-clone-big-small-no-bigtree

git clone -n --depth=1 --filter=tree:0 \
  https://github.com/cirosantilli/test-git-partial-clone-big-small-no-bigtree
cd test-git-partial-clone-big-small-no-bigtree
git sparse-checkout set --no-cone small
git checkout

You could also select multiple directories for download with:

git sparse-checkout set --no-cone small small2

This method doesn't work for individual files however, but here is another method that does: https://stackoverflow.com/questions/2466735/how-to-sparsely-checkout-only-one-single-file-from-a-git-repository/52270527#52270527

In this test, clone is basically instantaneous, and we can confirm that the cloned repository is very small as desired:

du --apparent-size -hs * .* | sort -hs

giving:

2.0K    small
226K    .git

That test repository contains:

  • a big/ subdirectory with 10x 10MB files
  • 10x 10MB files 0, 1, ... 9 on toplevel (this is because certain previous attempts would download toplevel files)
  • a small/ and small2/ subdirectories with 1000 files of size one byte each

All contents are pseudo-random and therefore incompressible, so we can easily notice if any of the big files were downloaded, e.g. with ncdu.

So if you download anything you didn't want, you would get 100 MB extra, and it would be very noticeable.

On the above, git clone downloads a single object, presumably the commit:

Cloning into 'test-git-partial-clone-big-small'...
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (1/1), done.

and then the final checkout downloads the files we requested:

remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 3 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), 10.19 KiB | 2.04 MiB/s, done.
remote: Enumerating objects: 253, done.
remote: Counting objects: 100% (253/253), done.
Receiving objects: 100% (253/253), 2.50 KiB | 2.50 MiB/s, done.
remote: Total 253 (delta 0), reused 253 (delta 0), pack-reused 0
Your branch is up to date with 'origin/master'.

Tested on git 2.37.2, Ubuntu 22.10, on January 2023.

TODO also prevent download of unneeded tree objects

The above method downloads all Git tree objects (i.e. directory listings, but not actual file contents). We can confirm that by running:

git ls-files

and seeing that it contains the directories large files such as:

big/0

In most projects this won't be an issue, but the perfectionist in me would like to avoid them.

I've also created a very extreme repository with some very large tree objects (100 MB) under the directory big_tree: https://github.com/cirosantilli/test-git-partial-clone-big-small

Let me know if anyone finds a way to clone just the small/ directory from it!

About the commands

The --filter option was added together with an update to the remote protocol, and it truly prevents objects from being downloaded from the server.

The sparse-checkout part is also needed unfortunately. You can also only download certain files with the much more understandable:

git clone --depth 1  --filter=blob:none  --no-checkout \
  https://github.com/cirosantilli/test-git-partial-clone-big-small
cd test-git-partial-clone-big-small
git checkout master -- d1

but that method for some reason downloads files one by one very slowly, making it unusable unless you have very few files in the directory.

Another less verbose but failed attempt was:

git clone --depth 1 --filter=blob:none --sparse \
  https://github.com/cirosantilli/test-git-partial-clone-big-small
cd test-git-partial-clone-big-small
git sparse-checkout set small

but that downloads all files in the toplevel directory: https://stackoverflow.com/questions/75311408/how-to-prevent-git-clone-filter-blobnone-sparse-from-downloading-files-on-t

The dream: any directory can have web interface metadata

This feature could revolutionize Git.

Imagine having all the code base of your enterprise in a single monorepo without ugly third-party tools like repo.

Imagine storing huge blobs directly in the repo without any ugly third party extensions.

Imagine if GitHub would allow per file / directory metadata like stars and permissions, so you can store all your personal stuff under a single repo.

Imagine if submodules were treated exactly like regular directories: just request a tree SHA, and a DNS-like mechanism resolves your request, first looking on your local ~/.git, then first to closer servers (your enterprise's mirror / cache) and ending up on GitHub.

I have a dream.

The test cone monorepo philosophy

This is a possible philosophy for monorepo maintenance without submodules.

We want to avoid submodules because it is annoying to have to commit to two separate repositories every time you make a change that has a submodule and non-submodule component.

Every directory with a Makefile or analogous should build and test itself.

Such directories can depend on either:

  • every file and subdirectory under it directly at their latest versions
  • external directories can be relied upon only at specified versions

Until git starts supporting this natively (i.e. submodules that can track only subdirectories), we can support this with some metadata in a git tracked file:

monorepo.json

{
    "path": "some/useful/lib",
    "sha": 12341234123412341234,
}

where sha refers to the usual SHA of the entire repository. Then we need scripts that will checkout such directories e.g. under a gitignored monorepo folder:

monorepo/som/useful/lib

Whenever you change a file, you have to go up the tree and test all directories that have Makefile. This is because directories can depend on subdirectories at their latest versions, so you could always break something above you.

Related:

10

Try this:

svn export https://github.com/PhantomX/slackbuilds/trunk/${directory}
2
  • This is the simplest way. It does not download all the repo. Note that we must add the /trunk and remove /tree/master. More here Commented Jan 23, 2017 at 1:38
  • Thanks! I just want to clone a folder of the repo and not all the folders in a repo. And this one really works for me.
    – Edper
    Commented Apr 18, 2017 at 9:32
3

working solution :

you need to turn off the compression:

git config --global core.compression 0

then you need to use shallow clone

git clone --depth=1 <url>

then most important step is to cd into your cloned project

cd <shallow cloned project dir>

now deopen the clone,step by step

git fetch --depth=N, with increasing N

eg.

git fetch --depth=4

then,

git fetch --depth=100

then,

git fetch --depth=500

you can choose how many steps you want by replacing this N,

and finally download all of the remaining revisions using,

git fetch --unshallow 

upvote if it helps you :)

2

I got this working finally thanks to the above. I was editing the file directly, getting errors by accidently including the quotes. Your .git/info/sparse-checkout file should include just the directory, no quotes. Here's an example: This gigantic project: https://github.com/SharePoint/sp-dev-fx-webparts, I wanted just the react-script-editor in the samples directory. https://github.com/SharePoint/sp-dev-fx-webparts/tree/master/samples/react-script-editor

I followed the directions above and got it working when my .git/info/sparse-checkout file had just this in it

samples/react-script-editor

1

This will clone a specific folder and remove all history unrelated to it.

git clone --single-branch -b {branch} [email protected]:{user}/{repo}.git
git filter-branch --subdirectory-filter {path/to/folder} HEAD
git remote remove origin
git remote add origin [email protected]:{user}/{new-repo}.git
git push -u origin master
1
  • This will still clone all history (--depth=1 to avoid), and all other directories (??? to avoid?).
    – Mussri
    Commented Jul 14, 2020 at 21:39
1

For GitHub repos, you can clone any sub-directories of any GitHub repository (at any reference) using https://github.com/HR/github-clone

1
  • Sadly this is unsuitable for Dockerfile scenarios as it would haul the entirety of Python in as a dependency just to use it once during the image build. Commented Jul 19, 2020 at 22:15
1

I'll answer with specific restrictions I faced.

  • In VSO, SSH was disabled by security, so couldn't use git archive. Only PAT/ CredentialProvider was supported.
  • Our branch had enormous code, so git clone would itself take 15 minutes, at minimum.

Here's what I did:

  1. One-time manual process: in VSO UI, create a new remote branch say 'emptyBranch' which has zero files (all are deleted).
  2. git clone --depth=1 --single-branch --branch emptyBranch <remote> (pretty quick)
  3. git checkout <branchToSyncTo> <directoryToSyncTo> (checks-out only specific folder)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .