etckeeper with git, a way to deal with huge repo

Question

I recently discovered that /etc/.git/ reached about 30G [sic!] of disk space. This repo is only for etckeeper. I have small experience with git.

I came up with two solutions:

1. Obvious one, delete old commits (above one year span). But could bcommits be dependent in a way?
2. git gc is another one. I never did this one before, only read git help gc. Is says that use of this command is encouraged. So as I understand, only internal structure (and some garbage collecting) is changed and cloning, getting a commit and committing is still possible without any changes, and data is not affected?

What is a good practice in such case?

Answer 1

The idea of git gc is to remove objects that are no longer accessible. As etckeeper simply adds commits, this will probably not help much. But it won't hurt either and maybe save a bit of space through repacking.

You probably managed to have some huge file in /etc/ in the past, which now still lives in your git history. Either throw away all your git history. (In case of /etc/ this might be an option.) Or try to remove the huge file from your history. Have a look at the BFG Repo-Cleaner.

Answer 2

Your repos probably contains some large files in the history. Can you give the output of du -hs, please? That would clarify whether any of those files are in the current /etc directory, and therefore in the HEAD of the gitkeeper /etc git repos. A more interactive approach is to use the handy tool ncdu. If there are unnecessary big files in the current /etc directory you could just delete them. However, I'll assume in the remainder of this answer that they are mostly in history and not in the current /etc directory.

An option is to rewrite the git repos to remove those large files. This would proceed in two steps.

1. Identify the large files.
2. Rewrite the repos to remove those files.

Note that the HEAD of the repo will stay the same, so it not affect your /etc directory.

I was recently peripherally involved in doing exactly this for a git repos (not my own). If you want more details, I could try to dig them out. However, the approach used there was very manual and DIY. Of the people involved, none of them, including myself, was a git expert. So, if there are existing tools that automate this, that might be better.

Answer 3

The huge repo is in /etc/.git. If you don't need to retain that history, you can simply delete that git repository entirely by the following methods:

1) Delete the .git directory manually with 'rm -rf .git' - I'd consider this an emergency running out of space option. I don't know if etckeeper freaks or what, but this will definitely fix the situation. Stackoverflow reference: https://stackoverflow.com/questions/1213430/how-to-fully-delete-a-git-repository-created-with-init

2) The other option is to do the same thing using etckeeper itself. From the etckeeper site:

"Is the history recorded in that repository something you need to preserve, or can you afford to just blow it away and check the current /etc into the new VCS?

In the latter case, you just need to follow three steps:

etckeeper uninit # deletes /etc/.git!
vim /etc/etckeeper/etckeeper.conf
etckeeper init

Reference: https://github.com/joeyh/etckeeper

You don't have to change the VCS by editing the etckeeper.conf file. If you leave the etckeeper VCS alone and do "etckeeper init", it just starts a new repo using git with the current state of the /etc directory.

Here's another reference from Turnkey Linux:

etckeeper has HUGE .git repo, how to remove???

Accepted answer by the maintainers: "Run: etckeeper uninit -f; etckeeper init"

I was running out space. I just did the step above and I deleted 15 GB of unneeded history. I think etckeeper's great to monitor the /etc directory on an ongoing basis, but I don't need 2 years of history.

Answer 4

I have little experience with git, so can only give you some hints from novice user. I mean, what I learned after versioning /etc and other folders, is that the easiest solution is always the best. Saying that, I mean to keep KISS rule (imagine yourself having to quickly free up some disk space). How are you going to do that, knowing that you don't need to keep /etc log history? Complex solutions are okay, if you have time aren't under pressure to maintain it.

In your case — and in fact mine too — the ideal solution was to simply remove .git subfolder and initialize repo anew. I know it's the easiest solution and not always applicable, but remember — pressure, time, simplicity, so search for the easiest solution.

Answer 5

I know what caused the problem. As it turned out a SAMBA4/AD ldb was in /etc/; the file is about ~500MB and git was making a snapshot of the whole file if something in the database has changed. The repo was about one year old, so size of it was appropriate ;)

Useful tips:

• Git repo is self-contained, so simple cp -r (scp, ... ) is feasible.
• Git internally is a bunch of blobs (mostly), so compressing isn't that helpful.
• git gc checks for unavailable objects and sometimes compress stuff to, but because of previous point it isn't that effective in terms of disk space saving.

So the solution was simply backup the repo (archive or clone for the latest version), then remove it and invoke in /etc/:

etckeeper init
etckeeper commit "First message in new repo."

And maybe tailor .gitignore to suit your needs. Thanks for answers, both was helpful and correct.