43

I have for many years had my entire $HOME directory checked into subversion. This has included all my dotfiles and application profiles, many scripts, tools and hacks, my preferred basic home directory structure, not a few oddball projects and a warehouse worth of random data. This was a good thing. While it lasted.

But it's gotten out of hand. The basic checkout is the same across dozens of systems, but not all that stuff is appropriate for all my machines. It doesn't even all play nicely with different distros.

I'm in the process of cleaning house -- separating the data out where it belongs, splitting out some scripts as separate projects, fixing some broken links in stuff that should be automated, etc.

My intent is to replace subversion with git for the toplevel checkout of $HOME, but I'd like to pare this down to just the things I'd like to have on ALL my systems, meaning dotfiles, a few directories and some basic custom scripts.

In reading up online a lot of people seem to be doing this using the symlink approach: clone into a subdirectory then create symlinks from $HOME into the repository. Having had my $HOME under full version control for over a decade, I don't like the idea of this approach and I can't figure out why people seem so averse to the straight checkout method. Are there pitfalls I need to know about specific to git as a top level checkout for $HOME?

P.S. Partly as an exercise in good coding, I'm also planning on making my root checkout public on GitHub. It's scary how much security sensitive information I've allowed to collect in files that ought to be sharable without a second thought! WiFi password, un-passphrased RSA keys, etc. Eeek!

9
  • 6
    Curious what lead to the belief that $HOME should be sharable without a second thought‽ Even encrypted RSA private keys shouldn't be shared.
    – derobert
    Aug 28, 2012 at 19:16
  • 3
    if you are in fact talking about putting the contents of your home directory into git just note: it is difficult (but not impossible) to dig through git history and carefully remove sensitive items permanently (git is designed to help prevent losing things), and also remember that when you switch branches or checkout an earlier revision git will change the permissions of your files to 644 after checkout which is bad for things like private ssh keys. however etckeeper is a solution for using git with permissions for /etc/
    – cwd
    Aug 28, 2012 at 19:29
  • @derobert: I'm well aware of that. I wasn't talking about making $HOME public, just dotfiles and convenience scripts. Those are where I've been finding stuff that doesn't belong. And yes, I should be able to share my .zshrc, .vimrc and similar things without having to sanitize them first!
    – Caleb
    Aug 28, 2012 at 19:53
  • 5
    If you haven't seen it, see the vcs-home wiki and mailing lists, which is basically people discussing exactly this -- how to keep your $HOME under revision control.
    – Jim Paris
    Aug 28, 2012 at 20:02
  • I don't know how much you can change the behaviour of git, but at least the way it works out-of-the-debian-repository it is quite greedy when it comes to searching for tracked/untracked/modified files and it automatically feels responsible for every file. mrb already stated this. Sometimes I get annoyed by this greedy behaviour even in relatively small projects I wouldn't want that in my home directory. Why do want to use git? I'm also using a versioning system to sync my configuration files across hosts and I'm quite happy with CVS because it's so simple! Git is very (too!) powerfull for that
    – Bananguin
    Aug 29, 2012 at 7:54

9 Answers 9

19

Yes, there is at least one major pitfall when considering git to manage a home directory that is not a concern with subversion.

Git is both greedy and recursive by default.

Subversion will naively ignore anything it doesn't know about and it stops processing folders either up or down from your checkout when it reaches one that it doesn't know about (or that belongs to a different repository). Git, on the other hand, keeps recursing into all child directories making nested checkouts very complicated due to namespace issues. Since your home directory is likely also the place where you checkout and work on various other git repositories, having your home directory in git is almost certainly going to make your life an impossible mess.

As it turns out, this is the main reason people checkout their dotfiles into an isolated folder and then symlink into it. It keeps git out of the way when doing anything else in any child directory of your $HOME. While this is purely a matter of preference if checking your home into subversion, it becomes a matter of necessity if using git.

However, there is an alternate solution. Git allows for something called a "fake root" where all the repository machinery is hidden in an alternate folder that can be physically separated from the checkout working directory. The result is that the git toolkit won't get confused: it won't even SEE your repository, only the working copy. By setting a couple environment variables you can tip off git where to find the goods for those moments when you are managing your home directory. Without the environment variables set nobody is the wiser and your home looks like it's classic file-y self.

To make this trick flow a little smoother, there are some great tools out there. The vcs-home mailing list seems like the defacto place to start, and the about page has a convenient wrap up of howtos and people's experiences. Along the way are some nifty little tools like vcsh, mr. If you want to keep your home directory directly in git, vcsh is almost a must have tool. If you end up splitting your home directory into several repostories behind the scenes, combine vcsh with mr for quick and not very dirty way to manage it all at once.

8
  • 3
    but why not just add '*' to your .gitignore file? That way git will ignore everything except files that are already in the repository, and you can add new files with git add -f <file>.
    – ALiX
    Aug 31, 2012 at 21:59
  • @ALiX: Because the git tools would still consider you to be working on your home dir repo even if you were in some subdirectory that was a separate git repo for some project. That solution would render your entire home directory off limits to all other git work.
    – Caleb
    Aug 31, 2012 at 22:05
  • 6
    but a '*' in your .gitignore means that all files not in your home-dir repo are ignored. and when you check out a new git repo in some subdirectory, everything should still work as expected (i think). As far as I know, the git tools will look for the first .git directory while moving up the directory hierarchy. So, when working in the subdirectory, the correct git repository will be used. Of course, if you are using git's environment variables, I guess things could get messy. But otherwise, I don't see why this wouldn't work.
    – ALiX
    Aug 31, 2012 at 22:17
  • @ALiX is right. Nested git repos appear to work fine as long as you gitignore them in the parent repo. I wonder what are the drawbacks of this very simple approach, aside from possible problems with git's environment variables. Oct 17, 2013 at 5:19
  • 1
    Been experimenting with this today. I think /* works better than * because it still ignores everything by default but makes it much easier to add directories. Instead of git add -f I'm using !-prefixed patterns like !/.vimrc and !/.gitignore (for the .gitignore file itself) to explicitly include things in the repo. Oct 17, 2013 at 21:34
14

I wouldn't want my entire home directory checked into version control simply because it means every subdirectory I go into would have the version-control context of my home dir. Commands like git checkout would have an actual action in that case, causing issues if I accidentally run something from the wrong directory, whether that something is git itself or a script that calls git.

It also makes it more likely to add something to the repo that you don't want, which would not have been an issue when you had everything checked in, but now becomes a problem. What if you accidentally add a private key file (perhaps out of habit) and push it to github?

Having said that, I think the primary disadvantages are not really technical — just wanting to save me from myself.

As for symlinking: You could clone your repo into a subdirectory, and have a script which updates any symlinks that need to be updated. The amount of maintenance required for this script might outweigh the benefits of having it at all, though; symlinking might turn out to be less work.

With symlinks, you can also easily make distro-specific (or even host-specific) additions that get checked into git. Your symlink-update script will ignore files intended for incompatible platforms or different hosts, and only update the appropriate ones.

Something like:

HOMEREPO=$HOME/homerepo
HOST=$(hostname)
UNAME=$(uname)

for dotfile in $HOMEREPO/shared/* $HOMEREPO/host-$HOST/* $HOMEREPO/uname-$UNAME/*
do
    target=$HOME/$(basename $dotfile)
    [ ! -r $target ] && ln -s $dotfile $target
done

Personally: I use symlinks, and I don't symlink directories; only the files within. This gives me some flexibility to make site-local changes in those directories (ie. add/remove files). Setting up my account on a new system is tedious because I have to recreate all the symlinks by hand.

5
  • Any git commands I run would either be for the home directory itself or would be buried at least one deep in a NON commited directory. Using svn this one folder isolation is quite effective and hasn't caused me any trouble in a decade. Your first paragraph indicates something else. Is this a actually a difference in the way git works?
    – Caleb
    Aug 28, 2012 at 19:57
  • Also my configs and scripts already have conditional logic for different hosts and platforms built in, so using a script to setup different links as conditionals doesn't seem like much of a gain over git's easy to mange branches. Still something I'm missing or is this going to come down to preference?
    – Caleb
    Aug 28, 2012 at 20:01
  • 3
    One-folder isolation doesn't really isolate in git — not sure about svn — but for an example, git init foo && mkdir -p foo/bar/baz/spam && cd foo/bar/baz/spam && git status (or other git commands) show that you're still in foo's version control context.
    – mrb
    Aug 28, 2012 at 21:26
  • Configs & scripts: Not all dotfiles support conditionals, which is why I suggested the alternate approach. These are all reasons that I think people prefer not to use version control for $HOME — and versioning isn't really valuable for dotfiles imo — but ultimately it's your home directory, so if you prefer to use git and these aren't problems for you, go for it!
    – mrb
    Aug 28, 2012 at 21:30
  • Thanks for the info. Actually your comment about git not allowing for isolation is the most useful bit. You might work that prominently into your answer. Subversion behaves very differently on that point and its significant for this use case.
    – Caleb
    Aug 31, 2012 at 18:25
9

To give another point of view: I have my $HOME under git since sometime now and didn't find any drawbacks. I obviously do not sync this git repo to github; I use a service which have private repos. I also do not put any media files or downloads or packages under git control.

  • git status is a kind of "to do, to clean" checklist.

  • I have a ~/tmp for temporary things, which is gitignored.

  • I like to see in git status anything that a recently installed software dare to add to my $HOME, and often delete these files, or even uninstall the culprits.

  • I add manually the really useful local files and dirs to .gitignore, which has a 'know what you do when installing things' benefit.

  • If I build a new VM or install a new PC, I just clone my remote home to $HOME and have immediately everything I need under hands.

  • Things like vundle for vim plugins are not necessary anymore.

I dislike complexity. When I tweak any rcfile, I just do it, commit and push. Then, as a reflex, I git pull in $HOME every other day, and have always the latest config. It is that simple.

Machines currently under this regimen: Home laptop, work PC, work VM, plus 3 or 4 remote servers.

4
  • Do you have any other git checkouts nested inside your home?
    – Caleb
    Sep 4, 2012 at 7:56
  • No, I put other things in a /work directory and do not clone little tools like vim pugins.
    – gb.
    Sep 5, 2012 at 10:49
  • 2
    I have work inside a ~/Sites and do this approach as well, there's no problem with nested git repos
    – philfreo
    Nov 2, 2012 at 15:49
  • 3
    I've been using this setup for a while. I have an 'alias sq=git status -uno' and don't bother much with .gitignore (every so often I take a look at all the cruft and then say "meh"). I've never had issues with nested git repos. I have a private server where I did a git init --bare that I push to over ssh (though I don't put passwords in the repo, I do have my notes files there).
    – unhammer
    Apr 1, 2016 at 7:37
6

I've tried both, and preferred the symlink approach in the end:

  • Check out to wherever
  • make install
  • Log out and in again to load the X settings

Disadvantages:

  • Have to move files to the repo before adding them
  • Have to maintain the list of symbolic links in the Makefile

Advantages:

  • No need for a massive .gitignore (I have 133 dotfiles in ~ on my humble Ubuntu box)
  • Can keep maintenance scripts and other ~-related stuff (such as Makefile and utility scripts) out of the way
  • Can version control personal and public settings separately

Restrictions:

  • Unlike @mrb, I only create symlinks in ~. That keeps the symlinking simple, and makes it trivial to notice new files in for example ~/.vim, at the cost of some very rare .gitignore maintenance.

That last two advantages tipped the scales in my case - I don't want to clutter the home directory, and I want to keep private and public content clearly separate.

The only application I know of which has (or at least had) problems with handling symlinks was Pidgin - It kept overwriting my symlinks with ordinary files.

3
  • Thanks for your input on the pro's and cons of each approach. In my follow up I've discovered there is a third aproach that might make the best of both worlds if you're ok setting up the extra wiring to get started.
    – Caleb
    Aug 31, 2012 at 18:43
  • I too have tried both approaches and found that I preferred the sym-link approach to maintaining a large .gitignore. However, I kind of over-engineered the maintenance scripts to work with Debian, Red Hat and Cygwin systems and am interested in examples of simpler alternatives. It's really useful to see your Makefile solution but I thought I'd let you know that the cleanup script is no longer available on GitHub. Apr 5, 2020 at 20:18
  • 1
    @AnthonyGeoghegan I really should update this answer, but in the meantime you could check out the latest cleanup code in the new repo.
    – l0b0
    Apr 5, 2020 at 20:26
3

Here's one: If you try to do git rebase -i --root and you have checked in .gitconfig in the first commit in the repository, git will temporarily remove the .gitconfig file, which in turn will make it unable to finish the rebase operation since it requires your name and your email to do that, which are stored in that file.

You may configure them back again and do git rebase --continue, but after I did that and finished the rebase operation, my git repository had gained an empty commit without a commit message before the commit that was previously the first commit in the repository, which I do not know how to get rid of.

I don't know what happens if you do git rebase -i <commit> instead, and .gitconfig is checked in together with any commit after <commit>.

Perhaps the easiest solution is to refrain from adding .gitconfig to the repository and instead list it in .gitignore.

2

This is how I do it:

  1. install a clean linux (not necessary, but makes life more pleasant in step 4)
  2. install etckeeper
  3. run git init in your home
  4. create .gitignore and add everything that looks like it doesn't interest you or that might change a lot. Be sure to add things like *.cache, *.lock etc. I don't recommend adding /* because you won't be notified automatically when something new is added to your home. It is a blacklist approach vs whitelist approach, where I basically want to keep my config for all software except for volatile stuff and some software I don't care about. When you later merge, migrate or compare systems, being able to diff everything is quite handy. You can set up your new systems much faster than if you would just have .bashrc and a few other dotfiles stored. This way you will keep configuration that you might otherwise set through the GUI, and be unaware of which dotfiles store the settings. (If it ever turns out you have committed volatile files, you can still tell git to assume-unchanged)
  5. run etckeeper init -d /home/username
  6. run git commit -d /home/username
  7. set up aliases in your shell to make the command line nicer, like homekeeper checkout

The reason for using etckeeper is that it will store metadata like permissions for your files (rather important for certain things like ssh keys). You should now have a pre-commit hook that will save metadata automatically. I'm not so sure about post-checkout. You should probably use etckeeper checkout xxx -d /home/user I will look into it a bit more and elaborate this answer.

0

My $HOME has been a git repository for years, and I have other repos under it, and haven't had any problems.

A sticking point for many people seems to be the .gitignore file, presumably because they don't want every untracked file to show up when doing git status. There is a concern that your ~/.gitignore might exclude things that you wouldn't want to exclude from other repos under your home directory.

I didn't add anything to my ~/.gitignore; instead, I set status.showUntrackedFiles = no in my ~/.git/config. Then the output of git status is clean. If for some reason you want to see the "normal" output, just say git status -unormal. Or you can also use git ls-files -u to list the untracked files under the CWD for a more targeted listing.

Alternatively, you can use git status -uno (or an alias for it) to get the same "quiet" status output, as someone suggested in a comment.

-1

My major problem with using Git on the home directory is that Git does not store file attributes such as file permissions and timestamps. For me it is important to know when certain files were created, that may or may not be the case for you. Furthermore, loosing permissions to files and directories such as .ssh is problematic. I understand that you plan on keeping .ssh out of Git, but there will be other places where permissions might matter (such as uncompressed website backups).

4
  • This is misleading if not factually wrong. Git by default preserves many file attributes including permissions. I've been keeping .ssh in git for some time now without an issue, proper secure permissions are preserved. What it doesn't do in the base configuration is preserve ownership or timestamps; yet if either of these this is an issue for a specific use case there are plugins that can make handling these additional properties part of the regular workflow (see metastore or git-cache-meta).
    – Caleb
    Sep 21, 2013 at 9:00
  • Even if it doesn't store them, how is that any worse than just having a home directory not in vcs? git is not going to actively overwrite mtimes unless you ask it to change a file.
    – poolie
    Apr 9, 2014 at 23:51
  • Do you really need the actual creation time for the file? I find the git log more useful than the file timestamp, because then I have the time of every modification made to the file (admittedly, the time of the commit, not of the actual modification, but that's close enough for me).
    – itub
    May 31, 2021 at 11:57
  • @itub For certain files, yes I do need the actual creation time. For example, diary notes in Markdown files. Or photographs, where the EXIF information may be missing. Or recorded telephone calls. Etc.
    – dotancohen
    Jun 1, 2021 at 10:11
-1

A git-based solution is especially useful if you need to deploy your files to different machines, and even more so if you have parts that are common to all machines, and parts that are specific to some machines. You can make multiple repositories and use a tool like multigit or vcsh to clone them over the same directory (your home dir in this case).

1
  • Thanks, but maybe you missed the question. I'm well aware of the uses for this (hence why I wanted to do it in the first place), this question was about any pitfalls somebody new to doing this with git (as I was when I asked) might not be aware of. This doesn't seem to answer that question at all.
    – Caleb
    Apr 15, 2015 at 6:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .