3

The example below is simplified to show the core of the problem, not the problem itself(my file tree is more complicated than that).

Let's say that I have two files I want to back up; one in ~/somedir/otherdir, the other in ~/otherdir/somedir/. I want to backup files from both directories in one git repository. How can I do this? Soft links only carry information about where file is stored, not the actual file, whereas hard links are somewhat foreign to me. Is this a case where hard links should be used?

Clarification: I want to use git because of four reasons: I want to store dotfiles/scripts/configurations that are text files and keep track of changes over time, I know git, I have a private git repository I could use to store them, and I want to be able to share these files across multiple PCs.

  • you cannot hard-link a folder – mazs May 3 '16 at 12:01
  • @mazs I gave an example of where are some files; I want to keep track of files, not folders. – MatthewRock May 3 '16 at 14:10
  • What is the motivation of using Git? In my experience all of the concepts of Git revolve around a repository which contents have a common root/base. You seem to want to have several unrelated files/projects/repositories lumped together...There are also other considerations such as those posed in this answer – forquare May 3 '16 at 14:32
  • @forquare the motivation is that the files I want to store are mostly dotfiles and scripts. Now I feel a bit stupid for not mentioning it. Anyway, git does fine job at tracking changes in such files and that's one of the features I'm after - the other being the backup mechanism - and the last one being free private github repo I've got. – MatthewRock May 4 '16 at 8:48
  • @MatthewRock that makes more sense :) Most of the setups I've seen to do similar involve having everything under one root and then having an install.sh script to symlink the files to the relevant locations - which makes it easy to keep everything up-to-date and to deploy onto other systems/other users. Though from your comments it looks like you'd rather not do this, so the second part of Stephen's answer seems best suited. – forquare May 4 '16 at 9:02
6

If you don't mind moving the files...

You could do this by moving the files into a git repository, and symlinking them to their old locations; you'd end up with

  • ~/gitrepo/somedir/otherdir/file1 moved from ~/somedir/otherdir/file1
  • ~/gitrepo/otherdir/somedir/file2 moved from ~/otherdir/somedir/file2
  • a symlink from ~/somedir/otherdir/file1 to ~/gitrepo/somedir/otherdir/file1
  • a symlink from ~/otherdir/somedir/file2 to ~/gitrepo/otherdir/somedir/file2

You can then safely commit the files in the git repository, and manipulate them using git, and anything using the old file paths will see whatever is current in the git workspace. Linking the files the other way round, or using hard links, would be dangerous because any git operation which re-writes the file (changing branches, reverting to a previous version...) would break the link. (Hopefully this explains why hard links aren't really a viable solution.)

With this kind of scenario you'll have to be careful with programs which re-write files completely, breaking links; many text editors do this, as do tools such as sed -i etc. A safer approach would be to move the entire folders into the git repository, and symlink the directories.

If you want to keep the files in place...

Another possibility is to create a git repository in your home directory, tell git to ignore everything, and then forcefully add the files you do want to track:

cd
git init
echo '*' > .gitignore
git add -f .gitignore
git commit -m "Initialise repository and ignore everything"
git add -f somedir/otherdir/file1
git commit -m "Add file1"
git add -f otherdir/somedir/file2
git commit -m "Add file2"

Once you've done this, you'll easily be able to track changes to files you've explicitly added, but git won't consider new files. With this setup it should also be safe to have other git repositories in subdirectories of your home directory, but I haven't checked in detail...

  • I don't really want to move the original files. I'd much prefer for them to stay where they are. – MatthewRock May 3 '16 at 12:48
  • I've added a variant which doesn't involve moving the original files. – Stephen Kitt May 3 '16 at 14:28
1

I wrote how I manage dotfiles in here on GitHub: https://github.com/jesuswasrasta/dotfiles

My solution doesn't involve moving the original files (even I don't like it), nor symlinking or so. The underlying system remains untouched, agnostic.

Dotfiles

This is my personal repo for dotfiles I use.

The idea

I need a lean way to store, backup and version dotfiles on my machines (*nix but not only).
I love Git, so I looked around for other people dealing with them using this DVCS, and I found some different apporaches: this is the one I ended up.

The inspiration come form an article from Nicola Paolucci, a former Atlassian developer instigator, as he defines itself :)
This is the orginal article: Best way to store dotfiles.
I like the idea very much.
I adapted it a little bit to better match my taste about command names and folders.

How it works

The technique consists in storing dotfiles in a Git bare repository, but in a customized $HOME/.dotfiles folder instead of the usual .git folder. Then, using a special Git alias command, I'm able to add and commit files I want to track in the repo, and pushing them to a remote (a private BitBucket repo in my case; I suggest to go private until you are sure not to push confidetial information).

Repository structure

I use a different Git branch for every machine; on master branch there are only common file I use as a template.

Master branch Contains:

  • common ignore file
  • common shell aliases I always use
  • other common files and configurations I like to have on all my machines

Starting from scratch

If you haven't been tracking your configurations in a Git repository before, you can start using this technique easily with these lines:

1) git init --bare $HOME/.dotfiles   
2) alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME'
3) dotfiles config --local status.showUntrackedFiles no 
4) dotfiles config --local core.excludesFile=.dotfilesignore 
5) echo ". ~/.zsh_aliases" >> $HOME/.zshrc  
6) echo "alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME'" >> $HOME/.zsh_aliases 
7) echo "alias dfg=dotfiles" >> $HOME/.zsh_aliases 
  1. The first line creates a folder ~/.dotfiles which is a Git bare repository that will track your files.
  2. Then we create a new command dotfiles that is nothing else than a customized alias to git command, configured to work only with your .dotfiles repo. We will use it instead of the regular git command when we want to interact with our configuration repository.
  3. We set the flag status.showUntrackedFiles no - local to the repository - to hide files we are not explicitly tracking yet. This is so that when you type dotfiles status and other commands later, files you are not interested in tracking will not show up as untracked.
  4. To make things more coherent, I use a custom .dotfilesignore as Git ignore file instead of the common .gitignore one. This way it's easier to remember where to write your exclusions.
  5. I usually store my aliases in a separate file, like .zsh_aliases, then I include it in my .zshrc appending this line to it: . ~/.zsh_aliases.
  6. Add the dotfiles alias definition by hand or use the the command provided for convenience.
  7. I usually create even another alias to my newly created command called dfg; I would call it df but, you know, there's already a df command on Unix boxes :)
    So I opted for dot files git acronym, dfg, that is (at least for me) easy to remember and convenient, as in QWERTY keyboards the three letters are adjacent.

As a Zsh user, in these example I worked with .zshrc and .zsh_aliases files; feel free to substitute them with .bashrc and .bash_aliases if you plan to use Bash as your preferred shell: they works the same way.

After you've executed the setup, any file within the $HOME folder can be versioned with normal Git commands, replacing git with your newly created dotfiles alias:

dotfiles status dotfiles add .vimrc dotfiles commit -m "Add vimrc"  
dotfiles add .bashrc dotfiles commit -m "Add bashrc" dotfiles  
push 

Install your dotfiles onto a new system

Prior to the installation make certain you have committed the dotfiles alias as in steps 5), 6) and 7).

Then be sure your repository ignores the folder where you'll clone it, so that you don't create weird recursion problems:

echo ".dotfiles" >> .dotfilesignore  

Now clone your Dotfiles repository into a bare repository in the .dotfiles dot folder of your $HOME:

git clone --bare <git-repo-url> $HOME/.dotfiles 

Define the alias in the current shell scope:

alias dotfiles='/usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME'

Checkout the actual content from the bare repository to your $HOME:

dotfiles checkout master 

The step above might fail with a message like this:

error:
The following untracked working tree files would be overwritten by
checkout:
    .zshrc

Please move or remove them before you can switch branches.

Aborting 

This is because your $HOME folder might already have some stock configuration files which would be overwritten by Git (like .zshrc in my case). The solution is simple: back up the files if you care about them, remove them if you don't care. Nicola provided us with a possible rough shortcut to move all the offending files automatically to a backup folder:

mkdir -p .dotfiles-backup && \ dotfiles checkout 2>&1 | egrep "\s+\." | awk {'print $1'} | \ xargs -I{} mv {} .dotfiles-backup/{} 

Re-run the check out if you had problems:

dotfiles checkout master 

Set the flag showUntrackedFiles to no on this specific (local) repository:

dotfiles config --local status.showUntrackedFiles no 

Set the flag core.excludesFile to .dotfilesignore on this specific (local) repository:

dotfiles config --local core.excludesFile=.dotfilesignore 

Now you have two options: create a new branch for the brand new machine you are installing, or checkout an existing branch and use it.

Create a new branch

dotfiles checkout -b <my-new-machine-branch-name> 

At this point you can start customizing your dotfiles and the commit them to this new branch. You can also diff your files form other branches and cherry-pick stuff you like form other branches.

Checkout an existing branch

In the previous steps we checked out master branch, that in my personal dotfiles repo contains only shared file and configurations I like to have on all my machines (my machines are not all 100% equal).
But you can commit all the dotfiles you like in master branch and always use it, or checkout an existing (other machine) into the new one if you like: simply switch branch like this:

dotfiles checkout <existing-machine-branch-name> 

Of course, you will have some conflicts like we mentioned before: handle them with the provided script or delete current dotfiles you planned to override.

Summary

You're done, from now on you can now type dotfiles or dfg commands to add and update your dotfiles:

dotfiles status dotfiles add .vimrc dotfiles commit -m "Add vimrc" 
dotfiles add .bashrc dotfiles commit -m "Add bashrc" 
dotfiles push 

Install script

As a shortcut not to have to remember all these steps on any new machine you want to setup, you can create a simple script like this:

git clone --bare <url-of-your-repo> $HOME/.dotfiles
function dotfiles {
   /usr/bin/git --git-dir=$HOME/.dotfiles/ --work-tree=$HOME $@
}
mkdir -p .dotfiles-backup
dotfiles checkout
if [ $? = 0 ]; then
  echo "Checked out dotfiles repository.";
  else
    echo "Backing up pre-existing dot files.";
    dotfiles checkout 2>&1 | egrep "\s+\." | awk {'print $1'} | xargs -I{} mv {} .dotfiles-backup/{}
fi;
dotfiles checkout
dotfiles config --local status.showUntrackedFiles no
dotfiles config --local core.excludesFile=.dotfilesignore

I stored it as a BitBucket snippet, so now I can call it like this:

curl -Lks https://bitbucket.org/snippets/fsantacroce/qedGpx | /bin/bash 

Appendixes Some other things you would know.

Which dotfiles are worth to be versioned?

There's not a unique answer, it depends :)

These are some of the files I always include:

  • .dotfilesignore, of course
  • Shell config file: .zshrc
  • Aliases file: .zshrc_aliases
  • oh-my-zsh: .oh-my-zsh/ folder
  • my common Git config and aliases: .gitconfig
  • ... to be continued ...

Desktop environemnt and apps dotfiles

I usually version even my Desktop Environment dotfiles (I usually use KDE) and configs about all the K* apps I use; the folder to take in account are:

  • .kde/ for KDE related stuff
  • .config/ for the apps, even non-KDE ones
  • .local/share/

This command:

kf5-config --path config 

shows you the config folders used by your current KDE installation.

Unfortunately the restore of these files is not easy as override existing files; in KDE, thre's a well-know ticket that asks the dev team to implement something ad-hoc for this, but nothing has been implemented right know. Anyway, I usually hand-pick the one I need when migrating form a machine to a new one (eg. KDE global shortcuts, Dolphin customizations, etc.).

Which dotfiles needs to be ignored?

Yes, there are may cache, temp and machine related files there's no need to include (or that can be dangerous to include), like:

Zsh temp and cache files:
* .zsh_history
* .zsh_save

Bash temp and cache files:
* .bash_history
* .bash_logout

And others, like these:
* .config/chromium/*
* .config/session/*
* .config/pulse/*
* .m2/*, Maven repository cache
* .IntelliJ*/*, IntelliJ settings: I have a repo apart for those :)
* ... to be continued ...

I add new folders and files every time I discover something useless to track.

Do you version some /etc files, too?

I don't always version /etc files, but when I do, I version: * Nano config files: see official man page to understand how to deal with Nano config files, and wich files need to be versioned.
* ... to be continued ...

  • 1
    Thanks for the answer! This answer is a link-only, which is unfit for the SE format. The link can go dead, repository can go private, account can be deleted... Point being - if you don't post the full answer in here, it's bad. I suggest elaborating more by modifying your answer and providing a full answer. Welcome to the U&L SE! – MatthewRock Apr 24 at 20:30
  • 1
    @MatthewRock I followed your suggestion, adding the full answer. Thank you for pointing me out :) – Ferdinando Santacroce Apr 26 at 6:56
  • Thanks, that's better! I learned quite a few things. If you want to further improve the answer, you could tailor it to answer my specific concerns; I think that the --work-tree, --git-dir flags coupled with nountracked/excludes is the "meat" of the answer. Either way, thanks! – MatthewRock Apr 29 at 21:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.