6

I configured git-annex to keep track of a directory containing several GB of data. Its content is replicated on an S3 remote, so I can drop some files to free some space and get them back when I need them.

I also have another computer where I would like to do the same thing. This other computer already contains most of the files that are stored on the S3 remote.

How can I tell git-annex to init a new repository on this other computer without downloading from S3 the files that it can find in the local directory?

2 Answers 2

4
+100

EDIT 2015-04-30:

anarcat wrote up a guide for this use case that he posted in the comments for this answer.

EDIT 2013-05-31:

Thanks for accepting! gioele pointed out in the comments that reinject doesn't work in direct-mode git annex repos, which makes sense, and kinda takes the wind out of my answer. slm found an alternative method using git annex add, which it looks like gioele went with. Check out slm's answer for more information on this.

Original

Haven't heard of git annex; that's a cool tool!

Ok, from what I've read of it on git annex's website, on the new computer you can do git clone from the S3 repository and it won't be expensive, since it's only copying symlinks. Then cd into it and git annex init <reponamehere> as usual to make git annex aware of the repo.

As far as I understand it, the tricky thing you're trying to do is make git annex aware of that machine's local copies of the repo's data. I found a command on its man page called reinject that I think will do what you want:

git annex reinject /path/to/files/* /path/to/repo

You may want to add the --fast option which might disable the "fsck" (git annex's term for checksumming the files) which is automatically performed by reinject. That of course is a bit dangerous and may not even work: it's not clear whether reinject accepts --fast or not.

Also, you may need some kind of find one-liner if the files you want to reinject are more complicated than a single directory. Something like:

find /path/to/files/* -type f -exec bash -c 'echo $1 "/path/to/repo/${1#/path/to/files}"' -- '{}' \;

That one will (I think) echo the path of every file in /path/to/files while echoing a target path in your repo with /path/to/files stripped off. Replace echo with git annex reinject once you've run it and you're sure the output is doing what you intend. Using find in conjunction with bash -c brings great power and great responsibility :)

Source: working in a data analysis job that involves a hilarious amount of batch operations on files, and pretty much being in love with git.

8
  • I'm not sure reinject does what you think: git-annex.branchable.com/bugs/…. I read this to mean that you want to add new files on top of files that were already in the repo but are not accessible on any existing annex. BTW nice job sleuthing this so far. I opened a ticket asking this question over on the developers site here: git-annex.branchable.com/forum/…
    – slm
    May 27, 2013 at 17:50
  • I agree that's a more typical use case, but I don't think there's any reason for them to forbid the use of reinject when there are other copies available, so I hope they didn't! Also, I thought the second method in that forum thread was pretty hilarious...that would eliminate the need for the bash variable expansion I was doing quite nicely. May 27, 2013 at 21:17
  • Another problem with this solution is that it does not work in direct mode: "git-annex: You cannot run this command in a direct mode repository."
    – gioele
    May 28, 2013 at 12:29
  • Ah, that makes sense. Guess there's not much point to reinject when the filepath contains the data. This makes me cringe, but perhaps you can just move the files onto their corresponding locations in the repo, then run git annex fsck? May 29, 2013 at 20:25
  • 1
    i wrote a complete howto for this here git-annex.branchable.com/tips/…
    – anarcat
    Apr 25, 2014 at 13:30
4

I asked your question to the developer of git-annex over on their forums. Here's my/your question:

I have a question on setting up a new repo. Is it possible to initialize a new repo using files which are already local on say my laptop, instead of downloading them from a S3 remote repository? Sorry if this is unclear. But I'm trying to make use of the files that are local on my laptop so I don's have to go through the ordeal of downloading them from S3.

I should also mention that I don't have a complete set of all the files locally on the laptop vs. what's already in the S3 repository.

The author of git-annex, Joey, said you can use one of the following options:

  1. Use git annex reinject to feed in specific files.
  2. Check all the local files into a temporary directory. Either the git-annex Assistant or a manual git annex add will notice if these files have the same content as files already in the repository. The files will then be available in two locations, the temp directory and wherever it was checked into your repository before. You can then delete the temp directory.
2
  • It looks like git annex add is the only way to go in direct mode.
    – gioele
    May 31, 2013 at 12:51
  • What was the actual workflow you used? I'd be curious to capture that here for others down the road. I'm sure this question will come up again and it wasn't exactly clearly defined.
    – slm
    May 31, 2013 at 13:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .