0

I work with many large files (~0.5-1TB) on a server. I would like a better way to do version tracking of these files on the server.

My limited understanding is that this may be possible with git LFS. I wouldn't actually be storing files, but rather using the storage that already exists.

Linux (or other OS) does keep track of the timestamp when I last accessed the files. But I would like to have more of a message, e.g. "I changed this data because XYZ" today (similar to 'git commit').

Is this a reasonable task for Git LFS? Or is this regular Git? (Also, is there another solution to version tracking in this case?)

  • What sort of files are they? Will a small change to the data result in a minor or major change to the underlying file (e.g. text file vs encrypted archive). If the changes will only be minor, then git could work well for your use case. If you are planning to only use git locally (no remote repo), then there is no point in using lfs, however if you are using a remote repo, lfs can reduce the size of the local repo. – Matt Feb 14 '18 at 13:09
  • git LFS will indeed store copies of the files, including historical values. So as time goes by, you'll fill your drive with old versions. It sounds like this may not be what you want, and instead just a change log. – Patrick Feb 14 '18 at 13:39
  • @Patrick "and instead just a change log" Could you add more details as to what you mean by this? How would one easily do this? – ShanZhengYang Feb 14 '18 at 18:13
2

Have you used git-annex?

https://git-annex.branchable.com/

git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space.

Git annex can manage a repo between multiple remotes and has different modes for each (client,transfer,backup,full backup and more). It also has an easier web interface, but you'll have to use the CLI often.

After you add files with git annex add ., it moves them to .git/annex/objects and replcaes all the files by symlinks. Then you can acquire/remove them (from the same computer or another remote) with get and drop, respectively. Committing them to the repo works exactly like a normal git repo but git-annex takes care of the files.

It's a barebones personal cloud storage solution.

  • 1
    Thanks! You like Huey Lewis and the News? Their early work was a little too new wave for my taste. But when Sports came out in '83, I think they really came into their own, commercially and artistically. – ShanZhengYang Feb 17 '18 at 18:33
  • 1
    Oh yeah! The whole album has a clear, crisp sound, and a new sheen of consummate professionalism that really gives the songs a big boost. He's been compared to Elvis Costello, but I think Huey has a far more bitter, cynical sense of humor. My question is - Why would someone reply to a man without marking his reply as the solution? – Patrick Bateman Feb 18 '18 at 4:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.