I'm a bit of a hoarder and my NAS is kinda the envy of my friends. But sharing my collection is a pain with everyone spread out many KMs across my city and Canada having garbage internet. I've had this idea of making "mini" disposable NAS units for them from random old PCs or rpi's, whatever is handy and dirt cheap. A single 1tb disk TOPS. To this end I've been trying to find a way to download files only once.

I want youtube-dl's --download-archive option in like aria2c, curl, lftp, rsync, or wget. I don't really care what the tool is as long as it's "popular" and packaged in most distros. If a transport is mandatory that doesn't bother me much but http or ssh would be preferred. Assuming a download is successful I don't want the file replaced if the dest file (disposable NAS copy) is deleted.

I've been googling for a bit and the closest I've come is using lftp and then processing the --log= output before feeding it back into --exclude-glob-from=FILE.

This seems like a lot of almost manual effort for something that I can't imagine I'm the first person to want. Has anyone ever done this before and can you share what tools you have used?


rsync should be found on any Linux distribution and can be used to sync a whole server, file system or directory in one go, with included or excluded file lists/patterns if you wish.

You can also do rsync over an ssh connection if you don't wish to setup the rsync service, though it will be somewhat slower. It has a --exclude-from=FILE option and a --exclude=PATTERN too (as well as include versions).

  • This was my first thought. But as you can see I talk about lftp --log= which would give me something "proper" to pipe into sed/awk/whatever. Going with rsync means trying to figure out the right amount of verbosity and then trowing the output into a file and trying to parse out useful crap. This would be even more work then what I'm currently complaining about. – Weedy Oct 10 '16 at 6:48
  • Apologies, I think I misunderstand your requirements. Why do you need to make an exclude list? O matter what the reason, the rsync output from the first time you run it will list the names of files it transfers. – Unbeliever Oct 10 '16 at 6:54
  • I would need an exclude list with a mirroring utility because I only want files downloaded once. Ever. – Weedy Oct 10 '16 at 14:26
  • OK, understood. Well the output of rsync with the -v option should be enough to build that list. There will be a few lines at the start and end of the output you don't need. There's no 'mega easy' way to do this that I can think of. – Unbeliever Oct 10 '16 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.