Generating sets of files that fit on a given media size for tar -T

Question

Can anyone suggest a script that will take as input the name of one or more directories and a media size, and output lists of files for input to tar using -T (assuming no compression)?

scdbackup/sdvdbackup sort of does this, but it's full of bloat that I don't need. So basically looking for something like this:

./splitTars file1 file2 .... 2.0T

where file can be a file or directory, and the last argument is the size of the media (e.g. 2TB). It should then output a file list for each tar archive and give a warning for files that are too big to fit on the media.

If nothing like this exists, one way to do it would be to create the list of files using find, re-arrange them in increasing or decreasing size, then start cutting the list up into pieces.

Answer 1

I think you have the knapsack problem for file sizes.

Which basically means, given a set of files, find the most optimal groups of files which can go into a backup media of a fixed size (our knapsack). Then you want to use the sets indicated to create tar archives, and transfer them one at a time from your local machine using whatever hard drive or flash drive you have.

I am outlining a sample solution in python - suit according to your programming skills.

1. Take all the input arguments in a python script, and find the file sizes of individual files or directories. You can possibly call a du -sm on each file or directory argument so that python doesn't have to do the hard work of finding individual sizes of directories.

2. Eliminate those over your media limit right away.

3. Add the rest to a list, and apply the knapsack algorithm to the entire set. Plenty of examples are around, like Mike's solution here.

4. Eliminate the ones which are added to the knapsack in one round, and return to step 3 with the leftover list.

5. Repeat steps 3-4-3 above until all the remaining items in the list can fit into one knapsack (i.e. the total size of leftover items is less than the media limit).

That should be it!

Answer 2

http://atlee.ca/blog/2006/01/26/splittar/

http://freshmeat.net/projects/splittar/

It's a start. Great minds think alike, even when naming their scripts :)

Answer 3

Finally got a chance to code it up myself

https://github.com/sabujp/splitTars

recently used it to split a 7TB+ directory across 5 LTO5 tapes.

Answer 4

find has an option to filter on size:

# find file1 file2 .... -size -2000G