Can anyone suggest a script that will take as input the name of one or more directories and a media size, and output lists of files for input to tar using -T (assuming no compression)?

scdbackup/sdvdbackup sort of does this, but it's full of bloat that I don't need. So basically looking for something like this:

./splitTars file1 file2 .... 2.0T

where file can be a file or directory, and the last argument is the size of the media (e.g. 2TB). It should then output a file list for each tar archive and give a warning for files that are too big to fit on the media.

If nothing like this exists, one way to do it would be to create the list of files using find, re-arrange them in increasing or decreasing size, then start cutting the list up into pieces.

  • I cannot understand why you mention -T option. This is used to take names of files from a file, and has nothing to do with media size. – enzotib Aug 12 '11 at 12:37
  • I'm tar'ing to tape, LTO5 tapes to be precise. The file list will be an argument to tar -T. I basically have multiple TB of data in multiple dirs that I need to put on the tapes and I don't want to split archives. – sabujp Aug 25 '11 at 21:29
  • atlee.ca/blog/2006/01/26/splittar freshmeat.net/projects/splittar It's a start. Great minds think alike, even when naming their scripts :) – sabujp Aug 26 '11 at 4:12
  • Finally got a chance to code it up myself github.com/sabujp/splitTars recently used it to split a 7TB+ directory across 5 LTO5 tapes. – sabujp Dec 3 '11 at 17:53

I think you have the knapsack problem for file sizes.

Which basically means, given a set of files, find the most optimal groups of files which can go into a backup media of a fixed size (our knapsack). Then you want to use the sets indicated to create tar archives, and transfer them one at a time from your local machine using whatever hard drive or flash drive you have.

I am outlining a sample solution in python - suit according to your programming skills.

  1. Take all the input arguments in a python script, and find the file sizes of individual files or directories. You can possibly call a du -sm on each file or directory argument so that python doesn't have to do the hard work of finding individual sizes of directories.

  2. Eliminate those over your media limit right away.

  3. Add the rest to a list, and apply the knapsack algorithm to the entire set. Plenty of examples are around, like Mike's solution here.

  4. Eliminate the ones which are added to the knapsack in one round, and return to step 3 with the leftover list.

  5. Repeat steps 3-4-3 above until all the remaining items in the list can fit into one knapsack (i.e. the total size of leftover items is less than the media limit).

That should be it!

| improve this answer | |

find has an option to filter on size:

# find file1 file2 .... -size -2000G
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.