Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Can anyone suggest a script that will take as input the name of one or more directories and a media size, and output lists of files for input to tar using -T (assuming no compression)?

scdbackup/sdvdbackup sort of does this, but it's full of bloat that I don't need. So basically looking for something like this:

./splitTars file1 file2 .... 2.0T

where file can be a file or directory, and the last argument is the size of the media (e.g. 2TB). It should then output a file list for each tar archive and give a warning for files that are too big to fit on the media.

If nothing like this exists, one way to do it would be to create the list of files using find, re-arrange them in increasing or decreasing size, then start cutting the list up into pieces.

share|improve this question

migrated from stackoverflow.com Aug 12 '11 at 12:13

This question came from our site for professional and enthusiast programmers.

    
I cannot understand why you mention -T option. This is used to take names of files from a file, and has nothing to do with media size. –  enzotib Aug 12 '11 at 12:37
    
I'm tar'ing to tape, LTO5 tapes to be precise. The file list will be an argument to tar -T. I basically have multiple TB of data in multiple dirs that I need to put on the tapes and I don't want to split archives. –  sabujp Aug 25 '11 at 21:29

4 Answers 4

I think you have the knapsack problem for file sizes.

Which basically means, given a set of files, find the most optimal groups of files which can go into a backup media of a fixed size (our knapsack). Then you want to use the sets indicated to create tar archives, and transfer them one at a time from your local machine using whatever hard drive or flash drive you have.

I am outlining a sample solution in python - suit according to your programming skills.

  1. Take all the input arguments in a python script, and find the file sizes of individual files or directories. You can possibly call a du -sm on each file or directory argument so that python doesn't have to do the hard work of finding individual sizes of directories.

  2. Eliminate those over your media limit right away.

  3. Add the rest to a list, and apply the knapsack algorithm to the entire set. Plenty of examples are around, like Mike's solution here.

  4. Eliminate the ones which are added to the knapsack in one round, and return to step 3 with the leftover list.

  5. Repeat steps 3-4-3 above until all the remaining items in the list can fit into one knapsack (i.e. the total size of leftover items is less than the media limit).

That should be it!

share|improve this answer

http://atlee.ca/blog/2006/01/26/splittar/

http://freshmeat.net/projects/splittar/

It's a start. Great minds think alike, even when naming their scripts :)

share|improve this answer

Finally got a chance to code it up myself

https://github.com/sabujp/splitTars

recently used it to split a 7TB+ directory across 5 LTO5 tapes.

share|improve this answer

find has an option to filter on size:

# find file1 file2 .... -size -2000G
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.