1

Here is the scenario, I have 1000 files in a folder each ~3 MB. I want to zip all files into 50MB each zip and remove the original files.

NOTE: 50MB may contain 20 files or 10 files but that zip package should be <=50MB.

I want to transfer those files without data loss. File must be of format (tar/gzip/bzip). Please provide suggestions to me if we have any other way to overcome the data loss.

Need to create a shell script.

  • Curious as to what the 50MB limit is for. – Drew Khoury Aug 8 '13 at 14:43
  • If you don't want to use split and you're happy with some fuzzy logic you could create a bash script that looped through the list of files and zipped a fixed number each time. – Drew Khoury Aug 8 '13 at 14:56
1

One solution is to use the split command.

The split command will split an archive in to multiple files, doing all the hard work for you.

Here is an example:

tar -cvf - file1 file2 file3 | split --bytes=50m --suffix-length=4 --numeric-suffix - myarchive.tar.

And to untar:

cat myarchive.tar.* | tar xvf -
| improve this answer | |
  • Thanks, It works. if I have 50 MB files in tar format..it would be better in transferring files through network without dataloss – Nabob Aug 8 '13 at 15:14
  • That's great. When you're ready don't forget to select the best answer to your question. – Drew Khoury Aug 9 '13 at 7:50
1

The zip package supports bzip2 compression and --split.

Will bzip2 meet your requirements?

-s splitsize --split-size splitsize Enable creating a split archive and set the split size. A split archive is an archive that could be split over many files. As the archive is created, if the size of the archive reaches the specified split size, that split is closed and the next split opened.

...

Split size is a number optionally followed by a multiplier. Currently the number must be an integer. The multiplier can currently be one of k (kilobytes), m (megabytes), g (gigabytes), or t (terabytes). As 64k is the minimum split size, numbers without multipliers default to megabytes. For example, to create a split archive called foo with the contents of the bar directory with splits of 670 MB that might be useful for burning on CDs, the command:

zip -s 670m -r foo bar

could be used.

To ensure data integrity, rsync with checksum verification. It's much slower, but will calculate checksum on both sides of the transfer.

-c, --checksum

This changes the way rsync checks if the files have been changed and are in need of a transfer. Without this option, rsync uses a lqquick checkrq that (by default) checks if each file's size and time of last modification match between the sender and receiver. This option changes this to compare a 128-bit checksum for each file that has a matching size. Generating the checksums means that both sides will expend a lot of disk I/O reading all the data in the files in the transfer (and this is prior to any reading that will be done to transfer changed files), so this can slow things down significantly.

From: man zip, man rsync

| improve this answer | |
0

untested

cd /the/directory
files=(*)
i=0
z=0
create_zip=true
for ((i=0; i<${#files[@]}; i++)); do
    if $create_zip; then
        ((z++))
        zip_file=prefix.$z.zip
        create_zip=false
    fi
    # add the file
    zip $zip_file "${files[i]}"
    # check the size
    if (( $(stat -c %s $zip_file) >= 50000000 )); then
        # remove the previous file
        zip -d $zip_file "${files[i]}"
        create_zip=true
        # decrement the file index so this file gets added to the next zip
        ((i--))
    else
        echo rm "${files[i]}"            ### remove "echo" if it's OK
    fi
done
| improve this answer | |
  • So basically keep creating zip archives until one is larger than 50MB, then use the last one? Sounds like a great way to waste CPU cycles and disk IO. – Sammitch Aug 23 '13 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy