Zip all files in directory to multiple independent outputs

Question

I am trying to zip all files in a single directory into multiple independent output zip files. What I mean by this is that I do not want to produce a split archive; rather I would like multiple zip files. Example: if I have a directory with N files (e.g., file1.txt, file2.txt, ..., fileN.txt) and I want to zip the directory such that no single zip file contains more than 100 MB, I would like each zip file to be opened and unzipped independently of any other zip file. I've used the zip command in Linux in this way:

zip -r -s 100m archive.zip input_directory

This yields multiple files with extensions .z00, .z01, .z02, ..., and these have to be joined together eventually, along with the archive.zip file, in order to retrieve the zipped files. Instead, I would like for a program to output a series of archive<#>.zip files, where each file can be unzipped.

Answer 1

@JayCravens gave a functional solution but here is my KSH take on the problem.

To do a "proper" job would require being able to estimate the sizes of the compressed files and optimizing the grouping by size but that's too complicated so here's a simple version that just uses the uncompressed file sizes and does not optimize:

#!/usr/bin/ksh
# pack up to 100000 bytes into a zip file.
function size {
    nameref bytes=$2
    stat $1 | grep Size | read _ bytes rest
    # How would this be written in bash?
}
typeset -A fileSize
typeset nam
typeset bytes
cd sub
for nam in *
do
    size $nam bytes
    fileSize[$nam]=$bytes
done

typeset -Z4 zz       # this says four digits with leading zeros
typeset -a zipList   # an array of which files go into a zip file
integer sumSize=0    # the sum of the sizes for that zip file
for ((zz=0;zz<9999;zz++))  # if we have more than 9999, this will break
do
    sumSize=0
    zipList=()
    for nam in ${!fileSize[@]}
    do
        zipList+=( $nam )
        (( sumSize+= ${fileSize[$nam]} ))
        unset fileSize[$nam]
        (( sumSize > 100000 )) && break
    done
    if (( ${#zipList[@]} > 0 ))
    then
        echo "Z${zz}.zip ${zipList[@]}"
        zip Z${zz}.zip ${zipList[@]}
    else
        break
    fi
done

The size function returns the file size into the variable listed as the second argument.

This starts by accumulating the file sizes into the fileSize associative array.

This the loops for as many zip files as are required, accumulating the files until a max size is exceeded and passing the accumulated list to ZIP. As each file is added, that element of the fileSize array is deleted so we don't store files twice. We break out of the second loop when the fileSize array is empty which means there are no more files to be zipped.

Some explanations on ksh (and bash) syntax:

• stat $nam | grep Size | read _ bytes rest means, run the stat program on the file and ignore everything but the line that includes the Size. On my Redhat system, that line reads Size: 19546 Blocks: .... Everything after the size we don't care about so that goes into the rest variable. The Size string goes into _. This won't work on bash because after the pipe is a sub-process. On ksh the sub-processes go right to left.
• nameref bytes=$2 This is another that won't work on bash. It functions like a pointer. The function puts the value into "bytes" and it automagically shows up in the caller's variable listed as arg#2, in this case, bytes.
• for nam in ${!fileSize[@]} means to cycle through the index values of the array. Given those names, we can find the size of the file.
• (( sumSize+=${fileSize[$nam]} )) means to add the $nam element of fileSize to the existing sumSize value.
• (( sumSize > 100000 )) && break the && means when the first part evaluates true, execute the second part. In this case, we break out when the size is exceeded.

Obviously, this was my test version where I had small files. If you don't want as few as 100,000 bytes, make that number bigger.

Answer 2

It's true, @tink is correct. This is kind of complex.
You'll need basic BASH knowledge because it requires a script.

I've crossed this bridge a few times though, and I have a way.

#!/bin/bash

file_count=0
total_size=0
group_start=01

# find files with their sizes and paths
find "$(pwd)" -type f -name "*" -exec du -b {} + > /tmp/temp_names

readarray -t file_data < /tmp/temp_names

for entry in "${file_data[@]}"; do
    file_size=$(echo "$entry" | awk '{print $1}')
    file_path=$(echo "$entry" | awk '{print $2}')
    
    total_size=$((total_size + file_size))
    file_count=$((file_count + 1))

    if [[ "$total_size" -gt 104857600 ]]; then # 100MB in bytes
        chunk_size=$((total_size - file_size))
        echo "Files $group_start to $((file_count - 1)) total $chunk_size bytes $(($chunk_size / 1024 / 1024)) MB"
        group_start=$file_count
        total_size=$file_size # resets byte count to the size of the last file.
    fi
done

if [[ $total_size -gt 0 ]]; then
    echo "Files $group_start to $file_count total $total_size bytes $(($total_size / 1024 / 1024)) MB"
fi

rm /tmp/temp_names

exit 0

This is a little messy, but it does the job.
I'm sure it can be purposed to your archiving task.

---

Update: Well, it can be purposed for the task... but wow.

Now, that was an unforeseen adventure. Did I ever think it would take multiple array types, numbered indicies & FIFO's? Not at all, but that's just how it goes sometimes.

#!/bin/bash

# yad gui for directory selection
input=$(yad --title "Select Directory" --file --directory)

cd "$input" || { echo "Directory not found."; exit 1; }

file_count=0
total_size=0
group_start=0
group_num=0

# find files with their sizes and relative paths
find . -type f -name "*" -exec du -b {} + > /tmp/temp_names

readarray -t file_paths < <(awk '{$1=""; print substr($0,2)}' /tmp/temp_names)
readarray -t file_sizes < <(awk '{print $1}' /tmp/temp_names)

rm /tmp/temp_names

for ((i=0; i<${#file_paths[@]}; i++)); do
    file_size=${file_sizes[i]}
    file_path=${file_paths[i]}

    total_size=$((total_size + file_size))
    file_count=$((file_count + 1))

    if [[ "$total_size" -gt 104857600 ]]; then # 100MB in bytes
        chunk_size=$((total_size - file_size))
        group_length=$((file_count - group_start - 1))
        echo "Group $group_num: Files $((group_start + 1)) to $file_count Total: $(($chunk_size / 1024 / 1024)) MB ($chunk_size bytes)"

        # create a zip only if there are files
        if [[ $group_length -gt 0 ]]; then
            zip_file="group_$group_num.zip"
            temp_dir=$(mktemp -d)
            for ((j=group_start; j<file_count - 1; j++)); do
                cp --parents "${file_paths[j]}" "$temp_dir/"
            done
            (cd "$temp_dir" && zip -r "$zip_file" .)
            mv "$temp_dir/$zip_file" "$input/"
            rm -r "$temp_dir"
        fi
        group_num=$((group_num + 1))
        group_start=$((file_count - 1))
        total_size=$file_size
    fi
done

# handle the last group
if [[ $total_size -gt 0 ]]; then
    group_length=$((file_count - group_start))
    echo "Group $group_num: Files $((group_start + 1)) to $file_count Total: $(($total_size / 1024 / 1024)) MB ($total_size bytes)"

    if [[ $group_length -gt 0 ]]; then
        zip_file="group_$group_num.zip"
        temp_dir=$(mktemp -d)
        for ((j=group_start; j<file_count; j++)); do
            cp --parents "${file_paths[j]}" "$temp_dir/"
        done
        (cd "$temp_dir" && zip -r "$zip_file" .)
        mv "$temp_dir/$zip_file" "$input/"
        rm -r "$temp_dir"
    fi
fi

exit 0

It was a word-splitting nightmare.
Hero of the day was the almighty awk, with awk '{$1=""; print substr($0,2)}'.

Save as: chunk_archiver.sh
Make executable: chmod +x chunk_archiver.sh
Run ./chunk_archiver.sh to choose a directory.

---

Update 2: Properly preserves the directory structure.
Files will be zipped into 100 MB chunks, recursively, while preserving the relative path.

---

Answer 3

I don't believe that what you're asking can be achieved without serious programming effort.

zip doesn't have a feature that will allow individual parts to be unzipped independently. Your only chance is a wrapper that creates a list of the files in said directory, starts by creating a zip (and keeping track of its name/numbering it), adding files to it until it reaches 100MB (or slightly more and removes the last file added if it becomes to big), and carries on with its job on the next increment.

Answer 4

I ended up writing a script in Python because I was unable to solve this using Bash, although both @JayCravens and @user1683793 provided good answers that I'll use in the future. I understand that Python is not native Unix/Linux, so this solution may not fit in this community. Nevertheless, since I was able to solve the problem, I thought it would be useful to post my solution.

In this case, I instantiate two zip file objects, one used as "permanent" storage (where the final zip file will be written from), and one used as temporary storage for individual files where I can measure the compressed size of the next file. I then add the next file to the permanent zip file if the combined size (permanent + temp) is less than my bound. This is not recursive, although it could be modified to be. Currently, I provide a DIRECTORY and a file extension, then all files in DIRECTORY with that file extension are looped over until complete.

import os
from pathlib import Path
import zipfile

MAX_ZIP_FILE_SIZE = 500e6  # bytes
DIRECTORY = Path("/home/me/my_directory")
FILE_EXTENSION = "csv"

archive_name = "archive" # file prefix, e.g., <archive_name>_1.zip
zip_file_number = 1 # zip file number, e.g., archive_<zip_file_number>.zip

my_zip = zipfile.ZipFile(
    file=DIRECTORY.joinpath(f"{archive_name}_{zip_file_number}.zip"),
    mode="w",
    compression=zipfile.ZIP_DEFLATED,
    compresslevel=6,
)
file_list = list(DIRECTORY.glob(f"*.{FILE_EXTENSION}"))
ii = 0
for file_number, file in enumerate(file_list, start=1):
    if ii == 0:
        my_zip.write(filename=file, arcname=file.name)
        if os.path.getsize(my_zip.filename) >= MAX_ZIP_FILE_SIZE:
            my_zip.close()
            zip_file_number += 1
            my_zip = zipfile.ZipFile(
                file=DIRECTORY.joinpath(f"{archive_name}_{zip_file_number}.zip"),
                mode="w",
                compression=zipfile.ZIP_DEFLATED,
                compresslevel=6,
            )
            ii = 0
    else:
        tmp = zipfile.ZipFile(
            file=DIRECTORY.joinpath(f"{archive_name}_{zip_file_number + 1}.zip"),
            mode="w",
            compression=zipfile.ZIP_DEFLATED,
            compresslevel=6,
        )
        tmp.write(file, arcname=file.name)

        if (
            os.path.getsize(my_zip.filename) + os.path.getsize(tmp.filename)
        ) >= MAX_ZIP_FILE_SIZE:
            my_zip.close()
            my_zip = tmp
            zip_file_number += 1
        else:
            my_zip.write(filename=file, arcname=file.name)
    ii += 1
    print(f"Completed {file_number}/{len(file_list)}\t\t\t", end="\r")

try:
    my_zip.close()
except Exception as e:
    print(e)
try:
    tmp.close()
except Exception as e:
    print(e)
print("\n")