Compress many directories with up to 1000 files in archive

Question

Trying all morning, but can't figure it out. Maybe you know.

I'd like to compress many directories into zip files, so that each directory is its own zip file. And if there are more than 1000 files in that folder, then make them into multiple zip files with 1000 files per piece.

When archiving, is it possible to set the maximum number of files and split them accordingly. I'm trying to compress many files into manageable chunks, so that each could be worked with independently.

Solutions I now have:

# make each folder into its own archive
for i in */; do zip -r "${i%/}.zip" "$i"; done

# make archives of 500 files per piece
find . ! -name '*.zip' -type f | xargs -n 500 | awk '{system("zip myarch"NR".zip "$0)}'

Sources: Zip Archive With Limited Number of Files and command to zip multiple directories into individual zip files

What I'm missing is how to make it so that the files from the find stay limited to that subdirectory that they were found in. If there are 500 files in there, it should put only these 500 files in that directory, if there are 2,400 files in there, it should make 3 archives, first two 1,000 files a piece, and last only 400 files.

Something along these lines should work, I think. But something is still wrong.

#for all subfolders, find all files, take them in 500 name chunks, and zip them up into numbered archives.
for i in */; do find "${i%/}" -type f | xargs -n 500| awk '{system("zip ${i%/}"NR".zip "$0)}'; done

Help appreciated, thanks!

Update. This bash script should work I'd think, but I don't understand why the zip files still go acro

#!/bin/bash

for i in */; do 
    printf $i
    find "${i%/}" -type f | xargs -n 500 | awk '{system("zip marych${i%/}"NR".zip "$0)}'; 

Update2:

Fixed it. Some quotation mark issues were the problem:

for i in */; do find "${i%/}" -type f| xargs -n 500 | awk '{system("zip '${i%/}'"NR".zip "$0)}'; done

done

Answer 1

Copyright James Daniel Marrs Ritchey. This material was created for submission at 'Compress many directories with up to 1000 files in archive', but can also be alternatively obtained from 'https://snippetly.blogspot.com/2019/12/zip-files-recursively-from-directories.html' under the terms of any of the following licenses: Comprehensible Open License 3.0 (https://jamesdanielmarrsritchey.blogspot.com/2019/06/comprehensible-open-license-30.html), MIT (https://opensource.org/licenses/MIT).

You could create zips with a specified number of files per zip per folder using PHP. Start by getting an array containing all the directories you need to process. Next loop over each directory, and create an array of files. Now create a new array of these files which contains X number of files per array value. Then loop over this array and with 7zip and pass the value so 7zip can zip all those files.

My code below is set for 5 files per zip. You can change that to whatever number you want. You will need p7zip, find, and php7 installed.

Code:

<?php

$top_directory = '/home/user1/files_to_zip';

$destination = '/home/user1/zips';

#Get a list of folders, including the top directory

$dirs = shell_exec("find $top_directory -type d");

#Deal with each directory separately

$dirs = explode("\n", $dirs);

$dirs = array_filter($dirs);

foreach ($dirs as $dir){

    $zip_name = basename($dir);

    #Get a list of files within one directory

    $files = shell_exec("find $dir -maxdepth 1 -type f");

    #Split list of files for one directory into array

    $files = explode("\n", $files);

    $files = array_filter($files);

    if (empty($files) === TRUE){

        goto end;   

    }

    #Create a new array which holds 5 files per value (each value should be formatted as expected by the ZIP utility for a list of files)

    $n = 0;

    $list = array('');

    foreach ($files as $file){

        $n++;

        if ($n <= 5){

            $keys = array_keys($list);

            $key = end($keys);

            $list[$key] = $list[$key] . " " . $file;

        } else {

            $n = 1;

            $list[] = '';

            $keys = array_keys($list);

            $key = end($keys);

            $list[$key] = $list[$key] . " " . $file;

        }

    }

    #Create the zip archives of files for the one directory

    $n = 0;

    foreach ($list as $value){

        $n++;

        $zip_name_2 = "{$zip_name}_{$n}.zip";

        shell_exec("cd $destination && 7z a $zip_name_2 $value");

    }

    end:

}

?>

Answer 2

I use tar for this. As long as you have a folder with many directories insides, doing tar -czvf filename.tar directory would tar all folders in one tar file.

see: https://www.howtogeek.com/248780/how-to-compress-and-extract-files-using-the-tar-command-on-linux/