What command can I use to create zips with a file number limit? I have a folder (no subfolders) of, say, 5000 files, so I would want a command that could divide that number and create 10 individual zip archives, each consisting of no more than 500 files.

I also don't want the resulting 10 zip files to be connected with each other, so that I can open them individually and won't need to open all 10 at the same time.

up vote 12 down vote accepted

You can use GNU parallel to do that as it can limit the number of elements to a job as well as provide a job number (for a unique zip archive name):

$ touch $(seq 20)
$ find . ! -name "*.zip" -type f -print0 | parallel -0 -N 5 zip arch{#} {}
  adding: 1 (stored 0%)
  adding: 10 (stored 0%)
  adding: 11 (stored 0%)
  adding: 12 (stored 0%)
  adding: 13 (stored 0%)
  adding: 14 (stored 0%)
  adding: 15 (stored 0%)
  adding: 16 (stored 0%)
  adding: 17 (stored 0%)
  adding: 18 (stored 0%)
  adding: 19 (stored 0%)
  adding: 2 (stored 0%)
  adding: 20 (stored 0%)
  adding: 3 (stored 0%)
  adding: 4 (stored 0%)
  adding: 5 (stored 0%)
  adding: 6 (stored 0%)
  adding: 7 (stored 0%)
  adding: 8 (stored 0%)
  adding: 9 (stored 0%)
$ ls
1   11  13  15  17  19  20  4  6  8  arch1.zip  arch3.zip
10  12  14  16  18  2   3   5  7  9  arch2.zip  arch4.zip

The option -N 5 limits the number of files to 5 per archive and is presented to zip in place of {}

The {#} (verbatim, not to be replaced by you during the invocation), is replaced by the job number, resulting in arch1.zip, arch2.zip etc.

The -print0 option to find and -0 option to parallel in tandem make sure that filenames with special characters are correctly handled.

  • I got this error: i.imgur.com/JoyPrfY.png From this command: find * ! -name "*.zip" -type f -print0 | parallel -0 -N 500 zip arch{13} {} – user8547 Nov 9 '14 at 16:07
  • @user8547 that is not GNU parallel, but the parallel included in moreutils, you best compile and install from source to get the latest security patches. ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2 – Anthon Nov 9 '14 at 16:15
  • 2
    @user8547 no, just run sudo apt-get install parallel. – terdon Nov 9 '14 at 16:20
  • 2
    @user8547 why arch{13}? You really need to use the # character. What shell are you using? – Anthon Nov 9 '14 at 16:32
  • 2
    @user8547 No that is the way to tell parallel to put the job number there, glad it worked out. – Anthon Nov 9 '14 at 16:37

A shell-only alternative: process batches of COUNT files via "${@:START:COUNT}" (range of positional parameters) and shift COUNT while incrementing a counter c to name the archives:

set -- *
c=1
while (($#)); do
  if [ $# -ge COUNT ]; then
    zip ${c}.zip "${@:1:COUNT}"
    c=$((c+1))
    shift COUNT
  else
    zip ${c}.zip "${@}"
    shift $#
  fi
done

The accepted answer worked perfectly fine for me. :) BUT, in case you don't have access to parallel (who knows why), here's an alternative I had come up with before:

find . ! -name '*.zip' -type f | xargs -n 500 | awk '{system("zip myarch"NR".zip "$0)}'

Which will create myarch1.zip, myarch2.zip, myarch3.zip, etc You might want to use the -0 trick Anthon suggested, if you have weird filenames.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.