Bash script to pack roughly 1GB of files into a tar archive

Question

I've got several folders with several thousand files, each folder roughly 3 - 10GB in size. Now, I'd like to tar those files inside the folders and each tar file should be roughly 1GB in size. Aftwards, I'd like to use Python to work on those tar files.

#!/bin/bash

dirlist=$(find $1 -mindepth 1 -maxdepth 1 -type d)
stored_date=$(date +%Y-%m-%d --date="-1 day")
#stored_date='2019-10-23'

for dir in $dirlist
do
(
    cd $dir
    tar_file=${PWD##*/}  
    tar_file="${tar_file}_${stored_date}.tar"

    echo "${tar_file}"

    tar -c $stored_date*.html --tape-length=1000M  -f ${tar_file}  --remove-files
)
done

It is working nicely to create the 1GB chunks - but with the ´--tape-length´ option, Python is running into all sorts of issues with

tarfile.ReadError: unexpected end of data

(plus: I'd like to work with Python as well on the files with are split over the edge of a tar archive)

Is there a Linux solution to this? I found star as opposed to tar but haven't tried it yet - I'd prefer to stay with the standard tar if possible.

Answer 1

How about nesting a second loop within each of your directory loops to track the size of each file before it is appended to the tar file? Here's a schematic pseudo-code of what I mean:

max_size=$((1024*1024*1024))
total_size=0
for dir in $dirlist ; do
  for foo in $dir/*; do
    this_size="$(stat -c"%s" $foo)"
    if [ $(($total_size + $this_size)) -le $max_size ] ; then
      tar --append ... $foo
      total_size="$(($total_size + $this_size))"
    else
      # start new tar file here
      tar -c ... $foo
      total_size="$this_size"
    fi
  done
done

Answer 2

AFAIK, python does not understand the tar format, so you probably like to use a tar archive module that may not be 100% compatible to the tar archive standard. This is an important thing to consider, there are many half baken tar implementations that usually only support a sub-set of the official feature set.

I recommend to use the star option -tsize without bringing star into multi volume mode. This prevents star from splitting files in the middle, but it is unable to archive files that are greater than the specified tape size.

In case that the default shell "sh" is POSIX compliant with respect to support for "$((expr))", otherwise replace "sh -c" by "ksh -c" or similar.

...how about:

cd /tmp
star -C path/to/archivedir -c tsize=1G \
new-volume-script='cd /tmp;sh -c "mv vol-last.tar vol\$((\$1-1)).tar" script' \
f=vol-last.tar .

This keeps the created TAR archives in /tmp. You would need to manually rename vol-last.tar to the finally expected volume number. I may think of enhancing star to run the new volume script to the end of each archive, including the last one.