0

I've got several folders with several thousand files, each folder roughly 3 - 10GB in size. Now, I'd like to tar those files inside the folders and each tar file should be roughly 1GB in size. Aftwards, I'd like to use Python to work on those tar files.

#!/bin/bash

dirlist=$(find $1 -mindepth 1 -maxdepth 1 -type d)
stored_date=$(date +%Y-%m-%d --date="-1 day")
#stored_date='2019-10-23'

for dir in $dirlist
do
(
    cd $dir
    tar_file=${PWD##*/}  
    tar_file="${tar_file}_${stored_date}.tar"

    echo "${tar_file}"

    tar -c $stored_date*.html --tape-length=1000M  -f ${tar_file}  --remove-files
)
done

It is working nicely to create the 1GB chunks - but with the ´--tape-length´ option, Python is running into all sorts of issues with

tarfile.ReadError: unexpected end of data

(plus: I'd like to work with Python as well on the files with are split over the edge of a tar archive)

Is there a Linux solution to this? I found star as opposed to tar but haven't tried it yet - I'd prefer to stay with the standard tar if possible.

7
  • 1
    taron Linux is the non-standard gtar that is not recommended as it causes all kinds of compatibility issues unless you use gtar to unpack as well. In noticeable cases, gtar is even unable to unpack own archives. star is closer to the standard than gtar...
    – schily
    Nov 13, 2019 at 10:31
  • @schily: Please cite some sources for your claims about gtar. My cursory internet search doesn't show any, and the default version of tar packaged by the likes of debian is gtar (debian is currently at v1.30), not star. Maybe your memory is some snapshot of an ancient version or weird corner-case? Nov 13, 2019 at 14:33
  • gtar has many of those corner cases that are never fixed even though they have been reported more than 20 years ago. If you live in a nutshell with only gtaravailable, is it obvious that you do not see most of the problems. Given that gtar needs at least 22 years to fix reported bugs, ancient is more or less the same as recent.
    – schily
    Nov 13, 2019 at 15:18
  • @schily - Does that mean you have no sources to cite, and no explanation why debian continues to package gtar? Does debian even package star as an alternative? What debian package should I use instead of their tar? Nov 13, 2019 at 15:29
  • Debian is not an OSS friendly distro and do not expect Debian to change their tar implementation as they have a vendor lock in to gtar, caused by the quirks from the strange gtar option names that are in conflict with other tars and the non-standard archives from gtar. User friendly distros include a recent star package. So if you do not like to compile things yourself, do not use Debian. BTW: there are of course plenty of sources for the problems caused by the non-compliant gtar archives, but you need to search for them and I do not keep a collection just to make people like you happy.
    – schily
    Nov 13, 2019 at 15:37

2 Answers 2

1

How about nesting a second loop within each of your directory loops to track the size of each file before it is appended to the tar file? Here's a schematic pseudo-code of what I mean:

max_size=$((1024*1024*1024))
total_size=0
for dir in $dirlist ; do
  for foo in $dir/*; do
    this_size="$(stat -c"%s" $foo)"
    if [ $(($total_size + $this_size)) -le $max_size ] ; then
      tar --append ... $foo
      total_size="$(($total_size + $this_size))"
    else
      # start new tar file here
      tar -c ... $foo
      total_size="$this_size"
    fi
  done
done
3
  • Warning: gtar --append has several issues and should not not be used since it could change the archive type in the middle.
    – schily
    Nov 12, 2019 at 14:10
  • @schily Oh? What alternative do you suggest in this case?
    – Chris
    Nov 13, 2019 at 7:23
  • @Chris have a look at my answer...
    – schily
    Nov 13, 2019 at 11:38
0

AFAIK, python does not understand the tar format, so you probably like to use a tar archive module that may not be 100% compatible to the tar archive standard. This is an important thing to consider, there are many half baken tar implementations that usually only support a sub-set of the official feature set.

I recommend to use the star option -tsize without bringing star into multi volume mode. This prevents star from splitting files in the middle, but it is unable to archive files that are greater than the specified tape size.

In case that the default shell "sh" is POSIX compliant with respect to support for "$((expr))", otherwise replace "sh -c" by "ksh -c" or similar.

...how about:

cd /tmp
star -C path/to/archivedir -c tsize=1G \
new-volume-script='cd /tmp;sh -c "mv vol-last.tar vol\$((\$1-1)).tar" script' \
f=vol-last.tar .

This keeps the created TAR archives in /tmp. You would need to manually rename vol-last.tar to the finally expected volume number. I may think of enhancing star to run the new volume script to the end of each archive, including the last one.

3
  • I've seen the star program as well and it looked tempting/good. I somehow was thinking to use out of the box tools from Debian, if they can do what I'm looking for. append with tar is working nicely for me so far - do you have any links to potential issues? And the Python side is working as well with import tarfile
    – Chris
    Nov 15, 2019 at 19:49
  • If you like to use best out of the box tools, you should switch to a OSS friendly distro and avoid Debian. If you like to get the gtar issues, have a look at their mailing list. There are frequent strange reports and the main problem is that gtar is not cleanly designed, so it does not control the various features and whether they are allowed with the current archive format.
    – schily
    Nov 16, 2019 at 11:16
  • I see, thank you. I'll try it out then.
    – Chris
    Nov 16, 2019 at 20:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .