I've got several folders with several thousand files, each folder roughly 3 - 10GB in size. Now, I'd like to tar those files inside the folders and each tar file should be roughly 1GB in size. Aftwards, I'd like to use Python to work on those tar files.
#!/bin/bash
dirlist=$(find $1 -mindepth 1 -maxdepth 1 -type d)
stored_date=$(date +%Y-%m-%d --date="-1 day")
#stored_date='2019-10-23'
for dir in $dirlist
do
(
cd $dir
tar_file=${PWD##*/}
tar_file="${tar_file}_${stored_date}.tar"
echo "${tar_file}"
tar -c $stored_date*.html --tape-length=1000M -f ${tar_file} --remove-files
)
done
It is working nicely to create the 1GB chunks - but with the ´--tape-length´ option, Python is running into all sorts of issues with
tarfile.ReadError: unexpected end of data
(plus: I'd like to work with Python as well on the files with are split over the edge of a tar archive)
Is there a Linux solution to this? I found star as opposed to tar but haven't tried it yet - I'd prefer to stay with the standard tar if possible.
tar
on Linux is the non-standardgtar
that is not recommended as it causes all kinds of compatibility issues unless you usegtar
to unpack as well. In noticeable cases,gtar
is even unable to unpack own archives.star
is closer to the standard thangtar
...gtar
. My cursory internet search doesn't show any, and the default version oftar
packaged by the likes of debian isgtar
(debian is currently at v1.30), notstar
. Maybe your memory is some snapshot of an ancient version or weird corner-case?gtar
has many of those corner cases that are never fixed even though they have been reported more than 20 years ago. If you live in a nutshell with onlygtar
available, is it obvious that you do not see most of the problems. Given thatgtar
needs at least 22 years to fix reported bugs, ancient is more or less the same as recent.gtar
? Does debian even packagestar
as an alternative? What debian package should I use instead of theirtar
?gtar
, caused by the quirks from the strangegtar
option names that are in conflict with other tars and the non-standard archives from gtar. User friendly distros include a recentstar
package. So if you do not like to compile things yourself, do not use Debian. BTW: there are of course plenty of sources for the problems caused by the non-compliant gtar archives, but you need to search for them and I do not keep a collection just to make people like you happy.