0

I am trying to tar, compress and encrypt a large backup folder but the process is slow as I believe gpg is single threaded. I have been tryping to use split to divide the root folder so instead of encrypting a single larger tar ball I would be doing many.

Is this even possible or what am I doing wrong?

tar -I pigz -cf - source \
| gpg -c -o destination  --passphrase-file xyz --batch --no-tty --cipher-algo AES256 --compress-algo none --no-options - 2 \
| split --additional-suffix=.tar.part -b 1G - "backup-test" >>/log.out
0
0

Pack

tar --create --format=posix --preserve-permissions --same-owner --directory $BASE/$name --to-stdout . |
    parallel --pipe --recend '' --keep-order --block-size 128M "xz -9 --check=sha256 | gpg --encrypt --recipient $RECIPIENT;echo bLoCk EnD" |
    pv > $TARGET/$FILENAME

Unpack

cat $TARGET/$FILENAME |
  parallel --pipe --recend 'bLoCk EnD\n' -N1 --keep-order --rrs 'gpg --decrypt | xz -d' |
  tar tv

-N1 is needed to make sure we pass a single record at a time. GnuPG does not support decrypting multiple merged records.

0

Thanks for your suggestions Ole. Further investigation showed the bottleneck to be Amazon lazy loading from S3. We had attached ebs volume from a snapshot. This is still slow have managed to speed it up using fio.

echo "File initialisation Starts Now: date" FILES=$(ls -d /mnt/database/files/database_*) echo $FILES

for f in $FILES do echo "Reading file $f" # This stops any output and lets the process carry on. sudo fio --opendir="$f" --rw=read --bs=128k --iodepth=32 --ioengine=libaio --direct=1 --name="$f" --openfiles=1000 >/dev/null 2>&1 &

done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.