I would like to speed up my archiving operations, I am usually doing 23 GiB (one Blu-Ray) backups.

I have found this: How to do large file parallel encryption using GnuPG and GNU parallel?

Since I don't understand this code at all (have never used parallel):

tar --create --format=posix --preserve-permissions --same-owner --directory $BASE/$name --to-stdout . |
    parallel --pipe --recend '' --keep-order --block-size 128M "xz -9 --check=sha256 |
    gpg --encrypt --recipient $RECIPIENT;echo bLoCk EnD" |

I would like to ask if anyone could kindly parse it for me. Thank you.

  • 2
    It makes it easier to understand what level of understanding you have, if you can answer these questions: Have you watched the intro videos pi.dk/1? Have you read the cheat sheet gnu.org/software/parallel/parallel_cheat.pdf? Have you walked through the tutorial man parallel_tutorial? Have you read chapter 1-2 of GNU Parallel 2018 (online: doi.org/10.5281/zenodo.1146014)? Have you read chapter 9 of the book? Have you read the EXAMPLE section of man parallel? Have you read the rest of man parallel? – Ole Tange Mar 16 '19 at 9:36
  • @OleTange Well, sadly No to all questions. Do you really think that this is relevant? For me to maybe include this info... Anyway, thank you for the perfect answer. I hope I will be fine now when adjusting it to my needs. – LinuxSecurityFreak Mar 16 '19 at 12:59

tar run command tar.

--create create a tar archive.

--format=posix use the POSIX format of tar archive. This means you can extract it on other systems that support the POSIX format.

--preserve-permissions keep the same permissions on the files

--same-owner keep the same owner of the file (only relevant when extracting as root)

--directory $BASE/$name change to the dir $BASE/$name before starting

--to-stdout instead of saving to a file, send output to stdout

. tar the whole directory

| pipe stdout to next command

parallel run parallel

--pipe use pipe mode, so input on stdin will be given as input on stdin to the command to run (and not as command line arguments, which is the normal mode).

--recend '' Normally GNU Parallel splits on \n. Disable that because input is not text, but binary data.

--keep-order Make sure the output of the first command run is printed before the output of the second command - even if the second command finishes first.

--block-size 128M Pass a block of 128 MB of data to the command.

"..." the command to run

| pipe stdout to next command

pv show the speed of which data is sent

> $TARGET/$FILENAME redirect stdout to $TARGET/$FILENAME

GNU Parallel starts this command for each 128MB block:

xz the command xz

-9 compress level 9

--check=sha256 include integrity check in the output to be able to catch bit errors (e.g. on faulty disks).

| pipe stdout to next command

gpg the GNU Privacy Guard command

--encrypt encrypt data coming on stdin

--recipient $RECIPIENT use $RECIPIENT's key for encryption

; command separator

echo bLoCk EnD print bLoCk EnD

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.