2

I am working on encrypting a stream with aespipe to a logfile, and it makes sense to append to the same file every time I run it, so I wanted to do something like

my_stream | aespipe -K my_key >> aes_logfile

I have already taken into account that aespipe writes out blocks of 512 bytes, and I made sure to pad my stream so that it always outputs data in increments of those sizes. I almost have my plan working, except that aespipe mangles the first byte of every 512 byte increment after the first file. For example, the following file that_file (the line ends with ~) is 512 bytes.

Hey StackOverflow, here is a  ~
small example. The lines are  ~
32 characters long (including ~
the trailing \n) and with     ~
aespipe you can encrypt this  ~
file with                     ~
                              ~
cat this_file | aespipe -K \  ~
 your_key_with_one_line.gpg \ ~
 > enc_file                   ~
                              ~
then you can concatenate onto ~
the encrypted file again with ~
 >> and get a new file.       ~
Note that this file is 512 B. ~
So it is the right block size ~

This is the output of the following:

# First go
cat that_file | aespipe -K my_key.gpg >  aes_logfile
# Second go
cat that_file | aespipe -K my_key.gpg >> aes_logfile

# Try decrypting
cat aes_logfile | aespipe -d -K my_key.gpg

annotated output:

Hey StackOverflow, here is a  ~
small example. The lines are  ~
32 characters long (including ~
the trailing \n) and with     ~
aespipe you can encrypt this  ~
file with                     ~
                              ~
cat this_file | aespipe -K \  ~
 your_key_with_one_line \     ~
 > enc_file                   ~
                              ~
then you can concatenate onto ~
the encrypted file again with ~
 >> and get a new file.       ~
Note that this file is 512 B. ~
So it is the right block size ~
Iey StackOverflow, here is a  ~    # <-- H became I!
small example. The lines are  ~
32 characters long (including ~
the trailing \n) and with     ~
aespipe you can encrypt this  ~
file with                     ~
                              ~
cat this_file | aespipe -K \  ~
 your_key_with_one_line \     ~
 > enc_file                   ~
                              ~
then you can concatenate onto ~
the encrypted file again with ~
 >> and get a new file.       ~
Note that this file is 512 B. ~
So it is the right block size ~

This happens every 512 bytes, and I have no idea why. Is there anything that can be done to fix this? (aespipe version 2.4d)

0

Well, I can reproduce it.

# dd count=1 if=/dev/zero | aespipe > aes_zero
Password: Error: Password must be at least 20 characters.
# dd count=1 if=/dev/zero | aespipe >> aes_zero
Password: Error: Password must be at least 20 characters.
# cat aes_zero | aespipe -d | hexdump -C
Password: Error: Password must be at least 20 characters.
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
          ^^
00000210  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400

Ciphertext:

# hexdump -C aes_zero
00000000  b3 a3 72 9b 18 5b 6b 73  e5 48 88 d1 8b 66 e3 10  |..r..[ks.H...f..|
00000010  a8 a0 ee d3 7e 08 91 86  6b f7 30 b2 ea 6a 58 0b  |....~...k.0..jX.|
[...]
000001e0  d9 2a 60 22 c9 4a 37 5c  47 21 65 0a bb f3 a8 7d  |.*`".J7\G!e....}|
000001f0  50 9f ef 38 9a c1 95 2b  ff 85 c7 16 cc 90 a7 18  |P..8...+........|
00000200  b3 a3 72 9b 18 5b 6b 73  e5 48 88 d1 8b 66 e3 10  |..r..[ks.H...f..|
00000210  a8 a0 ee d3 7e 08 91 86  6b f7 30 b2 ea 6a 58 0b  |....~...k.0..jX.|
[...]
000003e0  d9 2a 60 22 c9 4a 37 5c  47 21 65 0a bb f3 a8 7d  |.*`".J7\G!e....}|
000003f0  50 9f ef 38 9a c1 95 2b  ff 85 c7 16 cc 90 a7 18  |P..8...+........|
00000400

You can already tell that this is bad crypto, since the ciphertext repeats which is not supposed to happen at all.

Also nothing wrong with the ciphertext itself:

# dd count=1 if=aes_zero | md5sum
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00136312 s, 376 kB/s
f328ce6ff88545bc803b033561cbdffd  -
# dd count=1 skip=1 if=aes_zero | md5sum
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.00148213 s, 345 kB/s
f328ce6ff88545bc803b033561cbdffd  -

So this is identical and the wrong byte appears during decryption.

Actually what I naively expected is not off by one byte but random data following the first 512 bytes. Since for most usual encryption schemes, the IV should be different for each sector so as to avoid repeating ciphertext, and the result should be completely different.

Encrypting 1024 bytes in one go:

# dd count=2 if=/dev/zero | aespipe >> aes_zero2
Password: Error: Password must be at least 20 characters.
# hexdump -C aes_zero2
00000000  b3 a3 72 9b 18 5b 6b 73  e5 48 88 d1 8b 66 e3 10  |..r..[ks.H...f..|
00000010  a8 a0 ee d3 7e 08 91 86  6b f7 30 b2 ea 6a 58 0b  |....~...k.0..jX.|
...
000001e0  d9 2a 60 22 c9 4a 37 5c  47 21 65 0a bb f3 a8 7d  |.*`".J7\G!e....}|
000001f0  50 9f ef 38 9a c1 95 2b  ff 85 c7 16 cc 90 a7 18  |P..8...+........|
00000200  60 89 3e 37 87 1c 37 31  1a 11 50 b6 99 50 d3 74  |`.>7..71..P..P.t|
00000210  af a9 a2 30 3d e6 72 5f  f3 96 6d 3b 9e 5b 33 6f  |...0=.r_..m;.[3o|
...
000003e0  3f d3 2e 9a 18 ad 7a c9  5a ee 04 99 28 e6 af 3f  |?.....z.Z...(..?|
000003f0  a3 a9 71 be 1a 56 35 01  06 b9 57 dd fc 42 7c 47  |..q..V5...W..B|G|

As you can see, ciphertext repeats not at all. So your concatenating simply produces the wrong ciphertext (IV set incorrectly) and getting that much intact text back regardless is a big suprise and not good cryptography. You're not supposed to be able to move ciphertext around to a different offset and get a sensible result back (with only the first few bytes affected by wrong IV).


Try to reproduce the same with LUKS (aes-xts-plain64):

# truncate -s 1024 luks_zero
# losetup --find --show luks_zero
/dev/loop9
# cryptsetup open --type plain --cipher aes-xts-plain64 /dev/loop9 luks_zero
Enter passphrase for /dev/loop9: Error: Password must be at least 20 characters.
# dd if=/dev/zero of=/dev/mapper/luks_zero
dd: writing to '/dev/mapper/luks_zero': No space left on device
3+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000719156 s, 1.4 MB/s

This is zeroes, encrypted by LUKS. Duplicate ciphertext:

# sync
# dd count=1 seek=1 if=/dev/loop9 of=/dev/loop9
# hexdump -C /dev/loop9
00000000  1c 18 b1 e6 57 9c a1 60  8c 98 f0 d3 59 8b 97 0c  |....W..`....Y...|
00000010  bc fc 8a 62 68 52 51 f1  51 49 bf 21 2e f5 bc 84  |...bhRQ.QI.!....|
[...]
000001e0  4a e0 ef eb 8f 09 18 a8  73 95 0b 2c 59 01 69 1b  |J.......s..,Y.i.|
000001f0  92 71 e6 0d dd 3b 71 b5  22 f4 34 1c e1 4a 95 71  |.q...;q.".4..J.q|
00000200  1c 18 b1 e6 57 9c a1 60  8c 98 f0 d3 59 8b 97 0c  |....W..`....Y...|
00000210  bc fc 8a 62 68 52 51 f1  51 49 bf 21 2e f5 bc 84  |...bhRQ.QI.!....|
[...]
000003e0  4a e0 ef eb 8f 09 18 a8  73 95 0b 2c 59 01 69 1b  |J.......s..,Y.i.|
000003f0  92 71 e6 0d dd 3b 71 b5  22 f4 34 1c e1 4a 95 71  |.q...;q.".4..J.q|
00000400

Ciphertext is perfectly duplicated. Decryption result:

# hexdump -C /dev/mapper/luks_zero 
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200  b7 6c 5c 84 3d 7d dc 58  fc d5 ee 7f 20 c0 d5 09  |.l\.=}.X.... ...|
00000210  a8 78 f8 d8 38 4b 6a 24  bb b1 d2 65 b8 5c 2b ac  |.x..8Kj$...e.\+.|
[...]
000003e0  08 31 74 f4 59 92 3b 7f  0f fa b9 36 2e de 53 e2  |.1t.Y.;....6..S.|
000003f0  24 83 01 53 6e 56 dc a5  3a 3f 1b 4e d5 0a fd a9  |$..SnV..:?.N....|
00000400

512 Bytes of zero followed by complete garbage. That doesn't prove the crypto is good but it's not obviously lousy as the one you got from aespipe.


Now, since this is due to bad IV you can simply make sure to encrypt with correct IV in the first place. aespipe has the -O sectornumber flag for that:

   -O sectornumber
          Set IV offset in 512 byte units. Default is zero. Data is encrypted  in  512  byte  CBC
          chains  and  each  512  byte  chain  starts with IV whose computation depends on offset
          within the data. This option can be used to start encryption or decryption in middle of
          some existing encrypted disk image.

New way of encrypting and concatenating things:

# dd count=1 if=/dev/zero | aespipe -O 0 > aes_zero
Password: Error: Password must be at least 20 characters.
# dd count=1 if=/dev/zero | aespipe -O 1 >> aes_zero
Password: Error: Password must be at least 20 characters.

See that it matches the one we encrypted in one go:

# md5sum aes_zero aes_zero2 
3b0479e04a46a3d268dc4252e066d2b5  aes_zero
3b0479e04a46a3d268dc4252e066d2b5  aes_zero2

Verify decryption result is correct now:

# cat aes_zero | aespipe -d | hexdump -C
Password: Error: Password must be at least 20 characters.
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000400

The DNA matches, the timing works, everything checks out, ... well, it does not improve the crypto itself but fixes the immediate issue, maybe it's good enough for the purpose.

Dynamically deriving the correct -O sectornumber value from file size is left as an exercise to the reader.

|improve this answer|||||
  • Out of curiosity, any idea how one could tackle this type of problem without aespipe? (the issue is that the stream has no set termination time, and may run for days. So gpg doesn't seem to be a candidate.) – blaird Aug 27 '19 at 0:32
  • Well, as a 1:1 replacement (if it's really as simple as shown in example, all disk backed, no real pipe acrobatics) you could just use LUKS with the same level of discomfort (files, loop and crypt devices can all be resized dynamically and you just have to figure out the offset). You'd also not be limited to 512 byte boundaries then. For a proper solution I'm not sure right now, I just use full disk encryption then write files without additional encryption (well, the backup system might add its own encryption layer to things but I don't usually deal with this headache on a file-by-file basis). – frostschutz Aug 27 '19 at 0:42
  • gpg can naturally also work with pipes, it's just that the appending is a weird way to do things. There are some suggestions in this (dated) mailing list discussion: lists.gt.net/gnupg/users/45835 - their idea was to keep encrypted messages separate then concatenate in the decryption phase. Weird and inefficient but probably beats doing offset arithmetics. – frostschutz Aug 27 '19 at 0:59
  • ...and if you decide for this it might make a lot of sense to consolidate (re-encrypt) such an inefficient file into a single message whenever you logrotate.... – frostschutz Aug 27 '19 at 1:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.