How can I split a pipe into 16 byte sequences?

Question

I'm writing a bash script to automate the decryption of a certain file; however the file is encrypted in 16 byte blocks, rather than simply encoded in its entirety.

Here's some psuedo code to briefly explain why I'm trying to do:

cat encrypted.bin | \
buffer and output 16 bytes when asked | \
openssl --args >> decrypted.bin

I essentially need the pipe to open and close in order to split up the data, and am unsure of how this could be done in bash. I've looked at split, but since the encrypted file could be several GBs, I don't want to perform that many disk writes.

I should also point out that I'd like to know if there's a way to do this without looping through multiple pipes.

Answer 1

I believe you can use instead dd

dd allows you to read from a file and sent the output to where you want specifying as well a blocksize.

from the man page

DESCRIPTION Copy a file, converting and formatting according to the operands.

   bs=BYTES
          read and write up to BYTES bytes at a time

So I guess that

dd if=encrypted.bin bs=16|openssl --args >> decrypetd.bin

should work for you. Although I did not tested it with openssl.

Update based on comment from jordamn (thanks jordamn)

The inline way does not pipe everything straith into openssl, instead pipes 16 block pieces.

#Get the file size in bytes
total=`ls -l encrypted.bin|awk '{print $5}'`
echo  $total;
ret=0;
i=0;
counter=0;
while [ $counter -lt $total ]
do
   #counter to know how many block we read
   counter=$(($i * 16))
   #skip is the number of block based on our setting to skip
   dd if=encrypted.bin skip=$i ibs=16 bs=16 count=1 status=none |openssl --args >> decrypit.bin
   i=$(($i+1))
done

Answer 2

With recent-ish Linux (or any system with recent GNU coreutils), call split --filter.

<decrypted.bin split -b 16 --filter='openssl --args "$FILE" >> decrypted.bin'

Independently-encrypted blocks sounds like ECB, so openssl enc -d aes-128-ecb may be what you're after.

If you have a strange mode that the openssl command line tool doesn't support, you may be better off using a tool that supports that strange mode. (If you describe the mode, I may be able to offer suggestions.)

You hopefully know that already, but if you aren't using a standard mode (other than ECB), your encryption is probably insecure.

Answer 3

Another way to split up and buffer the stream may be to use xxd - make a hexdump or do the reverse with its -c and -p options.

# test
n=0
printf '%s' {1..1000} | 
   xxd -p -c 16 | 
   while IFS="" read -r hexstr; do 
      n=$((n+1))
      printf '%s\n' "$n: $hexstr size: $((${#hexstr}/2)) bytes"
      printf '%s' "$hexstr" | xxd -p -r | wc -c
   done


# split up standard output stream in 16 byte blocks to be decrypted
xxd -p -c 16 encrypted.bin | 
   while IFS="" read -r hexstr; do 
      printf '%s' "$hexstr" | xxd -p -r | openssl --args >> decrypted.bin
   done

Answer 4

A way to split a pipe into 16 byte sequences without looping through multiple pipes may be to use tools such as cstream or mbuffer (in addition to the already mentioned split -b 16 approach).

cstream, for example, is similar to dd. It not only has a -b num option to set the block size used for read/write, but also a -B num option to buffer an input block up to num bytes before writing. The -n num option limits the total amount of data to be copied to num bytes.