5

I was asked to create a mechanism to dump a (data on rest) encrypted mysql into an encrypted file. That mechanism must meet certain criteria, like:

  • Shall run under Linux
  • No write of unencrypted sql data, not even into temporary files
  • Generating of a md5 (or other hash) of the unencrypted dump stream
  • No use of process substitution since only sh shell is available

I would need something like that:

mysqldump | md5-tool >> dump.md5 | gzip | encryption-tool

My problem is that all hashing tools that I was able to find don't transport the stream, but only the checksum. The tool that comes closest is the BSD md5 with the -p option, but that is not available on Linux.

Does anyone know a tool with that this can accomplished?

Improve this question
4
  • @vlastimil, the bourne-shell tag refers to the historical Bourne shell, not just any not-so-featureful shell. They likely have some POSIX sh if they're on Linux.
    – ilkkachu
    Jun 9 at 14:15
  • @VlastimilBurián, the tag name could underline it better, what with people using "bourne shell" rather more loosely in practice.
    – ilkkachu
    Jun 9 at 14:29
  • 3
    Note that MD5 hasn't been cryptographically secure in quite some time. The question doesn't say what the hash will be used for, but since the question does mention cryptography, it seems worth mentioning.
    – Reid
    Jun 9 at 22:04
  • 2
    FYI, gzip is mostly obsolete except for compatibility. zstd compresses about as well but something like 5x faster on a single core, and its standard tool has options to use multiple cores.
    – Peter Cordes
    Jun 10 at 5:31

3 Answers 3

Reset to default
13

If it has to be done in standard sh syntax but you're on a system that supports /dev/fd/x, you can do the process substitution by hand:

{
  mysqldump | tee /dev/fd/3 | gzip | encryption-tool > dump.gz.enc
} 3>&1 | md5sum >> dump.md5

On systems without /dev/fd/x, use named pipes as shown by @ilkkachu. That's what shells with process substitution support generally also do¹ on systems that don't have /dev/fd/x.

¹ Except for earlier versions of ksh where that feature comes from in the mid-80s where process substitution support was conditional on the system supporting /dev/fd/x. Some other shells such as rc, zsh initially (a long time ago) only used temporary name pipes.

Improve this answer
2
  • Sicne the OP mentions Linux, /proc/self/fd/x is a drop-in replacement for /dev/md/x. Guaranteed working.
    – iBug
    Jun 12 at 5:54
  • Both /proc/self/fd/x and /dev/fd/x are guaranteed to work on a properly configured Linux system, and either could be unavailable if not (like when procfs is not mounted in /proc or at all). /dev/fd/x are the portable and canonical ones, the ones applications expect in general.
    – Stéphane Chazelas
    Jun 12 at 5:59
11

Without process substitution, so it works in a feature-limited POSIX shell, you can use tee with a named pipe. You'll just have to launch md5sum in the background manually:

mkfifo -m 600 p || exit
md5sum < p >> dump.md5 &
mysqldump | tee p | gzip | encrypt > final.backup
rm -f p
wait "$!" # make sure md5sum has finished computing the checksum and
          # written it before carrying on with the rest of the script.

Even though the named pipe has a name in the filesystem, it should work as a regular pipe, in that the files don't go the filesystem, let alone a disk. Though unless your database itself is encrypted, remnants of a stray backup copy on the disk are hardly critical. Of course you could make any temporary files, or the named pipe itself, on a tmpfs filesystem, thus making sure they only reside in memory.

Improve this answer
6
  • 1
    "you could make any temporary files, or the named pipe itself, on a tmpfs filesystem, thus making sure they only reside in memory" - if you're being this picky you'd want to disable swap too
    – Chris Davies
    Jun 9 at 14:36
  • .....and there is a strict limit of the amount data that can be buffered by the FIFO itself (ISTR some odd figure like 512 bytes)
    – symcbean
    Jun 9 at 16:34
  • 2
    @roaima, of course your swap is encrypted too if you care about stuff like this ;)
    – ilkkachu
    Jun 9 at 18:43
  • 1
    @symcbean, the size of any pipe buffers doesn't seem to be relevant here
    – ilkkachu
    Jun 9 at 18:44
  • @ikkachu User roaima raised the issue of unencrypted data in RAM - I'm just pointing out that there won't be much from a pipe.
    – symcbean
    Jun 10 at 18:37
9

In Bash, you can use tee and process substitution:

mysqldump | tee >(md5sum >> dump.md5) | gzip | encryption-tool

tee reads from its input, and writes to its output as well as one or more other files. So it writes to gzip and also to >(md5sum >> dump.md5).

What is >(md5sum >> dump.md5)? It's a process substitution - bash starts the command md5sum >> dump.md5, picks a spare file descriptor number for tee (in my case 63), hooks up tee's file descriptor 63 to md5sum's input, and then replace the process substitution with /dev/fd/63 so tee sees the command tee /dev/fd/63 and writes to its file descriptor 63 (as well as its output).

Improve this answer
3
  • 1
    > only sh shell is available
    – Vlastimil Burián
    Jun 9 at 13:21
  • 9
    But still a useful solution for anyone NOT encumbered with these constraints and a good illustration of how to use process substitution +1
    – symcbean
    Jun 9 at 16:37
  • And that trick can be useful to fork pipelines into multiples, too since tee allows to output to more than just 1 file/filedescriptor. Used that recently when i needed to grep 4 unrelated things into 4 separate files with only one download/extract
    – masterX244
    Jun 12 at 8:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .