I have a simple bash script to generate a compressed tar file and upload it to an external server for backup purposes. In addition to files on the server, I'd like to include dumps of a few mysql databases.
Due to disk space constraints, I can no longer afford to pre-generate the database dumps to a file on disk before generating the tar archive. What I'd like to do is somehow stream the dumps directly to tar.
My first attempt was to try creating a named pipe and having tar read that. For each database to dump, generate a pipe to a mysqldump process:
mkfifo $DB.sql mysqldump --result-file=$DB.sql $DB & echo $DB.sql >>$FILELIST
After generating a list of all files to backup:
tar c -P --files-from=$FILELIST | gzip | curl ...
I've discovered that this does not work as I expected. tar simply generates a pipe entry in the archive rather than reading the contents of the pipe.
I could not find any switches to tar to change the behaviour. Is there some way to get tar to read the pipe as if it were a file, or is there some other utility that would be capable of handling this type of situation?