3

I have a simple bash script to generate a compressed tar file and upload it to an external server for backup purposes. In addition to files on the server, I'd like to include dumps of a few mysql databases.

Due to disk space constraints, I can no longer afford to pre-generate the database dumps to a file on disk before generating the tar archive. What I'd like to do is somehow stream the dumps directly to tar.

My first attempt was to try creating a named pipe and having tar read that. For each database to dump, generate a pipe to a mysqldump process:

mkfifo $DB.sql
mysqldump --result-file=$DB.sql $DB &
echo $DB.sql >>$FILELIST

After generating a list of all files to backup:

tar c -P --files-from=$FILELIST | gzip | curl ...

I've discovered that this does not work as I expected. tar simply generates a pipe entry in the archive rather than reading the contents of the pipe.

I could not find any switches to tar to change the behaviour. Is there some way to get tar to read the pipe as if it were a file, or is there some other utility that would be capable of handling this type of situation?

0

I was thinking that you could use a fuse filesystem or something but then I realized that there is a problem with this in principle and that it is impossible to do.

You need tar's output to be streamed out to a remote system with curl, which means that tar must be able to write its output fully sequentially. But the tar file format requires that each member of the archive be preceeded with a header that describes it, and that header contains among other things the size of the member. But if the contents of the member file are themselves coming from a pipe then it will be impossible for tar to know in advance what the size will be and therefore impossible to write the header and proceed to the file data.

  • If there's enough space for one dump at a time, that would be possible, maybe with scriptfs (not sure whether it supports reporting the size in advance). – Gilles 'SO- stop being evil' May 27 '15 at 23:17
  • Yes, if there's enough space for one dump then something can be arranged somehow. I shouldn't say "impossible" so categorically. – Celada May 27 '15 at 23:37
  • I gave scriptfs a couple test runs. It reports the file's size as that of the script. tar did correctly get the output of the script, but truncated it at the reported script size. – kicken May 28 '15 at 0:24
0

Not quite sure that's what you want. To read from a pipe means to read from standard input. This is done using tar xf - (or tar xzf - for compressed streams), where the - indicates standard input. Similarly to write to standard output: tar cf - (or tar czf -).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.