Make tar from /dev/stdin file

Question

What I'm looking for is a way to pass in arbitrary data through STDIN, and have it get tar'd up as if its a normal file.

I tried this

$ echo test | tar -cf test.tar /dev/stdin

and that gives me a test.tar file with the contents dev/stdin. The stdin file is a link to /proc/self/fd/0.

What I want instead is for dev/stdin inside the TAR file to be a regular file with the text test inside. A similar thing happens with:

$ tar -cf test.tar <(echo test)

but the names are different.

Is this doable?

Answer 1

I don't think you can do what you want here. The problem with your approach is that tar deals in files and directory trees, which you're not providing it with commands such as this:

$ echo test | tar -cf test.tar /dev/sdtin

Even when you attempt to "wrap" your strings in temporary files using subshells such as this:

$ tar -cf test.tar <(echo test)

You can see your content is still being TAR'ed up using these temporary file descriptors:

$ tar tvf test.tar
lr-x------ vagrant/vagrant   0 2018-08-04 23:52 dev/fd/63 -> pipe:[102734]

If you're intent is just to compress strings, you need to get them into a file context. So you'd need to do something like this:

$ echo "test" > somefile && tar -cf /tmp/test.tar somefile

You can see the file's present inside of the TAR file:

$ tar tvf /tmp/test.tar
-rw-rw-r-- vagrant/vagrant   5 2018-08-05 00:00 somefile

Replicating data using tar

Most that have been working with Unix for several years will likely have seen this pattern:

$ (cd /; tar cf - .)|(cd /mnt/newroot; tar pxvf -)

I used to use this all the time to replicate data from one location to another. You can use this over SSH as well. Other methods are discussed in this U&L Q&A titled: clone root directory tree using busybox.

Backing up /

There's also this method if you're intent is to back up the entier HDD:

$ sudo tar -cvpzf backup.tar.gz --exclude=/backup.tar.gz --one-file-system /

References

• 4 Ways to Back Up Your Entire Hard Drive on Linux

Answer 2

You can create a tar file as a stream if you know the final size of what you want to stream (in bytes). It is then a matter of creating the initial 512 byte header before the data stream, and a trailing padding after it. Though it does depend on the exact variant of tar file, since there are a couple of different file formats. The following steps might work for you:

1. create a hole file of the same size as the data stream

   $ dd if=/dev/zero of=mystream bs=1 count=0 seek=$SIZE

   Note that mystream is now a "hole" file that doesn't actually occupy disk space even though it appears to have the same size, $SIZE, as the data stream.

2. use tar to create a header for a file named mystream of that size

   $ tar cf - mystream | head -c 512 > header

   Note that tar will want to create a file without hole, so it's therefore important to discard everything except the first 512 bytes. Then remove the hole file, so to avoid it being accidentally copied.

   $ rm mystream

3. create a padding trailer to fill out to a number of 512 blocks.

   $ head -c $((echo "scale=0; 512 - $SIZE % 512" | bc )) > trailer

4. make the tar stream of $STREAM (of size $SIZE) with header and trailer

   $ cat header $STREAM trailer

   Note, you'd probably want to stream that to something other than stdout.

Answer 3

With zsh, you can use the =(...) form of process substitution which uses a temporary file instead of a pipe. With GNU tar's --transform option, you can change the name of the archive member:

tar --transform='s/.*/test-file/' -zcf file.tar.gz =(echo test)

Would create a file.tar.gz with one test-file member with permissions 0600 and content test\n.

With bash (or zsh), you could do:

tar --transform='s/.*/test-file/' -zchf file.tar.gz /dev/stdin <<< "$(echo test)"

Here-strings also use temp files in those shells. Note however that command substitution strips every trailing newline character and adds one back and in bash would strip NUL bytes.