5

What I'm looking for is a way to pass in arbitrary data through STDIN, and have it get tar'd up as if its a normal file.

I tried this

$ echo test | tar -cf test.tar /dev/stdin

and that gives me a test.tar file with the contents dev/stdin. The stdin file is a link to /proc/self/fd/0.

What I want instead is for dev/stdin inside the TAR file to be a regular file with the text test inside. A similar thing happens with:

$ tar -cf test.tar <(echo test)

but the names are different.

Is this doable?

  • 1
    Does it need to be a tar file specifically? Other archiving tools (ar, zip) might do this but I don't know of a tar implementation that doesn't preserve device files. – Michael Homer Aug 5 '18 at 3:38
  • @MichaelHomer I'd prefer it to be a tar, but the real goal is to create a sparse file that's archived so it's portable. If this worked the way I wanted, the command I'd use would be pv /dev/sda | tar -cSf disk.tar /dev/stdin – Daffy Aug 5 '18 at 3:41
  • echo test ¦ tar -cf test.tar - – Rui F Ribeiro Aug 5 '18 at 3:46
  • i tard streams. unix.stackexchange.com/questions/151009/… – mikeserv Aug 5 '18 at 4:10
  • anyway, shitar might not be what you want, but you may get a clue there. i tried to touch on basic format requisites enough for me to understand later. – mikeserv Aug 5 '18 at 4:13
5

I don't think you can do what you want here. The problem with your approach is that tar deals in files and directory trees, which you're not providing it with commands such as this:

$ echo test | tar -cf test.tar /dev/sdtin

Even when you attempt to "wrap" your strings in temporary files using subshells such as this:

$ tar -cf test.tar <(echo test)

You can see your content is still being TAR'ed up using these temporary file descriptors:

$ tar tvf test.tar
lr-x------ vagrant/vagrant   0 2018-08-04 23:52 dev/fd/63 -> pipe:[102734]

If you're intent is just to compress strings, you need to get them into a file context. So you'd need to do something like this:

$ echo "test" > somefile && tar -cf /tmp/test.tar somefile

You can see the file's present inside of the TAR file:

$ tar tvf /tmp/test.tar
-rw-rw-r-- vagrant/vagrant   5 2018-08-05 00:00 somefile

Replicating data using tar

Most that have been working with Unix for several years will likely have seen this pattern:

$ (cd /; tar cf - .)|(cd /mnt/newroot; tar pxvf -)

I used to use this all the time to replicate data from one location to another. You can use this over SSH as well. Other methods are discussed in this U&L Q&A titled: clone root directory tree using busybox.

Backing up /

There's also this method if you're intent is to back up the entier HDD:

$ sudo tar -cvpzf backup.tar.gz --exclude=/backup.tar.gz --one-file-system /

References

2

You can create a tar file as a stream if you know the final size of what you want to stream (in bytes). It is then a matter of creating the initial 512 byte header before the data stream, and a trailing padding after it. Though it does depend on the exact variant of tar file, since there are a couple of different file formats. The following steps might work for you:

  1. create a hole file of the same size as the data stream

    $ dd if=/dev/zero of=mystream bs=1 count=0 seek=$SIZE

    Note that mystream is now a "hole" file that doesn't actually occupy disk space even though it appears to have the same size, $SIZE, as the data stream.

  2. use tar to create a header for a file named mystream of that size

    $ tar cf - mystream | head -c 512 > header

    Note that tar will want to create a file without hole, so it's therefore important to discard everything except the first 512 bytes. Then remove the hole file, so to avoid it being accidentally copied.

    $ rm mystream

  3. create a padding trailer to fill out to a number of 512 blocks.

    $ head -c $((echo "scale=0; 512 - $SIZE % 512" | bc )) > trailer

  4. make the tar stream of $STREAM (of size $SIZE) with header and trailer

    $ cat header $STREAM trailer

    Note, you'd probably want to stream that to something other than stdout.

  • gnu tar does have some kind of --sparse switch or something, but you also need to be careful about the blocking at the tail of the file - if it doesnt line up to a null padded block the tar will be technically trash (not actually a tar) but gnu tar tends to take that in stride and process it anyway. – mikeserv Aug 5 '18 at 5:39
2

With zsh, you can use the =(...) form of process substitution which uses a temporary file instead of a pipe. With GNU tar's --transform option, you can change the name of the archive member:

tar --transform='s/.*/test-file/' -zcf file.tar.gz =(echo test)

Would create a file.tar.gz with one test-file member with permissions 0600 and content test\n.

With bash (or zsh), you could do:

tar --transform='s/.*/test-file/' -zchf file.tar.gz /dev/stdin <<< "$(echo test)"

Here-strings also use temp files in those shells. Note however that command substitution strips every trailing newline character and adds one back and in bash would strip NUL bytes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.