8

I'm about to compress a large directory and I want to know how large, exactly, the resulting file will be.

I've tried using du:

$ tar -cv dir | du -h -
du: cannot access '-': No such file or directory

Then I tried using the file version of '-':

$ tar -cv dir | du -h /dev/stdin
1.0K

I'm certain this number isn't accurate. How can I get the size of stdin?

10

tl;dr: tar -cv dir | wc -c - | cut -d' ' -f 1 | awk '{print $1/1000"K"}'

du doesn't actually count the size of the file itself. It just asks the kernel to query the filesystem, which already keeps track of file size. This is why it's so fast. Because of that, and the fact that you're counting a stream, not a file, du doesn't work. My guess is that 1.0K is a hardcoded size for /dev/std* in the kernel.

The solution is to use wc -c, which counts bytes itself instead of querying the kernel:

$ tar -cv dir | wc -c

If you want output similar to du -h:

$ tar -cv dir | wc -c | awk '{print $1/1000"K"}'

The awk turns the number into a human-readable result.

  • 8
    Note that if you just omit wc's superfluous - then you don't need the subsequent cut command either. – Janis Jun 1 '15 at 2:11
  • 1
    1.0K is the block size of stdin. – Cody Allan Taylor Aug 25 '17 at 1:06
6

With GNU tar you can just do:

tar --totals -c . >/dev/null

...which will render output like...

Total bytes written: 5990400 (5.8MiB, 5.5GiB/s)

...on stderr. Similarly, with any tar (or stream) you can use dd to deliver a report on byte counts. This may or may not be preferable to wc, but dd defaults to a block-size of 512 bytes - which is identical to tar's block-size. If your system's PIPE_BUF is large enough, you can even expand dd's block-size to match tar's record size - which is 20 blocks, or 10240 bytes. Like this:

tar -c . | dd bs=bx20 >/dev/null
585+0 records in
585+0 records out
5990400 bytes (6.0 MB) copied, 0.0085661 s, 699 MB/s

This may or may not offer a more performant solution than wc.

In both the dd and tar use-cases you needn't actually dispose of the stream, though. I redirect to /dev/null above - but I could have as easily redirected to some file and still received the report on its size at the time it was written.

  • In case the file gets written to disk there would be no use to separately determine the size since that information is stored with the file. (+1) for the expected performance gain of dd (compared to wc). – Janis Jun 1 '15 at 4:53
  • 1
    @Janis - possibly true in the simplest case - but imagine rather that dd's output is passed on - to a compressor, say - and for whatever reason you find it desirable to know both the raw size of the archive and the compressed one. Also useful is to get an instant report on the record counts - tar is not just an archive, but a stream format. It can be used in ways other than just saving to a group of files to some other file. It is often useful for blocking a stream before modifying it. At each of those record boundaries is a whole block of NULs. – mikeserv Jun 1 '15 at 5:03
5

I'd suggest:

tar cf - dir | wc -c

A simple c (no leading - is required) is used to create a tar archive, f specifies an output file and - denotes that it be stdout. (Note that if you want just the size and there are many files beneath dir you may rather omit tar's v for performance reasons.)

  • @mikeserv; I seem to recall that I worked with tars in the past where specifying a tar-file (thus f and -) were required. - Just looked it up; without f - the tar assumed /etc/mt0 as default. – Janis Jun 1 '15 at 3:00
  • 1
    I've looked it up in a book that I used as a manual at that time, and I think was based on SysV R4. Few folks certainly recall what /etc/mt0 actually means - "magnetic tape" ;-) I'd be interested how Solaris' tar behaves (because Solaris is one of the contemporary OSes that are known to have still really old stuff in /bin). – Janis Jun 1 '15 at 4:32
  • @mikeserv; PS: The book mentions AT&T's "UNIX Programmers Manual Volumes 1, 2A, 2B" as source (but no manual date or UNIX release version, though; but must have been from the early 1980's, 1983, or so). – Janis Jun 1 '15 at 4:41
  • Have you ever seen this? Unrelated - but I just found it today, and thought you might like it. – mikeserv Jun 1 '15 at 5:10
  • I don't quite understand how this answer is different from mine. is it the presence of the -f flag to tar? – strugee Jun 1 '15 at 18:20
1

The wording of your question lends itself to the tar ... | wc -c answers above. I originally read your question with a silent assumption that you wanted the size to be reported while it was creating the tar file (perhaps tar's output was then being piped over a network link?).

In which case, I'd suggest pv -- pipe viewer. I've seen reference to it but have not yet had a chance to play with it.

References

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.