Why does tar's handling of stdout and - differ?

Question

After a comment from OP, I discovered that /dev/stdout gives blocks of 10 KiB even after disabling buffering, but - does not. Why is this? I could not find anything regarding this in man tar nor man stdout.

Note that /dev/stdout goes to 00002800, not 000000a1. The output is correct, except is padded with null bytes.

> mkdir -p /tmp/747613
> cd /tmp/747613
> echo 747613 > file.txt

> tar czf /tmp/archive_tgz .
> hd /tmp/archive_tgz
00000000  1f 8b 08 00 00 00 00 00  00 03 ed d1 41 0a c2 30  |............A..0|
00000010  10 85 e1 ac 3d 45 4e 90  66 9a 49 72 9e 2e 22 08  |....=EN.f.Ir..".|
00000020  ea a2 46 f0 f8 a6 8b a2  9b 76 21 04 11 ff 6f f3  |..F......v!...o.|
00000030  20 33 90 81 e7 06 d3 9d  6f 72 8e 4b 4a 8e fe 3d  | 3......or.KJ..=|
00000040  57 46 54 43 6c 8f 69 6c  7b 22 da c6 36 f6 3f cd  |WFTCl.il{"..6.?.|
00000050  98 fb ad 4e b3 b5 2d cb  7c 9d 2e 65 7b 6f 7f fe  |...N..-.|..e{o..|
00000060  a3 dc 70 3c 9d 8b ab 8f  da ef 8f a5 e0 94 74 a7  |..p<..........t.|
00000070  ff bc f6 2f 41 a5 f5 1f  64 14 63 7d bf 93 5e fe  |.../A...d.c}..^.|
00000080  bc ff ac 39 49 38 7c fb  0c 00 00 00 00 00 00 00  |...9I8|.........|
00000090  00 00 00 00 00 00 1f 78  02 88 2a 27 ac 00 28 00  |.......x..*'..(.|
*
000000a1

> tar czf /dev/stdout . | hd
00000000  1f 8b 08 00 00 00 00 00  00 03 ed d1 41 0a c2 30  |............A..0|
00000010  10 85 e1 ac 3d 45 4e 90  66 9a 49 72 9e 2e 22 08  |....=EN.f.Ir..".|
00000020  ea a2 46 f0 f8 a6 8b a2  9b 76 21 04 11 ff 6f f3  |..F......v!...o.|
00000030  20 33 90 81 e7 06 d3 9d  6f 72 8e 4b 4a 8e fe 3d  | 3......or.KJ..=|
00000040  57 46 54 43 6c 8f 69 6c  7b 22 da c6 36 f6 3f cd  |WFTCl.il{"..6.?.|
00000050  98 fb ad 4e b3 b5 2d cb  7c 9d 2e 65 7b 6f 7f fe  |...N..-.|..e{o..|
00000060  a3 dc 70 3c 9d 8b ab 8f  da ef 8f a5 e0 94 74 a7  |..p<..........t.|
00000070  ff bc f6 2f 41 a5 f5 1f  64 14 63 7d bf 93 5e fe  |.../A...d.c}..^.|
00000080  bc ff ac 39 49 38 7c fb  0c 00 00 00 00 00 00 00  |...9I8|.........|
00000090  00 00 00 00 00 00 1f 78  02 88 2a 27 ac 00 28 00  |.......x..*'..(.|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002800

> # Even with buffering disabled
> stdbuf -i0 -o0 -e0 tar czf /dev/stdout . | stdbuf -i0 -o0 -e0 hd
00000000  1f 8b 08 00 00 00 00 00  00 03 ed d1 41 0a c2 30  |............A..0|
00000010  10 85 e1 ac 3d 45 4e 90  66 9a 49 72 9e 2e 22 08  |....=EN.f.Ir..".|
00000020  ea a2 46 f0 f8 a6 8b a2  9b 76 21 04 11 ff 6f f3  |..F......v!...o.|
00000030  20 33 90 81 e7 06 d3 9d  6f 72 8e 4b 4a 8e fe 3d  | 3......or.KJ..=|
00000040  57 46 54 43 6c 8f 69 6c  7b 22 da c6 36 f6 3f cd  |WFTCl.il{"..6.?.|
00000050  98 fb ad 4e b3 b5 2d cb  7c 9d 2e 65 7b 6f 7f fe  |...N..-.|..e{o..|
00000060  a3 dc 70 3c 9d 8b ab 8f  da ef 8f a5 e0 94 74 a7  |..p<..........t.|
00000070  ff bc f6 2f 41 a5 f5 1f  64 14 63 7d bf 93 5e fe  |.../A...d.c}..^.|
00000080  bc ff ac 39 49 38 7c fb  0c 00 00 00 00 00 00 00  |...9I8|.........|
00000090  00 00 00 00 00 00 1f 78  02 88 2a 27 ac 00 28 00  |.......x..*'..(.|
000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00002800

> # Works fine (- means stdout)
> tar czf - . | hd
00000000  1f 8b 08 00 00 00 00 00  00 03 ed d1 41 0a c2 30  |............A..0|
00000010  10 85 e1 ac 3d 45 4e 90  66 9a 49 72 9e 2e 22 08  |....=EN.f.Ir..".|
00000020  ea a2 46 f0 f8 a6 8b a2  9b 76 21 04 11 ff 6f f3  |..F......v!...o.|
00000030  20 33 90 81 e7 06 d3 9d  6f 72 8e 4b 4a 8e fe 3d  | 3......or.KJ..=|
00000040  57 46 54 43 6c 8f 69 6c  7b 22 da c6 36 f6 3f cd  |WFTCl.il{"..6.?.|
00000050  98 fb ad 4e b3 b5 2d cb  7c 9d 2e 65 7b 6f 7f fe  |...N..-.|..e{o..|
00000060  a3 dc 70 3c 9d 8b ab 8f  da ef 8f a5 e0 94 74 a7  |..p<..........t.|
00000070  ff bc f6 2f 41 a5 f5 1f  64 14 63 7d bf 93 5e fe  |.../A...d.c}..^.|
00000080  bc ff ac 39 49 38 7c fb  0c 00 00 00 00 00 00 00  |...9I8|.........|
00000090  00 00 00 00 00 00 1f 78  02 88 2a 27 ac 00 28 00  |.......x..*'..(.|
*
000000a1

Answer 1

The difference in behaviour comes from tar: when writing, it applies a “blocking factor”, which by default uses 10240-byte records (that’s 2800 in hexadecimal). This happens even when compressing, which is why some tarballs result in error messages from gzip when they are extracted. The behaviour is disabled at the end of the archive when it is written to a regular file, or to standard out (although the manual says it isn’t).

When writing to /dev/stdout, tar believes it’s writing to a device, and applies its blocking factor. You can verify this by changing the blocking factor:

$ tar czfb /dev/stdout 1 . | hd
00000000  1f 8b 08 00 00 00 00 00  00 03 ed d1 31 0e 83 30  |............1..0|
00000010  0c 05 50 cf 9c 22 27 08  76 b0 e3 f3 74 48 25 24  |..P.."'.v...tH%$|
00000020  26 08 88 e3 03 43 55 c4  00 53 8a 2a fc 16 0f b6  |&....CU..S.*....|
00000030  e4 2f 7d 5f 43 71 b8 52  91 6d 92 0a ee e7 07 10  |./}_Cq.R.m......|
00000040  73 13 31 10 62 04 24 0c  2c e0 a4 7c 34 80 71 c8  |s.1.b.$.,..|4.q.|
00000050  af de 39 18 72 9a d2 c9  dd d5 fe 4f f9 fa dd 76  |..9.r......O...v|
00000060  c9 e7 39 97 fb b1 15 1c  99 4f fa d7 43 ff a4 21  |..9......O..C..!|
00000070  80 c3 72 91 be 1e de bf  b2 46 6a aa bb 63 18 63  |..r......Fj..c.c|
00000080  8c f9 b1 05 02 0c 89 df  00 0a 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200