`du -sh` and `du -sb` gives different result when file being accessed?

Question

I set up a file server with samba and I'm copying files to it from another computer under the same LAN. I set up SSH server and connected to it with my phone to monitor the process.

This is what I run on my phone in OpenSSH client (the server has GNU Core Utilities installed):

# while :; do du -sh /media/samba; sleep 1; done
288M    /media/samba
289M    /media/samba
290M    /media/samba
...
^C
# while :; do du -sb /media/samba; sleep 1; done
328267292       /media/samba
328267292       /media/samba
328267292       /media/samba
... ...
(repeated ~30 times)
361334588       /media/samba
361334588       /media/samba
... ...
(repeated ~30 times)
403280532       /media/samba
403280532       /media/samba
^C
#

The computer (running Windows 10) reports a transfer speed of approx. 1000KB/s and all files are 30~35 MB in size.

From my observation, du is reporting file size with -h, but "occupied space" with -b.

So why does du report different sizes when a file is growing?
How does du handle options -h and -b?

Answer 1

From my observation, du is reporting file size with -h, but "occupied space" with -b.

That's the other way round, for GNU du, quoting du --help:

-b, --bytes equivalent to '--apparent-size --block-size=1'

du without --apparent-size reports the disk usage (the sum of the st_blocks fields of unique files of any type including directory, computed by traversing the directory tree), while with --apparent-size (implied by -b) the sum of the st_size (the amount of data that would be read from the files for regular files).

A possible explanation for the apparent size to grow in large chunks while the disk usage grows gradually is if the contents of files are not transferred sequentially from beginning to end, but randomly. That's common for instance for bittorrent transfer, where chunks are transferred from different peers¹.

For instance, if you open a new file for writing, then write a 4KiB chunk of data at 1GiB offset, you'll see the file size grow to 1GiB+4KiB, but the disk usage will be 4KiB:

$ zsh
% zmodload zsh/system
% exec 3> file
% du -h file
0       file
% du -s file
0       file

% sysseek -u3 $((1<<30)) # seek to 1GiB
% head -c 4K < /dev/urandom >&3 # write 4KiB there
% du -h file
4.0K    file
% du -b file
1073745920      file

It's a very large sparse file that occupies only 4KiB of disk space

% sysseek -u3 0
% head -c 4K < /dev/urandom >&3
% du -h file
8.0K    file
% du -b file
1073745920      file

After 4KiB have been written at the start, the file size hasn't changed, but the disk usage has increased by as much.

% head -c 4M < /dev/urandom >&3
% du -h file
4.1M    file
% du -b file
1073745920      file

Same again after writing another 4MiB.

---

^{¹ though bittorrent clients would generally pre-allocate the space with fallocate() beforehand to guarantee the space and also helps the system with its disk allocating decisions. They may not be able to do that if they have to access the files of SMB where fallocate() may not be supported there.}

Answer 2

The difference is the flag

-h is for human readable output which is going use megabytes -bis using bytes. And of course -s is for summarize

From the du man page

-b --bytes Print sizes in bytes, overriding the default block size (*note Block size::).

-h --human-readable Append a size letter such as 'M' for megabytes to each size. Powers of 1024 are used, not 1000; 'M' stands for 1,048,576 bytes. Use the '-H' or '--si' option if you prefer powers of 1000.

-s --summarize Display only a total for each argument.

So in your first call du -sh it it summarizing the total in a human readable form using megabytes to the power of 1024.

In your second call du -sb it is summarizing the argument using bytes.