3

Does anyone understand why I might get these results? Note the discrepency in file size between the two commands below:

$ ls -lh gauss_landmarks_0000.npy 
-rw-rw-r-- 1 dparks dparks 1.1G Aug 16 12:43 gauss_landmarks_0000.npy

$ du -h gauss_landmarks_0000.npy 
20M     gauss_landmarks_0000.npy

This occurs on the machine shown below:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.3 LTS
Release:        16.04
Codename:       xenial

The results on my own linux mint laptop appear as expected:

$ lsb_release -a
No LSB modules are available.
Distributor ID: LinuxMint
Description:    Linux Mint 18.3 Sylvia
Release:        18.3
Codename:       sylvia
2
  • can you share the result of blockdev --getbsz /dev/<partition>
    – Siva
    Aug 23, 2018 at 20:08
  • 4096 on blockdev. But it looks like the sparse file answer pegged it. Aug 23, 2018 at 20:36

2 Answers 2

9

It is probably a sparse file. That means that not all blocks are allocated and the file uses much less space than the file size suggests. On read the missing blocks will read as zero.

You can also use the -s option to ls to see the allocated size, it should be the same as the size reported by du.

Edit

If you have a file that you know or suspects contains many zero bytes but is not sparse, you can use cp --sparse=always to make it sparse, potentially saving a lot of disk space.

cp --sparse=always -p file new_file
6
  • Brilliant, ls -s does indeed show 20M. The data stored in that file is indeed quite sparse, so this makes perfect sense. I never knew the filesystem made provisions for sparse files. Thanks! This was a zfs filesystem, FYI. Aug 23, 2018 at 20:35
  • My laptop file system is ext4, where I don't see this occur. So it appears to be a features of zfs that isn't in a typical configuration of ext4 fs. Aug 23, 2018 at 20:41
  • Sparse files have been supported since the early days of Unix. Every native Linux file system should support them.
    – RalfFriedl
    Aug 23, 2018 at 20:45
  • Interesting. ls -sh gauss_landmarks_0000.npy shows 1.1GB on my linux mint based ext4 FS which sits on a LVM controlled volume. I'll read up on this in any case. Aug 23, 2018 at 20:52
  • 1
    I added a command to make the file sparse again.
    – RalfFriedl
    Aug 24, 2018 at 5:38
0

In your case the difference was indeed caused by a sparse file as explained in the other answer.

Another source of differences when an entire directory is involved are hard links. By default du does not count hard links multiple times, which can result in du reporting much less then what e.g. ll -sh shows.

To check whether hard links are involved you can run du -l so that it also counts hard links many times, or you can run ll -i <DIR> | sort. The left column will show the inode id, and if it is repeated the files are hard links of each other.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .