0

This has me puzzled. I think it should be easy, but I must be missing something, as the results are not consistent.

I am backing up a long list of files to multiple disks, using rsync, using a list sorted in chronological order, such that the earliest files go on the 1st disk, later ones on the 2nd, and so on.

I go through the list adding up the file sizes, in 4k blocks, and note the date of the last file that fits. Then I create a list, using "find -not -newer and -newer"

STARTDATE="-newer /tmp/filedate.1"
ENDDATE="-not -newer /tmp/filedate.2"
find $SRC -type f ${STARTDATE} ${ENDDATE} -printf '%P\n' | sort > ${TEMPFILE}

and feed that to rsync using "--files-from" to actually do the copying.

rsync -a --progress --verbose --prune-empty-dirs --files-from=${TEMPFILE} ${SRC} ${TARGET}

I want to find out accurately where to split the files, so the disks are filled to capacity.

What I have at the moment:

#%T is the modification time, @ is seconds, 
#%p is the path less the command line part, and %k is disk usage in 1k blocks
#MAXSIZE is number of 4k blocks available on disk
    find $SRC -printf "%T@\t%p\t%k\n" | sort -n | \
    awk -vMS="$MAXSIZE"  '
      BEGIN { FS = "\t";fnumber = 0 }
      {rtot+=int(($3+3)/4); #edit; changed to ceiling on AlexP's advice
       if (rtot<MS) {final=$2;filesize=rtot;}
       else {
            rtot=int(($3+3)/4); #edit; changed to ceiling on AlexP's advice
            fnumber++;
            printf "touch -r \"%s\" /tmp/filedate.%s\n", final, fnumber | "/bin/sh"
            print "Found point " fnumber ". (" final ") 4096 Blocks:"  filesize " Space Left:" (MS-filesize)*4
            }
      }
    '

The disk details are:

    #tune2fs -l /dev/sdzc1
    tune2fs 1.41.4 (27-Jan-2009)
    Filesystem volume name:   <none>
    Last mounted on:          /share/external/sdzc1
    Filesystem UUID:          f3f2e855-b198-4d47-b76f-6526d16b0820
    Filesystem magic number:  0xEF53
    Filesystem revision #:    1 (dynamic)
    Filesystem features:      has_journal ext_attr resize_inode filetype needs_recovery extent flex_bg sparse_super large_file
huge_file uninit_bg dir_nlink extra_isize
    Filesystem flags:         signed_directory_hash
    Default mount options:    (none)
    Filesystem state:         clean
    Errors behavior:          Continue
    Filesystem OS type:       Linux
    Inode count:              122101760
    Block count:              488378007
    Reserved block count:     0
    Free blocks:              89451
    Free inodes:              122088914
    First block:              0
    Block size:               4096
    Fragment size:            4096
    Reserved GDT blocks:      907
    Blocks per group:         32768
    Fragments per group:      32768
    Inodes per group:         8192
    Inode blocks per group:   512
    Flex block group size:    16
    Filesystem created:       Sun May 11 13:45:08 2014
    Last mount time:          Wed Dec  7 11:44:24 2016
    Last write time:          Wed Dec  7 11:44:24 2016
    Mount count:              68
    Maximum mount count:      28
    Last checked:             Fri Feb 20 02:06:42 2015
    Check interval:           15552000 (6 months)
    Next check after:         Wed Aug 19 02:06:42 2015
    Reserved blocks uid:      0 (user admin)
    Reserved blocks gid:      0 (group administrators)
    First inode:              11
    Inode size:               256
    Required extra isize:     28
    Desired extra isize:      28
    Journal inode:            8
    First orphan inode:       75890825
    Default directory hash:   half_md4
    Directory Hash Seed:      1c7f838c-8614-4af0-8506-cd3659e1e5ac
    Directory Magic Number:   0x514E4150
    Journal backup:           inode blocks

So, by my thinking, there are 488378007 blocks of 4096 bytes, and 122101760 inodes of 256 bytes. Therefore there should be (488378007 x 4096) - (122101760 x 256) bytes available to be written. i.e. 1,969,138,264,064 which is 1,922,986,586 kB.

df shows total 1,922,858,380 1k blocks, (128,206 difference), = 480,714,595 4k blocks.

Disregarding that, the end result is that when I actually copy the files, even using the lower figure as a start point, the "Space Left" reported from the awk output isn't equal to the actual space left, by varying amounts, sometimes even running out of space completely.

Where have I gone wrong with the logic? I know I can fudge it by just decreasing the MAXSIZE, but I'd really like to understand what I am missing!

ps. I'm running this as root, so reserved space is not relevant.

Just to clarify the actual question: should I be able to sum the file and directory sizes (in whole 4k blocks) to get the total disk usage?

Additional edit: just to confuse further I've just filled up(?) a drive and am getting this from df -k:

Filesystem      1K-blocks       Used Available Use% Mounted on
/dev/sdzb1     2927209048 2925317912         0 100% /share/external/sdzb1

2927209048-2925317912=1891136, or used to when I was at school!

  • Just an FYI: this looks like a really bad idea that is already giving you trouble even before it's runs. You should find another way to organize your space, perhaps combining both disks in an array. – Julie Pelletier Dec 10 '16 at 3:13
  • Note that much more detailed troubleshooting is needed to clarify your mistake if you really want to get to the bottom of your question. Do a small loop that copies each file individually and exits on error to see exactly what is going on. Post the df you get after that and a du -s /target – Julie Pelletier Dec 10 '16 at 3:17
  • You should only have one find to get the list of files, and always consider them in that order. You might also be mixing up modification and access times; find -printf %T@ will give modification times, not access times. – DepressedDaniel Dec 10 '16 at 4:09
  • @Julie The idea is meant to combine archive and backup in such a way that most of the time only the most recent disk is updated. These are a growing number of external disks. – u628898 Dec 10 '16 at 4:38
  • @DepressedDaniel, it's the mod time that I want. That part is working fine. – u628898 Dec 10 '16 at 4:38
1

Two observations:

  • You need to round up the number of blocks used by a file, not down; if a file is 8192+1 bytes long, that last byte will allocate a 4 KiB block. (Because the "fragment size" is 4 KiB.)

  • The disk space needed for a file is not necessarily equal to the number of data blocks needed to hold the number of bytes in the file. It can be a little bit larger (for larger files which need more metadata to map their allocated blocks), or smaller (for very small files, which can be wholly stored in their inode). Plus, as user Stephen Kitt mentions, there is the entire issue of sparse files, which can have sizes much larger than the space they take on disk, and which may cause interesting problems when archived or copies to a different filesystem.

  • Some filesystems may use some disk space for their own purposes. Also, filesystems tend to misbehave when the disk space used approaches capacity. You really should plan to fill your disks not more than about 98% or 99%.

  • You could also mention sparse files for your second point. As far as space for a file system's own purposes, that doesn't come out of the available space (except when you add a journal to a journal-less FS): after formatting, all the space reported by df is usable for files. You're right though that filling a FS up is a bas idea. (There are subtleties with snapshots, thin provisioning, btrfs etc. but that's beyond the scope of this question.) – Stephen Kitt Dec 10 '16 at 9:44
  • %k is supposed to return whole 1k blocks, so that divided by 4 should be 4k blocks. I think that I was just trying to be too exact, which apparently is far from straightforward. Your point about only using 98-99% is well taken. Also, up till today, I wasn't taking any account of space taken by directories, which appear to be left out of the 'find' unless specifically requested. – u628898 Dec 10 '16 at 10:43
  • @Kevin: 9 blocks of 1 KiB make 3 blocks of 4 KiB, not 2. – AlexP Dec 10 '16 at 11:11
  • @AlexP, Oops! Yes, quite right. – u628898 Dec 10 '16 at 13:22
0

I'm going to answer my own question, with thanks to all who contributed and steered my thoughts:

Because of the way space is allocated as the disk is being written, depending on the file size and type, sparse files, etc., it is actually very difficult, if not impossible, to accurately predict how much space will be taken in advance.

Directories that have had files deleted from them may be larger than when first created, and this space won't be recovered. (unless the dir is deleted and recreated) Empty directories take space.

'Find' doesn't report directories unless specifically requested.

Space is written in full blocks, blocksize can vary between disks and can be read from e2fsdump.

'df' reports no available space after about 98%, even though it reports less used blocks than available:

# df -B4k --sync
Filesystem      4K-blocks       Used Available Use% Mounted on
/dev/sdzb1      731802262  731493169         0 100% /share/external/sdzb1
/dev/sdzc1      731802262  717225328         0 100% /share/external/sdzc1

'du' reports different usage to 'df':

# du -B4k -s /share/external/sdzb1 /share/external/sdzc1
731441722       /share/external/sdzb1
717173881       /share/external/sdzc1

Nevertheless, it is possible, using the initial startpoint for available space:

Space = (Total blocks x blocksize) - (Total inodes x inode size)

and allowing 300,000 to 500,000 blocks margin, to arrive at a reasonably accurate result. (within about 1%)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.