0

We are supposed to store our ongoing projects on a rather small (~4TB) data server. Not surprisingly, it is constantly overflowing and people need to move off less recent files manually.
Is there an easy (aka standard command-line) way to find out which users take up the most space in a directory? i.e. summing up the size of all files in a directory and all sub-directories belonging to each user?

Edit: ideally not following symlinks

  • 1
    Have you considered enabling disk quotas? – Kusalananda Mar 26 at 9:16
0

To find the disk usage by all users on the disk, you can use the following script:

#! /bin/bash

DIRECTORY_TO_SCAN=/home

readarray -t user_list<<<"$(cat /etc/passwd | cut -d ':' -f 1)"

for u in "${user_list[@]}" ; do
        printf "Scanning for user: %30s" "$u"
        du -ch $(find "$DIRECTORY_TO_SCAN" -user "$u" 2>/dev/null) 2>/dev/null | sed -e '/total/!d;s/^/    /'
done

This script will scan the list of all users, find the files you can read and they own and compute the total space those files use.


(The part below is kept for the sake of completeness)

If you want to know the full size of a directory, you can use the du command:

du -sh directory1 directory2

will print the size of the directory and all its contents.

To actually solve your problem, you could use user (or group) quotas. This would allow you to get a detailled report of how much space each user/group is taking and set hard limits about how much space they may use.

  • I can't actually solve the problem since I am not the admin. What usually happens is: I realise that the storage is full again and have to ask around who could free up some space. The response I usually get is: "I barely use any space on the server, so I can't". The directory tree is not branched by users, but by projects since more than one person is working on the same project. – MechEng Mar 26 at 7:54
  • The problem with the non-admin approach is that if the permissions on a directory are set that you cannot read it/list its contents, you cannot know what is in there and cannot find the space it uses. If it is such a problem, I think the server admin should do something (like group quotas and set one group per project or set up directory quotas). – Simon Doppler Mar 26 at 8:17
  • The solution only working for dirs that I have permission to read would be fine. I agree that it should be the responsibility of the admin... but can we for now just pretend that this is not an option and I would like an answer to the initial question. – MechEng Mar 26 at 8:23
  • @MechEng I updated my answer with a script that does what you want – Simon Doppler Mar 26 at 8:54
0

In situations like this, I usually first use du -kx | sort -rn | less to list the directories in a largest-first order. That tells me where the biggest individual disk hogs are, so I'll know where to look. But as @SimonDoppler said, if you don't have at least r-x access to all the sub-directories, your listing will be incomplete: you'll only be able to get the sizes of directories you'll be able to access.

Remember: normally you only need to have write access to a directory to delete any files in it. Only if the directory has the sticky bit set (i.e. the last letter in the permission letter string is a t instead of x), you need be the owner of the file in order to delete it.

If there is no quota available, you may need to do something like this:

#!/bin/sh
if [ "$1" = "" ] || [ "$1" = "-h" ]
then
    echo "Usage: ${0##*/} <directory> <username(s)...>" >&2
    exit 64 # EX_USAGE
fi
if ! [ -d "$1" ]
then
    echo "ERROR: directory $1 does not exist" >&2
    exit 66 # EX_NOINPUT
fi

REPORTROOT="$1"
shift
for U in "$@"
do
    # Find all files under $REPORTROOT owned by a particular user,
    # get their sizes and sum them together.
    DISKUSE=$(find "$REPORTROOT" -type f -user "$U" \
        -exec stat -c %s {} \+ 2>/dev/null \
        | awk '{s+=$1} END {printf("%ld\n", s)}')
    # Display the result for this user.
    printf "%16s: %d\n" "$U" "$DISKUSE"
done

Note that running this may take a while.

The "calculate a sum of a list of numbers" awk one-liner is from this Stack Overflow post. Note the comments of the answer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.