4

My goal is to measure the file size (in MB or GB) for every file on my Linux system, recursively recording the filename and filesize, and piping the output into a tab-delimited text file.

What would be the appropriate command for this?

One could try a command like this:

ls -lhR > outputfile.txt

but there are reasons for me to think this is highly inefficient:

http://mywiki.wooledge.org/ParsingLs

Would it be more appropriate to use a for loop?

5

6 Answers 6

6
sudo du -h / 2>/dev/null > out.csv
  • Use sudo to make sure you get a read on all files.
  • Use -h with du for a "human readable" size format.
  • Do not display errors (2>/dev/null, where 2 is standard error)
  • Send results to a file (> out.csv).

This produces a list with file size and path, with a tab between values.

1
  • 2
    du reports disk usage, not file size (and for directories, the disk usage of the directory plus that of all the files and directories it contains, recursively). You need -a to include non-directory files. It will also skip some hard links. With GNU du, du -hal --apparent-size / may be better Mar 13, 2018 at 18:08
4

Try this (assuming GNU find and that file names don't contain double quote character):

{
    printf '%s\t%s\n' size path
    find / \( -path /proc -o -path /sys \) -prune -o -printf '%s\t"%P"\n' |
        numfmt --to=iec
} > outputfile.csv

Check

man find | less +/'-printf format'

or

info --ind=-printf find
0
2

Note: The command below does slightly more than you asked for, but may be extremely useful for other people.


Here is a command I worked out with a colleague and gave to application support teams who need to deal with full disks:

find / -xdev \! -path /var/log/lastlog -printf '%s\t%p\n' | sort -rn | head | cut -f2- | xargs -n1 ls -lh | awk '{print $5, $NF}'

Here's the same command with line breaks for easier reading:

find / -xdev \! -path /var/log/lastlog -printf '%s\t%p\n' |
  sort -rn | head | cut -f2- |
  xargs -n1 ls -lh | awk '{print $5, $NF}'

This command prints the 10 largest files on the root filesystem, along with the human-readable file size for each.

This command needs to be run as root for accurate results.

-xdev avoids traversing filesystem boundaries, which can be important if NFS might be slow.

/var/log/lastlog is ignored as it's a sparse file that falsely reports a huge size (i.e. it's not contributing to disk full issues).

-printf is specific to GNU find. In this case it prints the file size in bytes followed by the full path of the file.

You can actually leave everything off after the head command and the only thing you will lose is the human-readable file sizes.

So the following works just fine:

find / -xdev \! -path /var/log/lastlog -printf '%s\t%p\n' | sort -rn | head

This command does not work if you have filenames with newlines in them. But as this command is intended for manual handling by a human operator (not use in scripts), that's not crucial.


Now, to answer your particular question:

My goal is to measure the file size (in MB or GB) for every file on my Linux system, recursively recording the filename and filesize, and piping the output into a tab-delimited text file.

I'm going to ignore the MB or GB requirement and simply report the filesize in bytes, as it is MUCH, MUCH easier to do. See above command for how to do this if you really need to.

sudo find / -printf '%s\t%p\n' > outputfile.txt
2

You should check out ncdu which is the du command with ncurses which gets the file size of all the folders by default.

1

With GNU stat and bash globstar:

shopt -s globstar
stat --printf="%s\t%n\n" -- ** | numfmt -d $'\t' --to=iec >out.csv

This does include directories, but well, they're files too...

As @Patrick points out, this could result in argument list too long because of the **. One could use find instead:

sudo find / -type f -exec stat --printf="%s\t%n\n" -- {} \; | numfmt -d $'\t' --to=iec
2
  • The goal is to run this across the whole filesystem. Unless the filesystem is really tiny, this is going to blow up with "argument list too long".
    – phemmer
    Mar 14, 2018 at 1:40
  • @Patrick Good point, I didn't think of that.
    – m0dular
    Mar 14, 2018 at 15:21
1

On macOS, I found this useful:

sudo find . -exec stat -f %N%t%z {} ';' > ~/output.tsv

This almost certainly also works for Linux distributions.

One positive is that it processes one file at a time and will thus not suffer the 'argument list too long' issue.

2
  • 1
    See stat man page for macOS and for a standard linux OS. The syntax is different, especially for formatting the output, so the statement that it works for linux is not true. Also, -exec <command> {} + is preferable, when it can be used, that means when we don't need to write anything after {}, find will invoke more than one processes when needed (see man find -> -exec) so it's ok regarding the maximum byte size of the command/arguments list.
    – thanasisp
    Dec 13, 2020 at 6:36
  • Thanks for the considered feedback @thanasisp. Dec 14, 2020 at 6:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .