Recursively output filename and file size for all subdirectories?

Question

My goal is to measure the file size (in MB or GB) for every file on my Linux system, recursively recording the filename and filesize, and piping the output into a tab-delimited text file.

What would be the appropriate command for this?

One could try a command like this:

ls -lhR > outputfile.txt

but there are reasons for me to think this is highly inefficient:

http://mywiki.wooledge.org/ParsingLs

Would it be more appropriate to use a for loop?

Answer 1

sudo du -h / 2>/dev/null > out.csv

• Use sudo to make sure you get a read on all files.
• Use -h with du for a "human readable" size format.
• Do not display errors (2>/dev/null, where 2 is standard error)
• Send results to a file (> out.csv).

This produces a list with file size and path, with a tab between values.

Answer 2

Try this (assuming GNU find and that file names don't contain double quote character):

{
    printf '%s\t%s\n' size path
    find / \( -path /proc -o -path /sys \) -prune -o -printf '%s\t"%P"\n' |
        numfmt --to=iec
} > outputfile.csv

Check

man find | less +/'-printf format'

or

info --ind=-printf find

Answer 3

Note: The command below does slightly more than you asked for, but may be extremely useful for other people.

---

Here is a command I worked out with a colleague and gave to application support teams who need to deal with full disks:

find / -xdev \! -path /var/log/lastlog -printf '%s\t%p\n' | sort -rn | head | cut -f2- | xargs -n1 ls -lh | awk '{print $5, $NF}'

Here's the same command with line breaks for easier reading:

find / -xdev \! -path /var/log/lastlog -printf '%s\t%p\n' |
  sort -rn | head | cut -f2- |
  xargs -n1 ls -lh | awk '{print $5, $NF}'

This command prints the 10 largest files on the root filesystem, along with the human-readable file size for each.

This command needs to be run as root for accurate results.

-xdev avoids traversing filesystem boundaries, which can be important if NFS might be slow.

/var/log/lastlog is ignored as it's a sparse file that falsely reports a huge size (i.e. it's not contributing to disk full issues).

-printf is specific to GNU find. In this case it prints the file size in bytes followed by the full path of the file.

You can actually leave everything off after the head command and the only thing you will lose is the human-readable file sizes.

So the following works just fine:

find / -xdev \! -path /var/log/lastlog -printf '%s\t%p\n' | sort -rn | head

This command does not work if you have filenames with newlines in them. But as this command is intended for manual handling by a human operator (not use in scripts), that's not crucial.

---

Now, to answer your particular question:

My goal is to measure the file size (in MB or GB) for every file on my Linux system, recursively recording the filename and filesize, and piping the output into a tab-delimited text file.

I'm going to ignore the MB or GB requirement and simply report the filesize in bytes, as it is MUCH, MUCH easier to do. See above command for how to do this if you really need to.

sudo find / -printf '%s\t%p\n' > outputfile.txt

Answer 4

You should check out ncdu which is the du command with ncurses which gets the file size of all the folders by default.

Answer 5

With GNU stat and bash globstar:

shopt -s globstar
stat --printf="%s\t%n\n" -- ** | numfmt -d $'\t' --to=iec >out.csv

This does include directories, but well, they're files too...

As @Patrick points out, this could result in argument list too long because of the **. One could use find instead:

sudo find / -type f -exec stat --printf="%s\t%n\n" -- {} \; | numfmt -d $'\t' --to=iec

Answer 6

On macOS, I found this useful:

sudo find . -exec stat -f %N%t%z {} ';' > ~/output.tsv

This almost certainly also works for Linux distributions.

One positive is that it processes one file at a time and will thus not suffer the 'argument list too long' issue.