3

My goal is to measure the file size (in MB or GB) for every file on my Linux system, recursively recording the filename and filesize, and piping the output into a tab-delimited text file.

What would be the appropriate command for this?

One could try a command like this:

ls -lhR > outputfile.txt

but there are reasons for me to think this is highly inefficient:

http://mywiki.wooledge.org/ParsingLs

Would it be more appropriate to use a for loop?

4

Try this (assuming GNU find and that file names don't contain double quote character):

{
    printf '%s\t%s\n' size path
    find / \( -path /proc -o -path /sys \) -prune -o -printf '%s\t"%P"\n' |
        numfmt --to=iec
} > outputfile.csv

Check

man find | less +/'-printf format'

or

info --ind=-printf find
6
sudo du -h / 2>/dev/null > out.csv
  • Use sudo to make sure you get a read on all files.
  • Use -h with du for a "human readable" size format.
  • Do not display errors (2>/dev/null, where 2 is standard error)
  • Send results to a file (> out.csv).

This produces a list with file size and path, with a tab between values.

  • 2
    du reports disk usage, not file size (and for directories, the disk usage of the directory plus that of all the files and directories it contains, recursively). You need -a to include non-directory files. It will also skip some hard links. With GNU du, du -hal --apparent-size / may be better – Stéphane Chazelas Mar 13 '18 at 18:08
2

Note: The command below does slightly more than you asked for, but may be extremely useful for other people.


Here is a command I worked out with a colleague and gave to application support teams who need to deal with full disks:

find / -xdev \! -path /var/log/lastlog -printf '%s\t%p\n' | sort -rn | head | cut -f2- | xargs -n1 ls -lh | awk '{print $5, $NF}'

Here's the same command with line breaks for easier reading:

find / -xdev \! -path /var/log/lastlog -printf '%s\t%p\n' |
  sort -rn | head | cut -f2- |
  xargs -n1 ls -lh | awk '{print $5, $NF}'

This command prints the 10 largest files on the root filesystem, along with the human-readable file size for each.

This command needs to be run as root for accurate results.

-xdev avoids traversing filesystem boundaries, which can be important if NFS might be slow.

/var/log/lastlog is ignored as it's a sparse file that falsely reports a huge size (i.e. it's not contributing to disk full issues).

-printf is specific to GNU find. In this case it prints the file size in bytes followed by the full path of the file.

You can actually leave everything off after the head command and the only thing you will lose is the human-readable file sizes.

So the following works just fine:

find / -xdev \! -path /var/log/lastlog -printf '%s\t%p\n' | sort -rn | head

This command does not work if you have filenames with newlines in them. But as this command is intended for manual handling by a human operator (not use in scripts), that's not crucial.


Now, to answer your particular question:

My goal is to measure the file size (in MB or GB) for every file on my Linux system, recursively recording the filename and filesize, and piping the output into a tab-delimited text file.

I'm going to ignore the MB or GB requirement and simply report the filesize in bytes, as it is MUCH, MUCH easier to do. See above command for how to do this if you really need to.

sudo find / -printf '%s\t%p\n' > outputfile.txt
2

You should check out ncdu which is the du command with ncurses which gets the file size of all the folders by default.

1

With GNU stat and bash globstar:

shopt -s globstar
stat --printf="%s\t%n\n" -- ** | numfmt -d $'\t' --to=iec >out.csv

This does include directories, but well, they're files too...

As @Patrick points out, this could result in argument list too long because of the **. One could use find instead:

sudo find / -type f -exec stat --printf="%s\t%n\n" -- {} \; | numfmt -d $'\t' --to=iec
  • The goal is to run this across the whole filesystem. Unless the filesystem is really tiny, this is going to blow up with "argument list too long". – Patrick Mar 14 '18 at 1:40
  • @Patrick Good point, I didn't think of that. – m0dular Mar 14 '18 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.