1

I have a path of about 1 million folders with a total size of 300GB. When I run the du -sh * command, I get the error I mentioned in the title. How can I combine using xargs with this command or will xargs solve my problem? However, I still need a separate command to find folders larger than 20GB under the same path. Need help to use these two commands.

Path example: /data/dataold/exports/

The output should be like below (You can think of what I have stated below as the file and folder names under the main path I have specified on the above line ):

4.0K xyz.sh
12K asdasda.txt
10G QWERT
1G ASDFGH

Best regards.

Improve this question
3
  • 1
    ncdu is a very nice utility for this purpose.
    – Akash Karnatak
    Jun 7, 2021 at 18:27
  • 1
    Possibly of interest: accessing a million-sized directory is horribly slow in most filesystems (which is why usually deeper trees are used). If the filenames are not random, usually each filename is stored in xx/yy/zz/filename, where xx, yy, zz are taken from a fast hash of "filename" itself. That allows much faster access in almost all scenarios.
    – LSerni
    Jun 7, 2021 at 18:35
  • 1
    Would du -hd1 work? That tells du to list each first-level subdirectory itself, rather than first having the shell enumerate them and try to pass the entire list as arguments.
    – Gordon Davisson
    Jun 7, 2021 at 19:22

2 Answers 2

Reset to default
2

One option is to use find, but I suggest one of the other solutions shown below.

find /data/dataold/exports -mindepth 1 -maxdepth 1 -exec du -sh {} \;

Assuming that /data/dataold/exports contains the subdirectories

foo
bar
baz

(and maybe more), it will run

du -sh /data/dataold/exports/foo
du -sh /data/dataold/exports/bar
du -sh /data/dataold/exports/baz

etc.

Option -mindepth avoids running the du command for /data/dataold/exports, and -maxdepth avoids this for subdirectories of a subdirectory, e.g. for /data/dataold/exports/foo/something.

As suggested in cas' comment, you can use

find /data/dataold/exports -mindepth 1 -maxdepth 1 -exec du -sh {} \+

instead of ... -exec du -sh {} \;, if your version of find supports this. With + instead of ;, find will execute du once for every time the linux arg buffer is filled (approx 2MB) instead of executing it once for every file/directory. The former is much faster.

Citing Stéphane Chazelas' comment: "Note that calling several independent invocations of du (like with find -exec) may give different numbers if there are hard links across those top level directories, as they won't get deduplicated if found by different invocations of du."

With GNU du you can limit the depth to be displayed using option -d or --max-depth=N:

du -h -d 1 /data/dataold/exports

This will do the calculation for all subdirectories but limit the output to a depth of 1 below the starting point, so in the example shown above it should print the total size for

/data/dataold/exports/foo
/data/dataold/exports/bar
/data/dataold/exports/baz

etc. and for

/data/dataold/exports

The second solution, if available, should be preferred because it doesn't need to start a new du process for every subdirectory (in case of -exec ... \;) or for every set of subdirectories that fills up the arguments buffer (in case of -exec ... \+).

If your version of du does not support option -d you can use

du -h /data/dataold/exports

and filter the output to remove everything below the first level of subdirectories.

If you want to filter the output by numeric comparison, I suggest to omit the option -h. To avoid waiting for the slow file system access while testing the filtering, I suggest to redirect the output to a file, e.g.

du -d 1 /data/dataold/exports > outputfile

or

du -d 1 /data/dataold/exports 2>errors | tee outputfile

and process the contents of outputfile later.

Example:

awk '$1 > 20e9` outputfile

If your du doesn't support option -d you could use something like

du /data/dataold/exports > outputfile
awk '$1 > 20e9 && $1 != /\/.*\/.*\/.*\/.*\/.*/` outputfile

This will print all lines that have a number bigger than 20 * 10^9 in the first field and a value that does not contain 5 (or more) slashes in the second field. The number of slashes in the second condition is tailored to the starting directory /data/dataold/exports and will print e.g. /data/dataold/exports/foo but not e.g. /data/dataold/exports/foo/bar.

Improve this answer
12
  • Hi Bodo. Thank you for your answer. Can I use your first option with du -sh * , and do you know how can I use your first option for finding folders larger than 20GB? Best regards.
    – jrdba123
    Jun 7, 2021 at 18:35
  • @jrdba123 The main point is not to use * because this would be replaced with a list of all file and directory names, possibly resulting in the "Argument list too long" error. You should not ask two different questions in one post. I suggest to write a separate question about finding directories larger than 20GB with a link to this question. Use the gray "share" below your question to get the link.
    – Bodo
    Jun 7, 2021 at 18:44
  • Hi Bodo, the main problem is that there are 1 million folders under the path I specified, and I want to find the size of each one separately. Using du -sh alone only gives me the size for the main path, I know it's 300GB already.
    – jrdba123
    Jun 7, 2021 at 18:50
  • @jrdba123 Did you try the command? It should call du individually for every file or directory in /data/dataold/exports. Try find /data/dataold/exports -maxdepth 1 to see the output of file and directory names. Or do you want to list all subdirectories of /data/dataold, then use find /data/dataold -maxdepth ... You should not yet accept the answer if it is unclear or doesn't work for you.
    – Bodo
    Jun 7, 2021 at 18:56
  • There are 1 million folders. It takes hours to calculate, I'll reply again tomorrow. But I think it has to be du -sh * to provide the answer I want.
    – jrdba123
    Jun 7, 2021 at 19:03
0

Try not to summarize and then let du do the work. Apply sort with -human order then cap at a specific size:

du -h /data/dataold/exports/ | sort -h

use -r in sort for listing in reverse order, i.e. large to small files.

For selecting the size, a simple trick with awk would do:

du -h /data/dataold/exports/ | awk -F'G' '$1~/^[0-9]+$/ && $1>=20' | sort -h

I.e.: use G (from the GB size output) as field separator, check if the field is a number only (to exclude wrong matches with the G-separator) and then select numbers larger 20 only. Sorting is optional here.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .