378

I would like to list all files in the order of big to small in size and the files could be present anywhere in a certain folder.

3

12 Answers 12

443

Simply use something like:

ls -lS /path/to/folder/

Capital S.

This will sort files by size.

Also see:

man ls

-S     sort by file size

If you want to sort in reverse order, just add -r switch.

Update:

To exclude directories (and provided none of the file names or symlink targets contain newline characters):

ls -lS | grep -v '^d' 

Update 2:

I see now how it still shows symbolic links, which could be folders. Symbolic links always start with a letter l, as in link.

Change the command to filter for a -. This should only leave regular files:

ls -lS | grep '^-'

On my system this only shows regular files.

update 3:

To add recursion, I would leave the sorting of the lines to the sort command and tell it to use the 5th column to sort on.

ls -lR | grep '^-' | sort -k 5 -rn

-rn means Reverse and numeric to get the biggest files at the top. Down side of this command is that it does not show the full path of the files.

If you do need the full path of the files, use something like this:

find . -type f  -exec du -h {} + | sort -r -h

The find command will recursively find all files in all sub directories of . and call du -h (meaning disk usage -humanreadable) and then sort the output again. If your find/sort doesn't support -h, replace with du -k and sort -rn. Note that size and disk usage are not the same thing.

7
  • 7
    du gives the disk usage which is different from the file size. With (GNU) du -h, numerical sort won't work (you'll need the -h GNU option to sort). xargs expect a list of possibly quoted words as input so it won't work if filenames contain blanks or quoted characters. Nov 3, 2012 at 23:57
  • 1
    I see that -S does descending size order, what about ascending order?
    – demongolem
    Mar 8, 2016 at 15:40
  • 4
    Following @StéphaneChazelas, this works for me: find . -type f -print0 | xargs -0 du -h | sort -rh. If you just want say the largest 30 files: find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 30.
    – user7543
    May 16, 2016 at 11:13
  • 1
    Hi! Could you please explain the + | syntax in + | sort -r -h?
    – Victor
    Nov 9, 2018 at 11:44
  • 1
    For Update 3 scenario, human readable would be ls -lRh | grep '^-' | sort -k 5 -rh | less
    – hakamairi
    Sep 24, 2019 at 21:20
34

You could use something like find and sort.

find . -type f -ls | sort -r -n -k7

(the -ls option is not standard but is found in many find implementations, not only the GNU one. In GNU find and others, it displays something similar to ls -li with a few exceptions, for instance, files with ACLs are not marked with a +)

If the file names may contain newline characters, with GNU find and GNU sort:

find . -type f -ls -printf '\0' | sort -zk7rn | tr -d '\0'
3
  • 1
    This assumes "group" column (from -ls) not containing any spaces (hence -k 7 parameter), but it isn't necessarily the case.
    – kolistivra
    Nov 3, 2014 at 16:00
  • 1
    why not simply doing find ... -print0 | sort -zk7rn0 ? Sep 28, 2016 at 10:24
  • Thanks, this helped a lot when I was looking for some large hidden files to be deleted when my Mac was running out of space :)
    – Sam Chats
    Nov 2, 2019 at 16:49
17

With zsh and GNU ls:

ls -ldU -- **/*(.OL)

Where (.OL) is a glob qualifier, . to select regular files only, OL to reverse order by length (file size, o for ascending order, O for descending).

(note that older versions of zsh had issues with file sizes over 2^32).

Some operating systems have a limit on the size of the argument list passed to a command. In those cases, you'd need:

autoload -U zargs
zargs ./**/*(.OL) -- ls -ldU

If you just want the list of files and not the detailed output, just do:

print -rC1 -- **/*(N.OL)

If you want to include hidden files (whose name starts with a dot, except . and ..) and search in hidden directories as well, add the D globbing qualifier:

print -rC1 -- **/*(ND.OL)
0
11

List files by size ascending would be:

ls -lSr

The options are:

  • l: long, shows detailed user,group,other attributes, date, etc.
  • S: orders listing by size (descending by default)
  • r: reverses order of listing
0
9

Saying that "the files could be present anywhere in a certain folder" implies that you want to recursively descend all directories (folders) within the starting directory (folder). This is what find is meant to do:

find . -type f -exec ls -lSd {} +

This "finds" all files in the current working directory (.). For each file found, an ls process is run to sort the objects found in size order. The + terminator to the -exec causes multiple arguments to be passed as a list to ls. Unless your directory (folder) contains a very large number of files, you should have one list (and thus one process forked), leading to the result you desire.

1
  • 1
    this is nice as it allows you to use the -h modifier on ls to show nice file sizes
    – shmish111
    May 16, 2015 at 8:48
6

File list display in reverse order: ls -lSrh

For ascending order: ls -lSh

1
  • 1
    +1 for using the -h flag for human-friendly output. Jul 3, 2016 at 0:34
4

Try these, it works fine for me.

$ find /home/san -type f -printf '%s %p\n'| sort -nr | head -n 10

# find /root -type f -exec ls -lS {} + | head -n 10 | awk '{ print $5, $9 }'

Not perfect answer though but works to some extent

$ ls -lS |grep  '^-' | head -n 6 
3
  • 1
    awk works fine if filenames don't contain any spaces.
    – Eir Nym
    Mar 20, 2016 at 8:01
  • yes .. use 'sed' and replace space with underscore? if that helps Apr 12, 2016 at 9:29
  • No, I use sed to cut output
    – Eir Nym
    Apr 12, 2016 at 11:21
3

Adding to delh's answer and Stéphane Chazelas' comment...

find -print0 combined with xargs -0 adds support for blanks / spaces / whatnots.

du -h | sort -rn doesn't sort properly between different byte multiples, e.g. 1.1M will show after 128K, which is wrong.

sort -rh (--human-numeric-sort) takes care of that, but it only works on GNU's version.

The commands below will provide the desired output.

Human-readable, on GNU's sort / Linux:

find . -type f -print0 | xargs -0 du -h | sort -rh

In kilobyte units, on BSD / OSX / others:

find . -type f -print0 | xargs -0 du -k | sort -rn

For BSD / OSX, also see https://unix.stackexchange.com/a/188375/82895.

3

As a variation of the original question, if you want to see the cumulative size of files in the subdirectories:

#!/bin/bash
find ${1:-.} -maxdepth 1 -type d -exec du -sm {} \; | sort -nr

The sizes will be displayed in Megabytes ( the m in du -sm). Other values accepted by du are -k for kilobytes, -g for gigabytes. Using -h for human-readable display is not possible because it will break sorting.

Here is a version that uses sed to append the M for megabyte:

find ${1:-.} -maxdepth 1  -type d  -exec du -sm {} \; | sort -nr | sed -E 's/^([0-9]+)/\1M/g'

The directory to display is set via ${1:-.} which will use the first command line argument if provided, or use the current directory if called without arguments.

NOTE: This can take a long time with a lot of files. The option -type d will only list subdirectories and exclude files in the current folder; if you also want to see the files in the current folder then remove it.

Note: you might want to use ncdu instead which is available in most linux repos (on ubuntu/debian apt install ncdu) as well as on osx (brew install ncdu). Apart from visualising the sorted tree ncdu can also delete a selected folder with d.

2
  • +1 for ncdu. Nice cmd tool for browsing files sorted by size.
    – dr0i
    Nov 23, 2021 at 15:16
  • Another +1 for ncdu, nice to be reminded that the command-line doesn't always have to consist of wrestling with convoluted find chains to do something as basic as analysing filesizes in a directory. Also happened to already be included on my Cygwin installation, too. Apr 7, 2022 at 6:07
2

I wrote something to this extent a while back. You could pass an argument to specify how many files to list, or just type big, in what case you get 10.

big () { 
    NUM_FILES=10;
    if [ $1 ]; then
        NUM_FILES=$1;
    fi;
    du | sort -nr | head -n $NUM_FILES
}
1

All about finding, filtering, and sorting with find, based on file size

Main answer

I would like to list all files in the order of big to small in size

I like pretty output, so here's the solution I like most. Replace the . right after find with the path to any directory you want to check:

# Sort by size (in descending order), printing size in kiB
find . -type f -printf '%s\t%p\n' | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' | sort -rn

Example output when run in my eRCaGuy_hello_world/python dir here. Notice the pretty output, sorted by kiB size, in descending order:

    15.280273 kiB ./hello_george.py
    11.862305 kiB ./enum_practice.py
     6.858398 kiB ./slots_practice/slots_practice.py
     6.658203 kiB ./struct_packed.py
     6.295898 kiB ./textwrap_practice_1.py
     4.714844 kiB ./raw_bytes_practice.py
     3.784180 kiB ./README.md
     3.559570 kiB ./yaml_import/import_yaml_test.py
     3.413086 kiB ./socket_talk_to_ethernet_device.py
     2.354492 kiB ./auto_white_balance_img.py
     2.121094 kiB ./for_loop_basic_demo.py
     1.935547 kiB ./socket_udp_send_packed_struct_TODO.py
     1.692383 kiB ./binary_tree_traverse_and_print_TODO.py
     1.633789 kiB ./system_call_subprocess_pipe.py
     1.507812 kiB ./autogenerate_c_or_cpp_code.py
     1.322266 kiB ./binary_tree_TODO.py
     1.116211 kiB ./change_environment_variables_in_parent_process_TODO.py
     0.938477 kiB ./hello_world.py
     0.722656 kiB ./yaml_import/my_config1.yaml
     0.708984 kiB ./yaml_import/my_config2.yaml
     0.258789 kiB ./slots_practice/Link to 10. __slots__ Magic — Python Tips 0.1 documentation%%%%%+.desktop
     0.176758 kiB ./slots_practice/Class and Instance Attributes – Real Python.desktop
     0.137695 kiB ./autogenerated/myheader.h

Explanation:

  1. -type f finds only files

  2. -printf '%s\t%p\n' prints the file size in bytes (%s), followed by a tab (\t), followed by the filename (%p). From man find:

    %p     File's name.
    %s     File's size in bytes.
    
  3. The entire awk portion converts bytes to kiB. You can remove it to get bytes.

    To get pretty-formatted, right-aligned bytes, or kiB, MiB, GiB, or TiB, use the following:

    # bytes (ugly)
    find . -type f -printf '%s\t%p\n' | sort -rn
    
    # bytes (pretty)
    find . -type f -printf '%s\t%p\n' \
        | awk '{printf("%11i B ", $1); \
        for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
        | sort -rn
    
    # kiB (kibibytes) [what I use the most]
    find . -type f -printf '%s\t%p\n' \
        | awk '{printf("%13.6f kiB ", $1/(1024)); \
        for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
        | sort -rn
    
    # MiB (mebibytes)
    find . -type f -printf '%s\t%p\n' \
        | awk '{printf("%13.6f MiB ", $1/(1024*1024)); \
        for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
        | sort -rn
    
    # GiB (gibibytes)
    find . -type f -printf '%s\t%p\n' \
        | awk '{printf("%13.6f GiB ", $1/(1024*1024*1024)); \
        for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
        | sort -rn
    
    # TiB (tebibytes)
    find . -type f -printf '%s\t%p\n' \
        | awk '{printf("%13.6f TiB ", $1/(1024*1024*1024*1024)); \
        for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
        | sort -rn
    
  4. The sort -rn part does a reverse numerical sort to sort in descending order by size.

Off on tangents: list of (closely-related or overlapping) examples I find really useful:

1. (Same as previously above) Sort by size (in descending order), printing size in kiB

find . -type f -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
    | sort -rn

Sample output is shown above.

2. Sort by name

Pipe to sort -k 3 --ignore-case to sort by name instead.

Limitation: as it is currently written, this is only sorting by the first portion of the name up until the first space within the name, if there are spaces in the name.

find . -type f -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
    | sort -k 3 --ignore-case

Example output:

     1.507812 kiB ./autogenerate_c_or_cpp_code.py
     0.137695 kiB ./autogenerated/myheader.h
     2.354492 kiB ./auto_white_balance_img.py
     1.322266 kiB ./binary_tree_TODO.py
     1.692383 kiB ./binary_tree_traverse_and_print_TODO.py
     1.116211 kiB ./change_environment_variables_in_parent_process_TODO.py
    11.862305 kiB ./enum_practice.py
     2.121094 kiB ./for_loop_basic_demo.py
    15.280273 kiB ./hello_george.py
     0.938477 kiB ./hello_world.py
     4.714844 kiB ./raw_bytes_practice.py
     3.784180 kiB ./README.md
     0.176758 kiB ./slots_practice/Class and Instance Attributes – Real Python.desktop
     0.258789 kiB ./slots_practice/Link to 10. __slots__ Magic — Python Tips 0.1 documentation%%%%%+.desktop
     6.858398 kiB ./slots_practice/slots_practice.py
     3.413086 kiB ./socket_talk_to_ethernet_device.py
     1.935547 kiB ./socket_udp_send_packed_struct_TODO.py
     6.658203 kiB ./struct_packed.py
     1.633789 kiB ./system_call_subprocess_pipe.py
     6.295898 kiB ./textwrap_practice_1.py
     3.559570 kiB ./yaml_import/import_yaml_test.py
     0.722656 kiB ./yaml_import/my_config1.yaml
     0.708984 kiB ./yaml_import/my_config2.yaml

3. Only show files greater than a certain size (-size +<number>)

Add -size +3k to find to show only files > 3kiB, or -size +3072c to show only files > 3072 bytes (same thing in this case). From man find:

-size n[cwbkMG]
      File uses n units of space, rounding up.  The following suffixes can be used:

      `b'    for 512-byte blocks (this is the default if no suffix is used)

      `c'    for bytes

      `w'    for two-byte words

      `k'    for Kibibytes (KiB, units of 1024 bytes)

      `M'    for Mebibytes (MiB, units of 1024 * 1024 = 1048576 bytes)

      `G'    for Gibibytes (GiB, units of 1024 * 1024 * 1024 = 1073741824 bytes)

Examples

  1. > 3 kiB (-size +3k):

    find . -type f -size +3k -printf '%s\t%p\n' \
        | awk '{printf("%13.6f kiB ", $1/(1024)); \
        for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
        | sort -rn
    

    Output:

    15.280273 kiB ./hello_george.py
    11.862305 kiB ./enum_practice.py
     6.858398 kiB ./slots_practice/slots_practice.py
     6.658203 kiB ./struct_packed.py
     6.295898 kiB ./textwrap_practice_1.py
     4.714844 kiB ./raw_bytes_practice.py
     3.784180 kiB ./README.md
     3.559570 kiB ./yaml_import/import_yaml_test.py
     3.413086 kiB ./socket_talk_to_ethernet_device.py
    
  2. > 3072 bytes (-size +3072c):

    find . -type f -size +3072c -printf '%s\t%p\n' \
        | awk '{printf("%13.6f kiB ", $1/(1024)); \
        for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
        | sort -rn
    

    Output: exact same as just above

4. Put quotes around the path

Sorted by name in this case:

find . -type f -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); printf("\""); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\"\n", $NF)}' \
    | sort -k 3 --ignore-case

Output:

     1.507812 kiB "./autogenerate_c_or_cpp_code.py"
     0.137695 kiB "./autogenerated/myheader.h"
     2.354492 kiB "./auto_white_balance_img.py"
     1.322266 kiB "./binary_tree_TODO.py"
     1.692383 kiB "./binary_tree_traverse_and_print_TODO.py"
     1.116211 kiB "./change_environment_variables_in_parent_process_TODO.py"
    11.862305 kiB "./enum_practice.py"
     2.121094 kiB "./for_loop_basic_demo.py"
    15.280273 kiB "./hello_george.py"
     0.938477 kiB "./hello_world.py"
     4.714844 kiB "./raw_bytes_practice.py"
     3.784180 kiB "./README.md"
     0.176758 kiB "./slots_practice/Class and Instance Attributes – Real Python.desktop"
     0.258789 kiB "./slots_practice/Link to 10. __slots__ Magic — Python Tips 0.1 documentation%%%%%+.desktop"
     6.858398 kiB "./slots_practice/slots_practice.py"
     3.413086 kiB "./socket_talk_to_ethernet_device.py"
     1.935547 kiB "./socket_udp_send_packed_struct_TODO.py"
     6.658203 kiB "./struct_packed.py"
     1.633789 kiB "./system_call_subprocess_pipe.py"
     6.295898 kiB "./textwrap_practice_1.py"
     3.559570 kiB "./yaml_import/import_yaml_test.py"
     0.722656 kiB "./yaml_import/my_config1.yaml"
     0.708984 kiB "./yaml_import/my_config2.yaml"

5. Showing only files of a certain type (.yaml in this case)

Pipe to grep at the end to filter only on files of a certain type (.yaml in this case). Sorted by size here:

find . -type f -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
    | sort -rn | grep -i '\.yaml'

Output:

     0.722656 kiB ./yaml_import/my_config1.yaml
     0.708984 kiB ./yaml_import/my_config2.yaml

6. Showing only files NOT of a certain type (not .yaml in this case)

To not show files of a certain type, add -v to grep (shown as -iv here, since it includes -i still):

find . -type f -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
    | sort -rn | grep -iv '\.yaml'

Output:

    15.280273 kiB ./hello_george.py
    11.862305 kiB ./enum_practice.py
     6.858398 kiB ./slots_practice/slots_practice.py
     6.658203 kiB ./struct_packed.py
     6.295898 kiB ./textwrap_practice_1.py
     4.714844 kiB ./raw_bytes_practice.py
     3.784180 kiB ./README.md
     3.559570 kiB ./yaml_import/import_yaml_test.py
     3.413086 kiB ./socket_talk_to_ethernet_device.py
     2.354492 kiB ./auto_white_balance_img.py
     2.121094 kiB ./for_loop_basic_demo.py
     1.935547 kiB ./socket_udp_send_packed_struct_TODO.py
     1.692383 kiB ./binary_tree_traverse_and_print_TODO.py
     1.633789 kiB ./system_call_subprocess_pipe.py
     1.507812 kiB ./autogenerate_c_or_cpp_code.py
     1.322266 kiB ./binary_tree_TODO.py
     1.116211 kiB ./change_environment_variables_in_parent_process_TODO.py
     0.938477 kiB ./hello_world.py
     0.258789 kiB ./slots_practice/Link to 10. __slots__ Magic — Python Tips 0.1 documentation%%%%%+.desktop
     0.176758 kiB ./slots_practice/Class and Instance Attributes – Real Python.desktop
     0.137695 kiB ./autogenerated/myheader.h

Or, not of multiple types: ignore both .yaml and .py files here by piping to grep -Eiv '\.(yaml|py)':

find . -type f -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
    | sort -rn | grep -Eiv '\.(yaml|py)'

Output: this leaves only .md, .desktop, and .h files for me in this case, still sorted by size:

     3.784180 kiB ./README.md
     0.258789 kiB ./slots_practice/Link to 10. __slots__ Magic — Python Tips 0.1 documentation%%%%%+.desktop
     0.176758 kiB ./slots_practice/Class and Instance Attributes – Real Python.desktop
     0.137695 kiB ./autogenerated/myheader.h

7. Showing only files NOT of a certain type (not .yaml nor .py in this case) and > some size (1 kiB in this case)

Add size +1k back in to only show files > 1 kiB:

find . -type f -size +1k -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
    | sort -rn | grep -Eiv '\.(yaml|py)'

Output: only 1 file fits this filter:

     3.784180 kiB ./README.md

8. (Figure out which file extensions to add to git lfs next) Find all files in your git repo not in the .git folder, > 100 kiB, which are not PDF files nor .c nor .h files

Note: this command is very useful when trying to figure out which binary files you should add to git lfs next, in order to reduce the size of your regular git repo contents.

Let's say you are already tracking all PDF files in git lfs, and you want to find all large files > 100 kiB now which are not PDF, not .c, and not .h files, so you can track those with git lfs next. Note that you also need to exclude the .git directory, as I explain in detail here: How do I exclude a directory when using find?.

Here is how:

# List all files of the desired types, sorted, >100 kiB, and NOT in the .git
# dir.
# - Note that since we are using the `-i` flag with `grep`, this ignores the
#   case of the excluded files, so it
#   excludes .pdf, .PDF, .pDf, .c, .C, .h, .H, etc. 
find . -not \( -path "./.git" -type d -prune \) \
    -type f -size +100k -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
    | sort -rn | grep -Eiv '\.(pdf|h|c)'

If I run this in the root of my eRCaGuy_hello_world repo I get the following output:

  1003.039062 kiB ./cpp/bin/c
   943.083008 kiB ./cpp/bin/a.ii
   943.055664 kiB ./cpp/bin/constexpr_max.ii
   937.910156 kiB ./cpp/bin/libgtest.a
   937.910156 kiB ./cpp/bin/1/libgtest.a
   875.296875 kiB ./cpp/bin/gtest-all.o
   827.643555 kiB ./cpp/bin_hello_world/hello_world.ii
   796.139648 kiB ./cpp/bin/class_inheritance_and_using_override_in_class.ii
   795.375000 kiB ./cpp/bin/using_override_in_class.ii
   795.291992 kiB ./cpp/bin/remove_vowels_in_string.ii
   794.961914 kiB ./cpp/bin/make_path_to_file.ii
   794.777344 kiB ./cpp/bin/hello_world_basic.ii
   777.974609 kiB ./markdown/photos/Fire_Drone_High_Quality_300x173.gif
   544.968750 kiB ./cpp/bin_hello_world/hello_world.s
   521.539062 kiB ./cpp/bin/b
   515.898438 kiB ./cpp/unordered_map_practice/unordered_map_hash_table_implicit_key_construction_test
   485.964844 kiB ./test_photos/test1.jpg
   314.006836 kiB ./markdown/photos/pranksta6.jpg
   311.539062 kiB ./cpp/bin_hello_world/hello_world.o
   283.306641 kiB ./markdown/photos/pranksta7.jpg
   278.929688 kiB ./markdown/photos/pranksta3.jpg
   216.493164 kiB ./markdown/photos/pranksta4.jpg
   214.109375 kiB ./cpp/bin/tmp
   209.582031 kiB ./cpp/bin/libgmock.a
   205.605469 kiB ./markdown/photos/pranksta5.jpg
   201.375000 kiB ./cpp/template_function_sized_array_param/template_func
   196.929688 kiB ./cpp/bin/gmock-all.o
   171.101562 kiB ./cpp/template_function_sized_array_param/regular_func
   169.750000 kiB ./cpp/bin_hello_world/hello_world
   158.937500 kiB ./cpp/bin/onlinegdb--process_10_bit_video_data
   158.906250 kiB ./cpp/bin/process_10_bit_video_data
   149.671875 kiB ./linux/systemd-by-example/README.md
   131.186523 kiB ./markdown/photos/pranksta2.jpg

To just see the extensions of these files now, let's pipe the entirety of the above to awk '{print($3)}' | sed 's/.*\///' | grep -oE "(^[^.]*$|\.[^0-9]*\..*$)" | sort -u. The awk part grabs just the path, which is the 3rd space-separated field (column) from the left. The sed part retains just the contents after the last /. The grep part then keeps only the extension, including the dot (.), if it has one, and the whole string otherwise. And finally, sort -u removes duplicates to leave only unique strings. (For more-advanced examples to focus on extracting more-complicated extensions, such as .tar.gz from file.10.5.2.tar.gz, see my answer here: Grabbing the extension in a file name).

Here's that final command:

find . -not \( -path "./.git" -type d -prune \) \
    -type f -size +100k -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
    | sort -rn | grep -Eiv '\.(pdf|h|c)' \
    | awk '{print($3)}' \
    | sed 's/.*\///' | grep -oE "(^[^.]*$|\.[^0-9]*\..*$)" | sort -u

And the output:

.a
b
c
.gif
hello_world
.ii
.jpg
.md
.o
onlinegdb--process_10_bit_video_data
process_10_bit_video_data
regular_func
.s
template_func
tmp
unordered_map_hash_table_implicit_key_construction_test

Manually reviewing that, I can see it might be a good idea to add the following binary extensions to be tracked by git lfs for this repo:

.gif
.jpg
.o

I could also optionally manually track some of those extension-less executables.


Quick reference: list extensions of all files > 10 kiB in size (-size +10k):
Note: I've modified the awk call near the end to grab just the end of the filename (where the extension is), in case the filename has spaces in it. This is done with awk '{print($NF)}':

find . -not \( -path "./.git" -type d -prune \) \
    -type f -size +10k -printf '%s\t%p\n' \
    | awk '{printf("%13.6f kiB ", $1/(1024)); \
    for (i=2; i<NF; i++) printf("%s ", $i); printf("%s\n", $NF)}' \
    | sort -rn \
    | awk '{print($NF)}' \
    | sed 's/.*\///' | grep -oE "(^[^.]*$|\.[^0-9]*[\.]?[^0-9]*$)" | sort -u

Going further

In the examples above, instead of using find with -printf '%s\t%p\n' to output size information, you could use du instead. I haven't gone down that path to get the right options and formatting I'd want, however, like I have done with find, so the find commands are good enough for now.

References

  1. ChatGPT helped me figure a lot of this out, especially getting me started on the awk stuff, and working on the regular expressions for sed and grep. All content above, including the code and words, however, is my own, and was written and tested directly by me.
  2. I also used https://regex101.com/ to test and develop the regular expressions.
  3. My answer: How do I exclude a directory when using find?
  4. For more-advanced examples to focus on extracting more-complicated extensions, such as .tar.gz from file.10.5.2.tar.gz, see my answer here: Grabbing the extension in a file name
4
  • Any reason for the downvote? Do I have an error or mistake (I'll fix it)? Did I not add value to the answers already here? Is it just too long for your liking? Jan 16, 2023 at 6:56
  • Most of your answer assumes that names do not contain newlines (see Why is looping over find's output bad practice?). The question in furthermore not interested in recursively looking for files but are only concerned with looking in a particular directory. Your answer then continues on various tangents that are irrelevant to the question. The last two pipelines that you present could be simplified greatly by recognising that awk, grep and sed very seldom have to be used together (awk can often trivially do the job of the other two tools).
    – Kusalananda
    Jan 16, 2023 at 7:20
  • I did not downvote, but for example to exclude .yaml or .py files you can simply use: find . ! -name '*.yaml' ! -name '*.py' (and you avoid using the grep) Jan 16, 2023 at 7:25
  • 2
    Answer is way long. It does go on tangents. And yet, the first solution is exactly what I need at 03:22 on a Monday morning to recover space on a server that will be used by several thousand people as soon as the sun comes up. Upvoted with gratitude.
    – wistlo
    Feb 13, 2023 at 8:24
0

Kindly run below one-liner bash shell script for listing all the files recusrively with size of files of multiple folders. Note: Kindly change folder location according to your demand.

/usr/bin/du -a /var/log /home/user/log |sort -nr |awk '{print $NF}'|while read l ;do echo "file $(du -csh $l|grep -vi total) " ;done|nl

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .