0

Some months ago I recovered all my photos and videos from a large 4TB drive. They are on a NAS. They are grouped in folders by extension. I have one folder called 'ts' with a few million of mostly small .ts files. Most of them probably will be useless, but the size of the folder is around 50GB, so there could be still some useful files in there. I thought I use SSH find the largest files and move them somewhere else to check them out. Accessing the folder via SMB or in the browser hasn't given me any results as loading the files never ends.

What would be the best way to find the largest file using SSH? Other strategies are welcome to.

4
  • (1) "few million of mostly small .ts files" – Removing files that are way too small may be a good preliminary step. (2) Do you really want to find largest file(s)? Or files large enough? finding files with size above some threshold does not require sorting. Review them, then lower the threshold and repeat. (3) For ls -S *.ts the argument list is created by the shell and it may be too long. For ls -S . the argument list is short and fixed. What command did you use exactly? Mar 31, 2023 at 14:26
  • @KamilMaciorowski Sorry, this message that the argument was too long didn't concern ls but mv for trying to move some of the files to another folder. I will try Artem's answer first and if it takes too long, I will remove a bunch of small files first. That is a very good suggestion, thank you. Mar 31, 2023 at 15:20
  • @KamilMaciorowski On second thought it makes sense to delete the smallest files first. Mar 31, 2023 at 15:28
  • @KamilMaciorowski if I use find . -type f -size -100k -delete. Will find delete whilst finding the files or will it look for the files first and then delete them one by one? Mar 31, 2023 at 16:39

1 Answer 1

1

Ten largest files:

ls -S | head -n 10

Ten smallest files

ls -S -r | head -n 10
4
  • Do you know if there is there a performance difference between ls,find and du? Mar 31, 2023 at 15:29
  • @JelmerBrands in my experience, for non-recursive usage, the performance differences don't really matter - the vast dominant amount of time is spent asking the operating system for file properties (fstatat) and these calls are all the same. Sorting a million entries is not nice computationally, but don't forget that even modest NAS CPUs will have billions of elementary comparisons they can do per second. The problem might end up being that listing millions of files and keeping that list in RAM leads to swapping and hence incredibly slow sorting. In that case, only more RAM would help. Mar 31, 2023 at 22:29
  • @Artem How much RAM are we roughly talking about? I have 10GB in my NAS (8+2 internal). Would this be too little for a few million small files? Also, I thought that only listing the 10 largest files would speed up things. Is this assumption wrong? Apr 1, 2023 at 8:18
  • 10GB of RAM is absolutely sufficient for 10 million files. Even in the worst case scenario a single file structure may occupy at most ~512 bytes in RAM which is around 5GB of RAM. Apr 1, 2023 at 11:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .