How to list sample of files from a directory

Question

I have a directory that contains hundred thousands of files.
I need to list sample of these files (example 10 files), without processing all the files found in the directory which will take too much processing time.

Answer 1

I don't think you can sample from the whole file list without reading them all in some way or the other, even on the filesystem level.

Unless their names follow a pattern that is (eg. fileXXXXXXX), in which case you could pregenerate a random list of names before accessing the files. For such a large amount of files it would be odd if their names were random.

But let's assume you're not that lucky. Using find is prefered over ls, since it can escape output with null, making it immune to nonstandard characters in filenames. If we don't want to read all the files, it is the fastest to use the ones at the start of the listing. To get a better sample, I would use a bigger sample first ($oversamplesize below) and then do a random subselection of size $samplesize from there. I didn't manage to make sort -R or shuf work well with null separators, so the shuffling and final selection is done by awk:

 find ~ -type f -print0 | 
 grep --null --null-data -m ${oversamplesize:-100} . | 
 awk -v samplesize=${oversamplesize:-11} -vRS='\0' -vORS='\0' \
   '{ a[NR]=$0 } END {srand(); while (i<samplesize) { b=(int(rand()*10000)%samplesize); if (b in c) {continue;} else {c[b]=a[b]; print a[b]; i++;} }; }' | 
 xargs -0 echo # echo here being just a dummy

Two notes here. For some reason, it often also prints an empty filename, so I increased the sample size just in case. The trivial note is not to forget to change the search path (~ here) and the final command.