At the system level the
readdir(3) call is used to return a stream of directory entries, until error or that list runs out:
The readdir() function returns a pointer to the next directory entry in
the named directory stream dirp. It returns NULL upon reaching the end
of the directory or detecting an invalid seekdir() operation.
Whether and how
fs.readdir to obtain individual directory entries in turn and not read them all into memory at once.
readdir(3) is not recursive; it will not check the contents of child directories; for that the
fts(3) call is typically used. Higher level languages may use
fts(3) or they may instead manually recurse over successive
closedir calls for each new child directory found during the
Specific high level language implementations may be inefficient for large numbers of files; I believe that the Go language has a concurrent filesystem recursing implementation, and that various fixes have been made over the years to implementations.