1

Is there a unix command I can use to stream the items/contents of a directory? Using Node.js, we can read everything into memory with:

fs.readdir(dir, (err, items) => {});

but I am looking to stream items, for a very large directory, say with more than 10 million folders/files in it.

The tail command is for reading files not folders TMK, so is there some unix utility that can stream the contents of folders?

closed as unclear what you're asking by Ipor Sircer, Jeff Schaller, Thomas, schily, G-Man Sep 2 '18 at 21:25

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • I think you're just asking for the find command then. find . -type f will provide a stream of filenames which you can do do what you want with. – slm Sep 2 '18 at 0:35
  • yes I guess find can stream results back, is there some way to fine-tune find so that it doesn't read too many items at once or whatever? – Alexander Mills Sep 2 '18 at 0:38
  • Yes, find does offer many options and tests to refine or narrow down the results; too many to mention here. Refer to the man page. BTW, it reads ALL directory entries, and then decides which of those to select and act upon / print. – RudiC Sep 2 '18 at 7:29
1

In Unix you can use the find command to stream files & directories or both. The most basic command is this:

$ find . 

This will stream a list of files & directories which can then be passed through to another command via a pipe, | or you can use find's built in ability to run another command via -exec.

$ find . -type f -exec grep <somestring> {} \;

Or

$ find . -type f | ....

If you just want the contents of a single directory you can restrict find via the -depth switch:

$ find . -type f -depth 1 | ....
2

At the system level the readdir(3) call is used to return a stream of directory entries, until error or that list runs out:

 The readdir() function returns a pointer to the next directory entry in
 the named directory stream dirp.  It returns NULL upon reaching the end
 of the directory or detecting an invalid seekdir() operation.

Whether and how fs.readdir maps to the above will depend on how that function is implemented in JavaScript; there may be a way to call fs.readdir to obtain individual directory entries in turn and not read them all into memory at once.

readdir(3) is not recursive; it will not check the contents of child directories; for that the fts(3) call is typically used. Higher level languages may use fts(3) or they may instead manually recurse over successive opendir, readdir, and closedir calls for each new child directory found during the readdir(3) loop.

Specific high level language implementations may be inefficient for large numbers of files; I believe that the Go language has a concurrent filesystem recursing implementation, and that various fixes have been made over the years to implementations.

Not the answer you're looking for? Browse other questions tagged or ask your own question.