What is the difference in how linux implements commands which accept input via pipe vs. command arguments?

Question

Although there are many similar questions, I cannot find one that directly answers my question. I understand the basics of OS memory management, processes, communication between processes, etc. However, it is not obvious to me the fundamental difference between the implementation of commands that can receive input via pipes and those that can only accept command arguments.

Why would a command accept input via one method but not the other, and how can one can easily tell which input method a command accepts?

As a trivial example, why can I pipe the filenames output by ls to cat for display, but cannot do the same for rm? Is there a philosophical reason, deep technical reason, or arbitrary choice in implementation that drives this difference in behavior?

ls | cat
delete_me_1.txt
delete_me_2.txt

ls | rm
rm: missing operand

Asked differently, if I am writing a linux command, what technical factors do I need to consider to choose whether to accept command input via pipe, command arguments, or both?

Answer 1

However, it is not obvious to me the fundamental difference between the implementation of commands that can receive input via pipes and those that can only accept command arguments.

Well, whether to use one or the other (or both?), that's a thing that the program's developer decides; it has nothing to do with the operating system other than that the operating system offers the means to pass arguments, and to read from an input file descriptor, and present the standard input as such.

So, this question predates Linux by more than 20 years, which is the minimum age of the idea of standard input and command line arguments (the birth of UNIX, and I'm sure Thompson, Ritchie, Kernigan and McIlroy didn't come up with it out of the blue).

Asked differently, if I am writing a linux command, what technical factors do I need to consider to choose whether to accept command input via pipe, command arguments, or both?

Usability and fitness for your purpose!

If you process some data it's probably more useful to pipe it in as standard input, especially if you can produce output already before the program generating your input has quit. Since you can pass command line arguments only at the start of your program, it makes no sense to even wonder what to do if you're doing something where you need to stream through data, with a continuous producer.

If you're changing the behaviour of something or calculate something based on a variable number of parameters, then command line arguments are probably easier to use than a structured way of passing these as input.

So, this really boils down to how you want your program to be used. This is not a technical issue - it's a issue of purpose.

Matter of fact: it's 2022. For any modern language that is widely used, there's libraries to parse structured input data – for example, JSON – into dictionary/map/key-value data structures that can easily be used inside your program. Same for parsing command line arguments! Often, the command line argument parser can also parse configuration files (and you can use your standard input as configuration file, for example CLI11), so you don't even have to choose, technically, for a program that could do either.

Answer 2

Why would a command accept input via one method but not the other,

Ultimately, because the programs were made that way.

More practically, programs tend to accept input in the way that makes most sense: if the input is a set of filenames, or options modifying the program behaviour, it's usually given on the command line. Usually those are text and short to short-ish. If the input is a blob of data, it's read via stdin. The data to be processed might be binary data, and of a great length.

E.g. both cat and rm take a list of filenames as command line arguments (both can also take options). sed also takes filenames on the command line, but it also takes options, some of which can specify the actual instructions it applies on the data from those files. They all operate on the named files: cat prints their contents, rm removes them, sed works on the contents the way it's told.

It's just that both sed and cat read stdin by default if no filenames are given.

and how can one can easily tell which input method a command accepts?

You read the manuals. Or guess, based on what makes sense.

As a trivial example, why can I pipe the filenames output by ls to cat for display, but cannot do the same for rm?

Because cat takes a stream of data from stdin by default, rm only takes a list of filenames. Note that if you do ls | cat, it's not cat acting on the listed filenames, but on the listing itself. Even if ls prints foo.txt, ls |cat outputs foo.txt, but cat foo.txt would output the contents of foo.txt. ls * |cat is not the same as cat *.

Now, piping via a plain cat without options is a bit useless, because it just passes the data on as-is. It might make more sense with cat -n.

And yes, the output of ls is essentially a stream of data, just like any text file. To a human eye, it might look like a list of filenames, but it's really hard to properly use it as such, to the point that people have written articles like https://mywiki.wooledge.org/ParsingLs to argue against it. And it's a common enough as a misconception that those articles have been linked here on numerous occasions. (Yes of course there are tools like xargs that (try to) translate a stream of data to a list of command line arguments, but, well... just remember that the newline character is valid in filenames.)

Is there a philosophical reason, deep technical reason, or arbitrary choice in implementation that drives this difference in behavior?

That sounds like something that's in the eye of the beholder, but FWIW, I only see a practical technical reason (see above).

_{(* other cases: sed can also its processing instructions from a file, one named via a command-line option (-f). That makes sense if the script is long, the same way it makes sense to store C programs in files. Some programs read configuration files that contain the same options they take on the command line. That's a choice some make, and makes sense too, since that way the user has to learn only one set of magic words, and files work better for permanent storage.)}

Answer 3

If you are designing a machine to cut wood, you would think that the input would be wood in quantity and that parameters such as cutting thickness would be set for the duration of the task.

Many tasks on data relate to this analogy. It lends itself to a wide variety of arrangements and architectures: chaining (pipes), parallelization, distribution. The paradigm is very flexible and powerful.

wood > board 15cm | style moulding | transport | varnishing | transport > deposit

So the questions "on what" and "in what way" are answered first. If a task can work on an arbitrarily large volume of something, it is necessarily an input, so the other information are settings or arguments.

Well-designed programs are filters, that is, they simply cooperate through pipes. You will notice that very common programs sed, grep, awk, gzip, cat, md5sum, base64, bash, ssh etc. are designed to read standard input. Alternatively, they also admit file names as arguments, which are data sources and not the data itself. Filename is metadata whilst content is data.

md5sum < /etc/crontab # don't care metadata
md5sum /etc/crontab # same result with metadata

rm is equivocal, if it were to accept an entry, it could be the paths to be deleted, but it seemed more convenient to keep them as arguments.

In any case there are many ways to switch from the input stream to the arguments. When transforming input into arguments, one must take into account the limits of the interpreter and other parsing traps (quotes, separators, substitutions)

ls | xargs -d'\n' rm # apply rm according to arg length limits
         # NOT broken by spaces in filenames
         # broken by newlines in filenames

ls | parallel -m rm # apply rm whith many parallel processes
         # NOT broken by spaces in filenames
         # broken by newlines in filenames

ls | while read;do rm "$REPLY";done # apply rm one by one
         # NOT broken by spaces in filenames
         # broken by newlines in filenames

ls | split -l1 --filter='read && rm "$REPLY"'
         # why make it simple when you can make it complicated

rm $(ls) # apply rm all at once
         # broken by arg length limit
         # broken by space in filenames

rm * # apply rm all at once
     # broken by arg length limit
     # NOT broken by newline in filenames

(Conclusion: if you want problems, put newlines in filenames)

Sometimes, when you process business data, you tend to put in arguments some informations that caracterize the data, for example the type of data to process or other metadata. I think that it is a good idea to make difference between metadata and other arguments. metadata would take place naturally with the data, provided that the machine is able to read it and adapt.

Example:

hard wood > board 15cm | style moulding | transport | varnishing | transport > deposit

In this case, you have gain if the cutting and moulding machines are able to adapt their power and blades, instead of an operator that would check the tunings.

There are many examples of programs that put some metadata and data together to build a single datastream:

tar -cf- /etc | tar -tf-

Finally, in a largely distributed business workflow, you should clearly make difference between informations that only concern a particular tasks, that is local arguments, and informations that follow the input, data and metadata, especially since traceability is desired.

Yet another thing to know: arguments are publicly visible by the command ps, so it is a very bad idea to design a program with a secret argument. Instead use an indirect input method.