f=test.txt sh -c 'cat "$f"' produces output because the variable assignment
f=test.txt occurs before the expansion of the (single-quoted) command-argument
'cat "$f"'. The single-quotes prevent the expansion from taking place until the subcommand
cat "$f" is executed.
f=test.txt cat $f does not produce output because the variable assignment
f=test.txt occurs after the expansion of the (unquoted) command-argument
f=test.txt cat $f does not produce any output
First I'll attempt to explain why the command
f=test.txt cat $f is not producing any output despite your expectation that it would. Here there might be some slight confusion about the order of evaluation in what's called a simple command.
You may have assumed that the variable assignment in the prologue of the command (i.e. the
f=test.txt assignment) occurs before the variable expansion in the body of the command (i.e. the expansion of
cat $f). But this is actually not the case. To verify this we can refer to the page on simple command expansion in the Bash Manual or to the subsection on simple commands in the Posix Specification. Both of these references include the following passage:
A "simple command" is a sequence of optional variable assignments and redirections, in any sequence, optionally followed by words and redirections, terminated by a control operator.
When a given simple command is required to be executed (that is, when any conditional construct such as an AND-OR list or a case statement has not bypassed the simple command), the following expansions, assignments, and redirections shall all be performed from the beginning of the command text to the end:
The words that are recognized as variable assignments or redirections according to Shell Grammar Rules are saved for processing in steps 3 and 4.
The words that are not variable assignments or redirections shall be expanded. If any fields remain following their expansion, the first field shall be considered the command name and remaining fields are the arguments for the command.
Redirections shall be performed as described in Redirection.
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
Notice that step 2 is where the variable expansion in the command occurs, but step 1 tells us that the variable assignments are saved until steps 3 and 4. It follows that the expression
cat $f is expanded to
cat (with no arguments) before the assignment
f=test.txt takes place. This explains why you're not getting any output.
For further discussion on this topic see the following posts:
f=test.txt sh -c 'cat "$f"' does produce output
Next I'll attempt to explain why the command
f=test.txt sh -c 'cat "$f"' does produce output. For this we'll want to look at the full list of general operations performed by the shell:
The shell reads its input from a file (see sh), from the -c option or from the system() and popen() functions defined in the System Interfaces volume of POSIX.1-2008. If the first line of a file of shell commands starts with the characters "#!", the results are unspecified.
The shell breaks the input into tokens: words and operators; see Token Recognition.
The shell parses the input into simple commands (see Simple Commands) and compound commands (see Compound Commands).
The shell performs various expansions (separately) on different parts of each command, resulting in a list of pathnames and fields to be treated as a command and arguments; see wordexp.
The shell performs redirection (see Redirection) and removes redirection operators and their operands from the parameter list.
The shell executes a function (see Function Definition Command), built-in (see Special Built-In Utilities), executable file, or script, giving the names of the arguments as positional parameters numbered 1 to n, and the name of the command (or in the case of a function within a script, the name of the script) as the positional parameter numbered 0 (see Command Search and Execution).
The shell optionally waits for the command to complete and collects the exit status (see Exit Status for Commands).
So you can see here that calling a function/built-in/executable/script (step 6 on this list) occurs after parsing of the simple command. Therefore the assignment
f=test.txt occurs before the program execution
sh -c 'cat "$f"'. And because the argument is single-quoted it is only parsed after the command executes. Therefore the subcommand expands to