compgen warning: -C option not working as I expected

Question

What is the correct way to use the compgen -C option?

I'm trying to learn about Bash programmable completion, and in particular the compgen builtin function. I'm experimenting with the different compgen command-line options, and I don't understand how the -C flag is supposed to work. From the GNU Bash Reference Manual:

-C command

command is executed in a subshell environment, and its output is used as the possible completions.

Based on this, I expect something like the following to work:

$ compgen -C 'echo "first_option second_option"' f
first_option

But instead, I get this:

$ compgen -C 'echo "first_option second_option"' f
-bash: compgen: warning: -C option may not work as you expect
first_option second_option  f

I've tried this with Bash version 4.2.45 on OS X 10.7 and with Bash version 4.2.25 on Ubuntu 12.04, and in both cases I get the same error:

-bash: compgen: warning: -C option may not work as you expect

Answer 1

Referring again to the bash manual page:

When using the -F or -C options, the various shell variables set by the programmable completion facilities, while available, will not have useful values.

I assume this is the "unexpected behaviour" it's referring to. The lack of $COMP_WORDS and $COMP_CWORD in the subshell created by compgen mean that your initial "f" doesn't get passed in to filter the results, hence the first two options you see. The third "f" -- I'll get on to.

It's even more unexpected at first glance that there seems to be no way to suppress this output (apart from piping away stderr) - but in fact there's a reasonable explanation. The arguments are intended to be used against the complete command, with which compgen shares code and arguments, and where they make more sense.

The complete command is used in the registration of a completion command, e.g.

complete -C /tmp/test.sh mycmd

And the bash source (in pcompletion, gen_command_matches() or thereabouts) explains how your "f" got into the results:

/* [...]
 $0 == cs->command         (command to execute for completion list)
 $1 == command name        (command being completed)
 $2 = word to be completed (possibly null)
 $3 = previous word
 [...] */

The third result in your example is of course your command input being passed back out as argument $2. Using the completion spec we created above, we can show more clearly how it's meant to be used with a script which annotates the arguments:

 $ cat > /tmp/test.sh << EOF
 x=0
 for arg in $0 $@; do⋅
   echo "$x:$arg"; x=$[$x+1]
 done
 EOF

then:

 $ complete -C /tmp/test.sh mycmd
 $ mycmd prevarg curarg<tab><tab>
 0:/tmp/test.sh   1:mycmd   2:curarg   3:prevarg

So it's not to say that -C can't be used by compgen; just that you're no better off than with $().

I suppose the remaining question is, "why are -C and -F defined against compgen at all?". I don't know.

Update

This answer originally contained my supposition

[Leaving -C and -F defined was] easier, maybe; or perhaps the authors decided a (rather obscure) warning was better than simply disabling what could be useful functionality.

alongside a suggestion to

make a bug report seeking clarification and/or improved documentation from the authors...

And I'm thrilled to say there was an authoritative answer courtesy of Chet Ramey basically confirming this. Quoting the bash-bug mailing list:

The command has total freedom to come up with possible completions; the completions don't even have to share a common prefix -- the higher-level completion code in readline takes care of that. Shell functions specified with -F work the same way.

The rationale is that it allows you to use binaries or scripts written in other languages in the same way you would use shell functions to generate programmable completions.

It's not very useful to call this at the command prompt, since much of the readline context is not present.