How to write automated tests for zsh completion?

Question

I have a project in which I automatically generate zsh completion functions. As I work on it, I've found corner cases and bugs which I'm just writing down and making sure to re-test when I make changes. Obviously, I want to write a proper testsuite, but I can't figure out how.

For bash completions, testing is quite easy -- set COMP_* variables, run the function, and check COMP_REPLY. I'd like to do something similar for zsh.

I've read the compsys docs to the best of my ability, but I'm not seeing a solution.

I'd like to set the context by hand, run my completions, and then see an array of descriptions or something.

Has anyone found a way to test completions?

Answer 1

It's a bit more involved to test completions in Zsh. This is because Zsh's completion commands can run only from inside a completion widget, which in turn can only be called while the Zsh Line Editor is active. To be able to do completion inside a script, we will need to use a so-called pseudo terminal, in which we can have an active command line on which to activate a completion widget:

# Set up your completions as you would normally.
compdef _my-command my-command
_my-command () {
        _arguments '--help[display help text]'  # Just an example.
}

# Define our test function.
comptest () {
        # Add markup to make the output easier to parse.
        zstyle ':completion:*:default' list-colors \
                'no=<COMPLETION>' 'lc=' 'rc=' 'ec=</COMPLETION>'
        zstyle ':completion:*' group-name ''
        zstyle ':completion:*:messages' format \
                '<MESSAGE>%d</MESSAGE>'
        zstyle ':completion:*:descriptions' format \
                '<HEADER>%d</HEADER>'

        # Bind a custom widget to TAB.
        bindkey '^I' complete-word
        zle -C {,,}complete-word
        complete-word () {
                # Make the completion system believe we're on a 
                # normal command line, not in vared.
                unset 'compstate[vared]'

                # Add a delimiter before and after the completions.
                # Use of ^B and ^C as delimiters here is arbitrary.
                # Just use something that won't normally be printed.
                compadd -x $'\C-B'
                _main_complete "$@"
                compadd -J -last- -x $'\C-C'

                exit
        }

        vared -c tmp
}

zmodload zsh/zpty  # Load the pseudo terminal module.
zpty {,}comptest   # Create a new pty and run our function in it.

# Simulate a command being typed, ending with TAB to get completions.
zpty -w comptest $'my-command --h\t'

# Read up to the first delimiter. Discard all of this.
zpty -r comptest REPLY $'*\C-B'

zpty -r comptest REPLY $'*\C-C'  # Read up to the second delimiter.

# Print out the results.
print -r -- "${REPLY%$'\C-C'}"   # Trim off the ^C, just in case.

zpty -d comptest  # Delete the pty.

Running the example above will print out:

<HEADER>option</HEADER>
<COMPLETION>--help    display help text</COMPLETION>

If you don't want to test the entire completion output, but just want to test the strings that would be inserted on the command line, then see https://stackoverflow.com/questions/65386043/unit-testing-zsh-completion-script/69164362#69164362