How to reduce the impact of changing the value of the `fpath` array on zsh starting time?

Question

I'm using zsh version 5.1.1 and tmux version 2.3, and recently I've noticed that starting a new shell (via a new window or pane in tmux) takes more time than before.

It's enough for me to type around 5 characters before the prompt is displayed. I would like to reduce this time, so I wrote a return statement at various positions inside my zshrc file until I found the line which was responsible for the increased starting time. It seems to be:

fpath=(~/.zsh/completion $fpath)

And I can reproduce my issue with the following minimal zshrc:

fpath=(~/.zsh/completion $fpath)
autoload -Uz compinit
compinit

The first line prepends the path ~/.zsh/completion to the array fpath. I do this because I want to write the files containing the code of my custom completion functions inside this directory.

To get a sense of how much time it takes for a shell to start, I found this command:

for i in $(seq 1 10); do /usr/bin/time zsh -i -c exit; done

It starts and exits 10 shells consecutively, each time measuring the time it took. Here's what it reports with the previous minimal zshrc:

0.36user 0.04system 0:00.42elapsed 96%CPU (0avgtext+0avgdata 6056maxresident)k
0inputs+160outputs (0major+9087minor)pagefaults 0swaps
0.30user 0.04system 0:00.36elapsed 96%CPU (0avgtext+0avgdata 5900maxresident)k
0inputs+160outputs (0major+8949minor)pagefaults 0swaps
0.30user 0.05system 0:00.37elapsed 96%CPU (0avgtext+0avgdata 6080maxresident)k
0inputs+160outputs (0major+8992minor)pagefaults 0swaps
0.31user 0.04system 0:00.37elapsed 95%CPU (0avgtext+0avgdata 5936maxresident)k
0inputs+160outputs (0major+8956minor)pagefaults 0swaps
0.30user 0.04system 0:00.36elapsed 96%CPU (0avgtext+0avgdata 6052maxresident)k
0inputs+160outputs (0major+9089minor)pagefaults 0swaps
0.32user 0.04system 0:00.38elapsed 96%CPU (0avgtext+0avgdata 5944maxresident)k
0inputs+160outputs (0major+8948minor)pagefaults 0swaps
0.32user 0.04system 0:00.37elapsed 95%CPU (0avgtext+0avgdata 6004maxresident)k
0inputs+160outputs (0major+8991minor)pagefaults 0swaps
0.32user 0.02system 0:00.36elapsed 96%CPU (0avgtext+0avgdata 6056maxresident)k
0inputs+160outputs (0major+9109minor)pagefaults 0swaps
0.30user 0.05system 0:00.36elapsed 96%CPU (0avgtext+0avgdata 6072maxresident)k
0inputs+160outputs (0major+9003minor)pagefaults 0swaps
0.31user 0.04system 0:00.36elapsed 96%CPU (0avgtext+0avgdata 6040maxresident)k
0inputs+160outputs (0major+9055minor)pagefaults 0swaps

I'm not sure to read this correctly, but it seems to indicate that on average a shell needs 0.32 second to start.

Here's what the same command produces with the same minimal zshrc after commenting the line fpath=(...) :

0.08user 0.02system 0:00.10elapsed 97%CPU (0avgtext+0avgdata 5608maxresident)k
0inputs+0outputs (0major+1721minor)pagefaults 0swaps
0.06user 0.00system 0:00.07elapsed 97%CPU (0avgtext+0avgdata 5660maxresident)k
0inputs+0outputs (0major+1723minor)pagefaults 0swaps
0.05user 0.00system 0:00.06elapsed 92%CPU (0avgtext+0avgdata 5680maxresident)k
0inputs+0outputs (0major+1720minor)pagefaults 0swaps
0.05user 0.00system 0:00.06elapsed 95%CPU (0avgtext+0avgdata 5724maxresident)k
0inputs+0outputs (0major+1734minor)pagefaults 0swaps
0.04user 0.02system 0:00.06elapsed 95%CPU (0avgtext+0avgdata 5748maxresident)k
0inputs+0outputs (0major+1730minor)pagefaults 0swaps
0.05user 0.00system 0:00.06elapsed 95%CPU (0avgtext+0avgdata 5692maxresident)k
0inputs+0outputs (0major+1724minor)pagefaults 0swaps
0.04user 0.01system 0:00.06elapsed 95%CPU (0avgtext+0avgdata 5636maxresident)k
0inputs+0outputs (0major+1728minor)pagefaults 0swaps
0.04user 0.01system 0:00.06elapsed 95%CPU (0avgtext+0avgdata 5628maxresident)k
0inputs+0outputs (0major+1727minor)pagefaults 0swaps
0.04user 0.01system 0:00.06elapsed 95%CPU (0avgtext+0avgdata 5684maxresident)k
0inputs+0outputs (0major+1722minor)pagefaults 0swaps
0.05user 0.00system 0:00.06elapsed 95%CPU (0avgtext+0avgdata 5728maxresident)k
0inputs+0outputs (0major+1731minor)pagefaults 0swaps

Now, on average, it seems to take only 0.05 second.

By default, the value of FPATH is:

/usr/local/share/zsh/site-functions:/usr/share/zsh/vendor-functions:/usr/share/zsh/vendor-completions:/usr/share/zsh/functions/Calendar:/usr/share/zsh/functions/Chpwd:/usr/share/zsh/functions/Completion:/usr/share/zsh/functions/Completion/AIX:/usr/share/zsh/functions/Completion/BSD:/usr/share/zsh/functions/Completion/Base:/usr/share/zsh/functions/Completion/Cygwin:/usr/share/zsh/functions/Completion/Darwin:/usr/share/zsh/functions/Completion/Debian:/usr/share/zsh/functions/Completion/Linux:/usr/share/zsh/functions/Completion/Mandriva:/usr/share/zsh/functions/Completion/Redhat:/usr/share/zsh/functions/Completion/Solaris:/usr/share/zsh/functions/Completion/Unix:/usr/share/zsh/functions/Completion/X:/usr/share/zsh/functions/Completion/Zsh:/usr/share/zsh/functions/Completion/openSUSE:/usr/share/zsh/functions/Exceptions:/usr/share/zsh/functions/MIME:/usr/share/zsh/functions/Misc:/usr/share/zsh/functions/Newuser:/usr/share/zsh/functions/Prompts:/usr/share/zsh/functions/TCP:/usr/share/zsh/functions/VCS_Info:/usr/share/zsh/functions/VCS_Info/Backends:/usr/share/zsh/functions/Zftp:/usr/share/zsh/functions/Zle

At the beginning of the variable, I noticed the path /usr/local/share/zsh/site-functions.
It was an empty directory, so I removed it and replaced it with a symbolic link whose target is my completion directory:

cd /usr/local/share/zsh; sudo rm site-functions; sudo ln -s ~/.zsh/completion site-functions

With this symbolic link, I don't need to change the value of fpath. The completion functions still work, and the starting time of the shells has been divided by 4, from around 0.32 second to around 0.08 second. The difference is noticeable to me. However, it doesn't feel right to use a system-wide directory.

Is there another, better, way to achieve the same result?

---

Edit:

Thanks to @thrig, I've realised that my ~/.zsh/ directory was in fact a symbolic link to a Dropbox directory:

ls -l .zsh
lrwxrwxrwx 1 user user 16 jan. 18 18:54 .zsh -> Dropbox/conf/zsh

I did this to have a backup of my configuration, and because I don't know how to use git.

If I move the completion functions to a directory outside Dropbox, zsh starting time becomes much lower.

---

Edit 2:

I don't understand, now it seems zsh starts slowly again, even if I put my completion functions outside of the Dropbox directory.

In case it helps, I have uploaded on a pastebin-like site the output of the following command:

strace -e trace=desc -o log -r zsh

According to man strace, the -e trace=desc option should trace all file descriptor related system calls. And the -r option should print a relative timestamp upon entry to each system call.

Here's a link for the log when fpath is unchanged.
And here's a 2nd link for the log when fpath is changed.

I'm unable to interpret the log files, however there's a big difference in size. The first one seems to include around only 4000 system calls, while the 2nd one has around 67000.

---

Edit 3:

I also recorded the output of the following command:

PS4='+[%D{%H:%M:%S.%.}]%N:%i> ' zsh -x

… with the script utility.

Here's a link to the log file when fpath is unchanged.
And here's a link to the log file when fpath is changed.

This time, the first log file has only around 600 lines, while the second one has around 100000 lines.

In the log file where fpath is changed, the functions which are called most often are compdef (around 69000 times), compinit (around 16000 times), and compdump (around 14000 times).

Thank you for all the help given in the comments.

Answer 1

I thought that maybe the problem came from my zsh version, so I purged the zsh package and compiled it from source:

% sudo aptitude install git-core gcc make autoconf yodl libncursesw5-dev texinfo

% git clone git://zsh.git.sf.net/gitroot/zsh/zsh
% cd zsh
% git checkout zsh-5.3.1

% ./Util/preconfig

% ./configure --build=x86_64-linux-gnu \
--prefix=/usr \
--includedir=/usr/include \
--mandir=/usr/share/man \
--infodir=/usr/share/info \
--sysconfdir=/etc \
--localstatedir=/var \
--libdir=/usr/lib/x86_64-linux-gnu \
--libexecdir=/usr/lib/x86_64-linux-gnu \
--bindir=/bin \
LDFLAGS="-Wl,--as-needed -g" \
--enable-maildir-support \
--enable-etcdir=/etc/zsh \
--enable-function-subdirs \
--enable-site-fndir=/usr/local/share/zsh/site-functions \
--enable-fndir=/usr/share/zsh/functions \
--with-tcsetpgrp \
--with-term-lib="ncursesw tinfo" \
--enable-cap \
--enable-pcre \
--enable-readnullcmd=pager \
--enable-custom-patchlevel=Debian \
--enable-additional-fpath=/usr/share/zsh/vendor-functions,/usr/share/zsh/vendor-completions \
--disable-ansi2knr

I found the dependencies and the configuration options by reading the INSTALL file as well as this gist found with Google:

https://gist.github.com/nicoulaj/715855

… and by looking at how the Ubuntu devs compiled the latest zsh packages:

https://launchpadlibrarian.net/280509421/buildlog_ubuntu-yakkety-amd64.zsh_5.2-5ubuntu1_BUILDING.txt.gz

There were some options which were written twice, or were not recognized by zsh, so I removed them. The value that the Ubuntu devs gave to LDFLAGS didn't seem to work on my machine, so I copied the one from the gist on Github. The options I kept are described by ./configure --help:

--build=BUILD           configure for building on BUILD [guessed]

--prefix=PREFIX         install architecture-independent files in PREFIX
                        [/usr/local]

--includedir=DIR        C header files [PREFIX/include]

--mandir=DIR            man documentation [DATAROOTDIR/man]

--infodir=DIR           info documentation [DATAROOTDIR/info]

--sysconfdir=DIR        read-only single-machine data [PREFIX/etc]

--localstatedir=DIR     modifiable single-machine data [PREFIX/var]

--libdir=DIR            object code libraries [EPREFIX/lib]

--libexecdir=DIR        program executables [EPREFIX/libexec]

--bindir=DIR            user executables [EPREFIX/bin]

 LDFLAGS                linker flags, e.g. -L<lib dir> if you have libraries in a
                            nonstandard directory <lib dir>

--enable-maildir-support
                        enable maildir support in MAIL and MAILPATH

--enable-etcdir=DIR     the default directory for global zsh scripts

--enable-function-subdirs
                        install functions in subdirectories

--enable-site-fndir=DIR same for site functions (not version specific)

--enable-fndir=DIR      the directory in which to install functions

--with-tcsetpgrp        assumes that tcsetpgrp() exists and works correctly

 --with-term-lib=LIBS   search space-separated LIBS for terminal handling

 --enable-cap           enable the search for POSIX capabilities (may
                        require additional headers to be added by hand)

--enable-pcre           enable the search for the pcre library (may create
                        run-time library dependencies)

--enable-readnullcmd=PAGER
                        pager used when READNULLCMD is not set

--enable-custom-patchlevel
                        set a custom ZSH_PATCHLEVEL value

--enable-additional-fpath=DIR
                        add directories to default function path

--enable-ansi2knr       translate source to K&R C before compiling

The compilation worked, but there were 2 failures among the 48 tests performed by the make check command:

% make
% make check
…
**************************************
46 successful test scripts, 2 failures, 0 skipped
**************************************
Makefile:187: recipe for target 'check' failed
make[1]: *** [check] Error 1
make[1]: Leaving directory '/home/user/GitRepos/zsh/Test'
Makefile:263: recipe for target 'check' failed
make: *** [check] Error 2

I couldn't get rid of them.

Finally, instead of using make install to install the binary, I used checkinstall to have a .deb that I can remove if I need to change the shell in the future (with a dpkg -r zsh):

% sudo checkinstall

During the installation, I had to give a short description (I used shell with lots of features), and more importantly I had to give a version. Without a version compatible with Debian policy, checkinstall wouldn't generate a .deb. I looked at the output of apt-cache policy zsh to check what is the naming scheme used by Debian, and chose 5.3.1-1ubuntu2.

% echo /bin/zsh >> /etc/shells
% chsh  →  /bin/zsh

These 2 lines were necessary to make Ubuntu recognize /bin/zsh as a valid login shell, and to make it my default shell.

Now, the shell version is 5.3.1 (instead of 5.1.1):

% zsh --version
    zsh 5.3.1 (x86_64-pc-linux-gnu)

And the shell starts quickly (around 0.04s):

% repeat 10 =time zsh -i -c exit

    0.06user 0.03system 0:00.12elapsed 80%CPU (0avgtext+0avgdata 5592maxresident)k
    0inputs+0outputs (0major+5668minor)pagefaults 0swaps
    0.06user 0.01system 0:00.09elapsed 77%CPU (0avgtext+0avgdata 5480maxresident)k
    0inputs+0outputs (0major+5634minor)pagefaults 0swaps
    0.04user 0.01system 0:00.07elapsed 82%CPU (0avgtext+0avgdata 5736maxresident)k
    0inputs+0outputs (0major+5641minor)pagefaults 0swaps
    0.04user 0.01system 0:00.07elapsed 77%CPU (0avgtext+0avgdata 5652maxresident)k
    0inputs+0outputs (0major+5645minor)pagefaults 0swaps
    0.03user 0.03system 0:00.07elapsed 82%CPU (0avgtext+0avgdata 5736maxresident)k
    0inputs+0outputs (0major+5634minor)pagefaults 0swaps
    0.04user 0.01system 0:00.07elapsed 78%CPU (0avgtext+0avgdata 5548maxresident)k
    0inputs+0outputs (0major+5624minor)pagefaults 0swaps
    0.04user 0.02system 0:00.07elapsed 78%CPU (0avgtext+0avgdata 5612maxresident)k
    0inputs+0outputs (0major+5655minor)pagefaults 0swaps
    0.04user 0.01system 0:00.07elapsed 77%CPU (0avgtext+0avgdata 5780maxresident)k
    0inputs+0outputs (0major+5624minor)pagefaults 0swaps
    0.03user 0.02system 0:00.07elapsed 78%CPU (0avgtext+0avgdata 5584maxresident)k
    0inputs+0outputs (0major+5641minor)pagefaults 0swaps
    0.03user 0.02system 0:00.07elapsed 78%CPU (0avgtext+0avgdata 5668maxresident)k

The first time I compiled, I forgot to switch to the latest release branch, so I was on master.
This development version of zsh had an even worse starting time than the version from the Ubuntu repositories. Around 0.42s to start a shell. But again, only if I added a directory to fpath in zshrc, and created a file inside the latter (a simple touch file was enough).

So I wonder whether the problem came from the compilation options, or the release version. I hope the release version I have compiled will remain quick…

Thank you @thrigh and @Stéphane Chazelas for helping me in the comments.