I don't have either Compton or Mutter installed, but the
xcompmgr sample X composition manager doesn't bother to determine if another composition manager is running, but goes on and just calls
XCompositeRedirectSubwindows() and handles the error that may be generated by it in the handler installed with
error (Display *dpy, XErrorEvent *ev)
if (ev->request_code == composite_opcode &&
ev->minor_code == X_CompositeRedirectSubwindows)
fprintf (stderr, "Another composite manager is already running\n");
This seems to be the only possible way to do it:
XCompositeGetOverlayWindow() will always succeed and map the overlay window, whether it's already in use or not:
This request indicates that the client wishes to use the Composite Overlay Window of this screen. If this Composite Overlay Window has not yet been mapped, it is mapped by this request.
Mutter is implemented as a dynamic library (usually loaded by gnome-shell), not as a separate process.
grep -sl mutter /proc/*/maps
on Linux will find all the processes that are using it.