Your display manager (lightdm, openbox, etc.) is a child of init which is owned by root. Init isn't set-uid because it's a very special process and is just started with uid of 0. The command
ps -eaH gives a structured view of parentage, the relevant bits are:
r 1 ? 00:00:00 init
r 1521 ? 00:00:00 lightdm
r 1531 tty7 00:00:12 Xorg
m 2035 ? 00:00:00 lightdm
m 2177 ? 00:00:00 gnome-session
m 2225 ? 00:00:00 ssh-agent
Where I prepended the owner (Root or Me) of the process. Remember that /proc isn't a real filesystem, but provides file-like access into internal kernel structures and the kernel can set whatever permissions deemed appropriate. Even though 2035 has a real and effective UID of me, the entries in /proc/2035 are owned by root and /proc/2035/fd has permissions 0700 (-r-x------). It isn't until we get to 2177 that I own the pseudo-files in proc because the PPID of 2177 was not UID root.
Why? Because if I was spawned by a root program, some of my files might allow me to leverage into system security. This used to be otherwise as the proc(5) man notes under /proc/sys/fs/protected_hardlinks:
The default value in this file is 0. Setting the value to 1
prevents a longstanding class of security issues caused by hard-
link-based time-of-check, time-of-use races, most commonly seen
in world-writable directories such as /tmp. The common method
of exploiting this flaw is to cross privilege boundaries when
following a given hard link (i.e., a root process follows a hard
link created by another user). Additionally, on systems without
separated partitions, this stops unauthorized users from "pinning" vulnerable set-user-ID and set-group-ID files against being upgraded by the administrator, or linking to special files.
While this note doesn't speak exactly to why the /proc/fd attack vector can be exploited, it does give the feel better than a few thousand lines of code from kernel/fs/proc.