OEL6 Slow boot && Millions of socket files in /tmp/orbit-gdm

Question

Background on question

Lately I've been having some trouble with slow booting on the servers, sometimes over 2 hours to boot stuck on "Enabling Local Filesystem Quotas:"

After searching I managed to narrow it down to /etc/rc.d/rc.sysinit specifically:

# Clean up various /tmp bits
[ -n "$SELINUX_STATE" ] && restorecon /tmp
rm -f /tmp/.X*-lock /tmp/.lock.* /tmp/.gdm_socket /tmp/.s.PGSQL.*
rm -rf /tmp/.X*-unix /tmp/.ICE-unix /tmp/.font-unix /tmp/hsperfdata_* \
   /tmp/kde-* /tmp/ksocket-* /tmp/mc-* /tmp/mcop-* /tmp/orbit-*  \
   /tmp/scrollkeeper-*  /tmp/ssh-* \
   /dev/.in_sysinit

The exact cause of the slow booting is rm -rf /tmp/orbit-* due to a truly massive amount of socket files being generated and not removed except on boot.

The files are owned by user & group GDM and are named linc-rand-0-longrandom
ex: linc-1e33-0-54b542d8de62e

The file count has grown to 1 million on at lease one occasion.
The size of the folder has been recorded to be up to 3.1GB in size as well.

note: the socket files themselves are empty however each file reserves the minimum amount of disk space depending of filesystem type and setup.
(usually 2048 or 4096)

While I have a temporary solution (See below) I would very much like to track down the cause.
And so this brings me to

The question:

How do I track down the event that is causing GDM to generate these files?

Notes and system info:

• OS: Oracle Linux 6.7 (current)
• Kernel Linux 3.8.13-118.2.2.el6uek.x86_64
• GNOME v2.28.2
• We connect using Teamviewer 9, This is our only suspect at this time.
• Possibly the same issue on opensuse 11.4
• What looks to be the same issue again on RHEL 6.6
• Another person who encountered this. (in comments)
• Quick fix for anyone googling this: boot into a live cd with the linux of your choice, mount the root drive and simply mv /tmp/orbit-gdm /tmp/(Anything not caught by /tmp/orbit-*) then rm it after the system boots with your choice of tools (rm -rf, perl, rsync ect.)

Update 1
We might have found a slightly better temporary solution.
su gdm -c linc-cleanup-sockets
To run this you need the ORBit2-devel package and user gdm needs a bash shell.

One-Liner:
sudo yum install ORBit2-devel -y && sudo chsh /bin/bash gdm && sudo su gdm -c linc-cleanup-sockets

Then add the last part to a cronjob. (Warning! While there are no apparent issues on my system chsh may still break something)

Update 2

Finally caught it in the act, It's generating about 15-20 files a second right now.

per DopeFish's suggestion: lsof | grep /tmp/orbit-gdm returns:

gnome-ses 13197 gdm 17u unix 0xffff8802f3460c40 0t0 45587 /tmp/orbit-gdm/linc-338d-0-5188b15791c84
gnome-ses 13197 gdm 18u unix 0xffff8802f3460540 0t0 45589 /tmp/orbit-gdm/linc-338d-0-5188b15791c84
gnome-ses 13197 gdm 23u unix 0xffff8801dbacb480 0t0 45023 /tmp/orbit-gdm/linc-338d-0-5188b15791c84
gnome-ses 13860 gdm 17u unix 0xffff88022b7ca3c0 0t0 45450 /tmp/orbit-gdm/linc-3624-0-66bb9358d7045
gnome-ses 13860 gdm 18u unix 0xffff8802ee3f84c0 0t0 45452 /tmp/orbit-gdm/linc-3624-0-66bb9358d7045
gnome-ses 13860 gdm 21u unix 0xffff880236d9c840 0t0 45571 /tmp/orbit-gdm/linc-3624-0-66bb9358d7045
gnome-ses 13860 gdm 24u unix 0xffff8801f0177b80 0t0 45024 /tmp/orbit-gdm/linc-3624-0-66bb9358d7045
gconfd-2 13929 gdm 11u unix 0xffff8802f0eec180 0t0 45441 /tmp/orbit-gdm/linc-3669-0-14722d39cdaf5
gconfd-2 13929 gdm 15u unix 0xffff8801fffef540 0t0 45449 /tmp/orbit-gdm/linc-3669-0-14722d39cdaf5
gconfd-2 13929 gdm 17u unix 0xffff88021e34f8c0 0t0 41772 /tmp/orbit-gdm/linc-3669-0-14722d39cdaf5
gconfd-2 13929 gdm 19u unix 0xffff8801d30d4740 0t0 50348 /tmp/orbit-gdm/linc-3669-0-14722d39cdaf5
at-spi-re 13953 gdm 11u unix 0xffff8801e171e480 0t0 47205 /tmp/orbit-gdm/linc-3681-0-14f52f55664c
at-spi-re 13953 gdm 12u unix 0xffff8801db8903c0 0t0 47206 /tmp/orbit-gdm/linc-3681-0-14f52f55664c
gnome-set 13957 gdm 15u unix 0xffff88021e34f540 0t0 41773 /tmp/orbit-gdm/linc-3685-0-75f6d21d31afb
gnome-set 13957 gdm 16u unix 0xffff8801db890ac0 0t0 41775 /tmp/orbit-gdm/linc-3685-0-75f6d21d31afb
gnome-set 13957 gdm 21u unix 0xffff880236d9c140 0t0 45018 /tmp/orbit-gdm/linc-3685-0-75f6d21d31afb
gnome-set 13957 gdm 24u unix 0xffff8802ef6f7800 0t0 41867 /tmp/orbit-gdm/linc-3685-0-75f6d21d31afb
bonobo-ac 13959 gdm 12u unix 0xffff8802266e7400 0t0 44959 /tmp/orbit-gdm/linc-3687-0-3a8011046106
bonobo-ac 13959 gdm 15u unix 0xffff880226763800 0t0 45517 /tmp/orbit-gdm/linc-3687-0-3a8011046106
gconfd-2 13976 gdm 11u unix 0xffff8801dbb097c0 0t0 45561 /tmp/orbit-gdm/linc-3698-0-2c02c3e28e028
gconfd-2 13976 gdm 17u unix 0xffff8801db93f180 0t0 45586 /tmp/orbit-gdm/linc-3698-0-2c02c3e28e028
gconfd-2 13976 gdm 19u unix 0xffff8801d86997c0 0t0 45737 /tmp/orbit-gdm/linc-3698-0-2c02c3e28e028
gconfd-2 13976 gdm 22u unix 0xffff8801d60e7b00 0t0 41871 /tmp/orbit-gdm/linc-3698-0-2c02c3e28e028
gconfd-2 13976 gdm 24u unix 0xffff8801d306f3c0 0t0 50355 /tmp/orbit-gdm/linc-3698-0-2c02c3e28e028
at-spi-re 13978 gdm 11u unix 0xffff8802e0222880 0t0 49490 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 12u unix 0xffff8801d8619b80 0t0 45740 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 14u unix 0xffff8802f0f29c40 0t0 49566 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 16u unix 0xffff8802edc7e440 0t0 41858 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 18u unix 0xffff8801d60b9400 0t0 41860 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 20u unix 0xffff8801d60dc440 0t0 41865 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 22u unix 0xffff8801d6171c00 0t0 48731 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 24u unix 0xffff880170bd5c40 0t0 199299807 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 30u unix 0xffff8801d6369b00 0t0 41886 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 32u unix 0xffff8801d63aa4c0 0t0 41889 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 34u unix 0xffff8801d4555800 0t0 50338 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
at-spi-re 13978 gdm 36u unix 0xffff8801dbbab8c0 0t0 47488 /tmp/orbit-gdm/linc-369a-0-1882b699bb820
bonobo-ac 13986 gdm 15u unix 0xffff8801fcd3cac0 0t0 49488 /tmp/orbit-gdm/linc-36a2-0-362b8becbb48a
bonobo-ac 13986 gdm 16u unix 0xffff8801fcc21180 0t0 49492 /tmp/orbit-gdm/linc-36a2-0-362b8becbb48a
gnome-set 13988 gdm 15u unix 0xffff8801d8699440 0t0 48707 /tmp/orbit-gdm/linc-36a4-0-32757e4fb7e52
gnome-set 13988 gdm 16u unix 0xffff8801dbb67400 0t0 48708 /tmp/orbit-gdm/linc-36a4-0-32757e4fb7e52
gnome-set 13988 gdm 20u unix 0xffff88022b7caac0 0t0 49575 /tmp/orbit-gdm/linc-36a4-0-32757e4fb7e52
metacity 13998 gdm 14u unix 0xffff8801d6171880 0t0 48732 /tmp/orbit-gdm/linc-36ae-0-23e4b5d7b7629
metacity 13998 gdm 17u unix 0xffff8801d6171180 0t0 41872 /tmp/orbit-gdm/linc-36ae-0-23e4b5d7b7629
metacity 13998 gdm 18u unix 0xffff8801d60e7400 0t0 41874 /tmp/orbit-gdm/linc-36ae-0-23e4b5d7b7629
plymouth- 14009 gdm 12u unix 0xffff8801d63aa140 0t0 48800 /tmp/orbit-gdm/linc-36b9-0-705176425477
plymouth- 14009 gdm 15u unix 0xffff8801d63a87c0 0t0 51377 /tmp/orbit-gdm/linc-36b9-0-705176425477
plymouth- 14011 gdm 12u unix 0xffff8801d6369780 0t0 41887 /tmp/orbit-gdm/linc-36bb-0-77c5f78814cf
plymouth- 14011 gdm 15u unix 0xffff8801d6369080 0t0 47583 /tmp/orbit-gdm/linc-36bb-0-77c5f78814cf
gnome-pow 14016 gdm 12u unix 0xffff8801d45561c0 0t0 47489 /tmp/orbit-gdm/linc-36c0-0-4cbeb08e2eb07
gnome-pow 14016 gdm 16u unix 0xffff8801d61a4ac0 0t0 50356 /tmp/orbit-gdm/linc-36c0-0-4cbeb08e2eb07
gnome-pow 14016 gdm 17u unix 0xffff8801d84f2b00 0t0 50358 /tmp/orbit-gdm/linc-36c0-0-4cbeb08e2eb07
gnome-pow 14018 gdm 12u unix 0xffff8801d30b3440 0t0 50339 /tmp/orbit-gdm/linc-36c2-0-20a8f4a82d8c3
gnome-pow 14018 gdm 16u unix 0xffff8801d30b30c0 0t0 50349 /tmp/orbit-gdm/linc-36c2-0-20a8f4a82d8c3
gnome-pow 14018 gdm 17u unix 0xffff8801d306fac0 0t0 50351 /tmp/orbit-gdm/linc-36c2-0-20a8f4a82d8c3
metacity 59825 gdm 14u unix 0xffff88012af42400 0t0 199299808 /tmp/orbit-gdm/linc-e9b1-0-34f9216742ceb

su gdm -c linc-cleanup-sockets

Cleaned 41672 files 16 still live

Update 3
Disabling teamviewer when the file generation was occurring slowed it by about %30 but did not stop it.

This was tested by enabling TV and Disabling it multiple times on two servers.
So while TV makes the issue worse it does not appear to be the root cause.

(another resource for anyone who comes across this with the same problem)
#!/bin/bash while true do su gdm -c linc-cleanup-sockets sleep 10 done

Update 4
I came across three files in /var/log of unusual size.
/var/log/gdm/:0-greeter.log.1 (1.2G)
/var/log/gdm/:0.log.4 (2.7G)
/var/log/Xorg.0.log.old (3.1G)

All three continue like this for thousands of lines.

:0-greeter.log.1

Window manager warning: Screen 0 on display ":0" already has a window manager; try using the --replace option to replace the current window manager.
Window manager warning: Screen 0 on display ":0" already has a window manager; try using the --replace option to replace the current window manager.
Window manager warning: Screen 0 on display ":0" already has a window manager; try using the --replace option to replace the current window manager.
Window manager warning: Screen 0 on display ":0" already has a window manager; try using the --replace option to replace the current window manager.

:0.log.4

AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 18 disconnected
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 12 disconnected
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 12 connected from local host ( uid=42 gid=42 pid=49017 )
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 18 connected from local host ( uid=42 gid=42 pid=49017 )
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 18 disconnected
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 12 disconnected
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 12 connected from local host ( uid=42 gid=42 pid=49018 )
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 18 connected from local host ( uid=42 gid=42 pid=49018 )
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 18 disconnected
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 12 disconnected
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 12 connected from local host ( uid=42 gid=42 pid=49019 )
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 18 connected from local host ( uid=42 gid=42 pid=49019 )
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 18 disconnected
AUDIT: Sat Jan 16 03:00:28 2016: 11177: client 12 disconnected

Xorg.0.log.old

[604786.284] AUDIT: Sat Jan 16 03:00:37 2016: 11177: client 12 connected from local host ( uid=42 gid=42 pid=49530 )
[604786.300] AUDIT: Sat Jan 16 03:00:37 2016: 11177: client 18 connected from local host ( uid=42 gid=42 pid=49530 )
[604786.325] AUDIT: Sat Jan 16 03:00:37 2016: 11177: client 18 disconnected
[604786.326] AUDIT: Sat Jan 16 03:00:37 2016: 11177: client 12 disconnected
[604786.366] AUDIT: Sat Jan 16 03:00:37 2016: 11177: client 12 connected from local host ( uid=42 gid=42 pid=49541 )
[604786.390] AUDIT: Sat Jan 16 03:00:37 2016: 11177: client 18 connected from local host ( uid=42 gid=42 pid=49541 )
[604786.414] AUDIT: Sat Jan 16 03:00:37 2016: 11177: client 18 disconnected
[604786.416] AUDIT: Sat Jan 16 03:00:37 2016: 11177: client 12 disconnected

Hopefully these are of use to someone who can help resolve this.

Update 5
More info

netstat -avep | grep gdm

unix 2 [ ACC ] STREAM LISTENING 33469 14221/gconfd-2 /tmp/orbit-gdm/linc-378d-0-739b151848686
unix 2 [ ACC ] STREAM LISTENING 49272 14203/gnome-session /tmp/orbit-gdm/linc-377b-0-446c29ed00d3
unix 2 [ ACC ] STREAM LISTENING 49290 14252/gnome-setting /tmp/orbit-gdm/linc-37ac-0-4c1973905a173
unix 2 [ ACC ] STREAM LISTENING 49298 14284/bonobo-activa /tmp/orbit-gdm/linc-37cc-0-7163b16fe1d1
unix 2 [ ACC ] STREAM LISTENING 49300 14243/at-spi-regist /tmp/orbit-gdm/linc-37a3-0-102d46a4e4c2
unix 2 [ ACC ] STREAM LISTENING 34703 14293/metacity /tmp/orbit-gdm/linc-37d5-0-399c705510867
unix 2 [ ACC ] STREAM LISTENING 43204 14295/plymouth-log- /tmp/orbit-gdm/linc-37d7-0-76c2752e521e6
unix 2 [ ACC ] STREAM LISTENING 45461 14303/polkit-gnome- /tmp/orbit-gdm/linc-37df-0-7e28a833c25b4
unix 2 [ ACC ] STREAM LISTENING 45476 14302/gnome-power-m /tmp/orbit-gdm/linc-37de-0-13af029d53568
unix 2 [ ACC ] STREAM LISTENING 44354 14298/gdm-simple-gr /tmp/orbit-gdm/linc-37da-0-677e125a55c9f
unix 3 [ ] STREAM CONNECTED 49319 14293/metacity /tmp/orbit-gdm/linc-37d5-0-399c705510867
unix 3 [ ] STREAM CONNECTED 49972 14302/gnome-power-m /tmp/orbit-gdm/linc-37de-0-13af029d53568
unix 3 [ ] STREAM CONNECTED 49317 14293/metacity /tmp/orbit-gdm/linc-37d5-0-399c705510867
unix 3 [ ] STREAM CONNECTED 49312 14243/at-spi-regist /tmp/orbit-gdm/linc-37a3-0-102d46a4e4c2
unix 3 [ ] STREAM CONNECTED 43203 14243/at-spi-regist /tmp/orbit-gdm/linc-37a3-0-102d46a4e4c2
unix 3 [ ] STREAM CONNECTED 49301 14284/bonobo-activa /tmp/orbit-gdm/linc-37cc-0-7163b16fe1d1
unix 3 [ ] STREAM CONNECTED 40228 14243/at-spi-regist /tmp/orbit-gdm/linc-37a3-0-102d46a4e4c2
unix 3 [ ] STREAM CONNECTED 49968 14295/plymouth-log- /tmp/orbit-gdm/linc-37d7-0-76c2752e521e6
unix 3 [ ] STREAM CONNECTED 49274 14203/gnome-session /tmp/orbit-gdm/linc-377b-0-446c29ed00d3
unix 3 [ ] STREAM CONNECTED 45460 14243/at-spi-regist /tmp/orbit-gdm/linc-37a3-0-102d46a4e4c2
unix 3 [ ] STREAM CONNECTED 49271 14221/gconfd-2 /tmp/orbit-gdm/linc-378d-0-739b151848686
unix 3 [ ] STREAM CONNECTED 49982 14298/gdm-simple-gr /tmp/orbit-gdm/linc-37da-0-677e125a55c9f
unix 3 [ ] STREAM CONNECTED 45475 14243/at-spi-regist /tmp/orbit-gdm/linc-37a3-0-102d46a4e4c2
unix 3 [ ] STREAM CONNECTED 49980 14298/gdm-simple-gr /tmp/orbit-gdm/linc-37da-0-677e125a55c9f
unix 3 [ ] STREAM CONNECTED 33588 14203/gnome-session /tmp/orbit-gdm/linc-377b-0-446c29ed00d3
unix 3 [ ] STREAM CONNECTED 44353 14243/at-spi-regist /tmp/orbit-gdm/linc-37a3-0-102d46a4e4c2
unix 3 [ ] STREAM CONNECTED 49314 14252/gnome-setting /tmp/orbit-gdm/linc-37ac-0-4c1973905a173
unix 3 [ ] STREAM CONNECTED 49309 14243/at-spi-regist /tmp/orbit-gdm/linc-37a3-0-102d46a4e4c2
unix 3 [ ] STREAM CONNECTED 49979 14221/gconfd-2 /tmp/orbit-gdm/linc-378d-0-739b151848686
unix 3 [ ] STREAM CONNECTED 49292 14252/gnome-setting /tmp/orbit-gdm/linc-37ac-0-4c1973905a173
unix 3 [ ] STREAM CONNECTED 49289 14221/gconfd-2 /tmp/orbit-gdm/linc-378d-0-739b151848686
unix 3 [ ] STREAM CONNECTED 49971 14221/gconfd-2 /tmp/orbit-gdm/linc-378d-0-739b151848686
unix 3 [ ] STREAM CONNECTED 49974 14302/gnome-power-m /tmp/orbit-gdm/linc-37de-0-13af029d53568
unix 3 [ ] STREAM CONNECTED 49316 14221/gconfd-2 /tmp/orbit-gdm/linc-378d-0-739b151848686
unix 3 [ ] STREAM CONNECTED 34702 14243/at-spi-regist /tmp/orbit-gdm/linc-37a3-0-102d46a4e4c2
unix 3 [ ] STREAM CONNECTED 45487 14303/polkit-gnome- /tmp/orbit-gdm/linc-37df-0-7e28a833c25b4

Update 6
Another data dump.
ss -xprms | grep /tmp/orbit-gdm

u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37a5-0-54e93c0d4f464 35701 * 0 users:(("gconfd-2",14245,17))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b7-0-3777a325db022 35704 * 0 users:(("gnome-settings-",14263,16))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b3-0-5db00428cfe2a 35714 * 0 users:(("at-spi-registry",14259,12))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b3-0-5db00428cfe2a 43672 * 0 users:(("at-spi-registry",14259,16))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b7-0-3777a325db022 43674 * 0 users:(("gnome-settings-",14263,22))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b3-0-5db00428cfe2a 43715 * 0 users:(("at-spi-registry",14259,22))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-3788-0-6dfd943157cc8 43931 * 0 users:(("gnome-session",14216,20))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37ef-0-1e25b2118b9a 45957 * 0 users:(("plymouth-log-vi",14319,15))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37f2-0-48c1a4853a3c7 46524 * 0 users:(("polkit-gnome-au",14322,14))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b3-0-5db00428cfe2a 47375 * 0 users:(("at-spi-registry",14259,18))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37a5-0-54e93c0d4f464 47381 * 0 users:(("gconfd-2",14245,19))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37ed-0-48c0a255e98d7 47382 * 0 users:(("metacity",14317,17))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37ed-0-48c0a255e98d7 47384 * 0 users:(("metacity",14317,18))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37a5-0-54e93c0d4f464 47449 * 0 users:(("gconfd-2",14245,21))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37f1-0-62b916ee3bde0 47450 * 0 users:(("gnome-power-man",14321,16))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37f1-0-62b916ee3bde0 47452 * 0 users:(("gnome-power-man",14321,17))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b3-0-5db00428cfe2a 47736 * 0 users:(("at-spi-registry",14259,20))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38e7-0-7cd34eb474672 47748 * 0 users:(("bonobo-activati",14567,3))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38f1-0-78fed63f87292 47760 * 0 users:(("gnome-power-man",14577,16))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b9-0-bf39f0ecfb18 48325 * 0 users:(("bonobo-activati",14265,16))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b3-0-5db00428cfe2a 48379 * 0 users:(("at-spi-registry",14259,26))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37a5-0-54e93c0d4f464 50179 * 0 users:(("gconfd-2",14245,15))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-376e-0-117b4e816d611 50182 * 0 users:(("gnome-session",14190,18))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b3-0-5db00428cfe2a 50208 * 0 users:(("at-spi-registry",14259,14))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-376e-0-117b4e816d611 50210 * 0 users:(("gnome-session",14190,23))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b3-0-5db00428cfe2a 50224 * 0 users:(("at-spi-registry",14259,24))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38e1-0-31fe4fb874870 51339 * 0 users:(("at-spi-registry",14561,15))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38ef-0-3a6bdd32842f6 51345 * 0 users:(("plymouth-log-vi",14575,15))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38e1-0-31fe4fb874870 52271 * 0 users:(("at-spi-registry",14561,14))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38e1-0-31fe4fb874870 52272 * 0 users:(("at-spi-registry",14561,16))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38e1-0-31fe4fb874870 53454 * 0 users:(("at-spi-registry",14561,12))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38f2-0-9677e70872af 53504 * 0 users:(("polkit-gnome-au",14578,14))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b3-0-5db00428cfe2a 54300 * 0 users:(("at-spi-registry",14259,32))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38e5-0-700f791271f64 54308 * 0 users:(("gnome-settings-",14565,18))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-3788-0-6dfd943157cc8 92320134 * 0 users:(("gnome-session",14216,19))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37f1-0-62b916ee3bde0 92320192 * 0 users:(("gnome-power-man",14321,19))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-376e-0-117b4e816d611 92320212 * 0 users:(("gnome-session",14190,25))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38e5-0-700f791271f64 92320226 * 0 users:(("gnome-settings-",14565,4))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37b7-0-3777a325db022 92320416 * 0 users:(("gnome-settings-",14263,23))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38f1-0-78fed63f87292 92320540 * 0 users:(("gnome-power-man",14577,15))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-37ed-0-48c0a255e98d7 92320754 * 0 users:(("metacity",14317,20))
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-38e1-0-31fe4fb874870 102557171 * 0
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-861e-0-600072181468c 102557177 * 0
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-86bb-0-50ee4ccb72362 102557178 * 0
u_str ESTAB 0 0 /tmp/orbit-gdm/linc-86bb-0-50ee4ccb72362 102557180 * 0

Update 6

Finally some progress, We have determined that this is directly linked to $DISPLAY

Some of the time $DISPLAY is set to :1.0 rather than the default :0.0

If anyone knows why this causes GNOME to generate a massive amount of socket files OR Why it is :1.0 some of the time either will do as an answer.

I will be opening another separate question for DISPLAY and will link it to this one.

---

This question is getting rather long if anyone has any suggestions on what I can do to streamline and remove the useless info from it please comment or suggest an edit.

Also if there is a particular reason I'm not getting any responses or upvotes please let me know in a comment so I can improve this question.

Answer 1

I'm having the same problem on RHEL6.8 with an 800 megabyte /tmp/orbit-walker directory (walker is my username). This prevented my system from booting.

I found the problem by adding init=/bin/bash to my boot line to get a prompt, then doing

openvt -- /bin/bash

and then opening the virtual terminal with alt-f1. Searching around with ps, I found the rm -rf /tmp/orbit-* process that was hanging.

I was kill the hung rm and then continue the boot process with

exec /sbin/init

It is very difficult to delete the contents of a directory with a million files. Both 'find' and 'rm' insist on reading all the filenames and sorting them. They both run for hours and then abort with 8G of core saying "too many files".

Here's something that works.

(cd /tmp/orbit-walker; /bin/ls -1 -f | xargs /bin/rm)

The -f option makes /bin/ls print without sorting and the -1 does it a line at a time.