0

from lsof we can see the following output

lsof /var | grep delete
rsyslogd   9664      root    4w   REG  253,2 25589554694  67267903 /var/log/messages-20210513 (deleted)
rsyslogd   9664      root    7w   REG  253,2  9865832185  67294059 /var/log/secure-20210619 (deleted)
libvirtd   9666      root   21r   REG  253,2    10406312 134328488 /var/lib/sss/mc/initgroups (deleted)
qmgr      10241   postfix    8r   REG  253,2    10406312 134328488 /var/lib/sss/mc/initgroups (deleted)
gdm-sessi 13304      root    8r   REG  253,2    10406312 134328488 /var/lib/sss/mc/initgroups (deleted)   <-----------------------
dbus-daem 14198       gdm    4r   REG  253,2    10406312 134328488 /var/lib/sss/mc/initgroups (deleted)
dbus-daem 14535       gdm    5r   REG  253,2    10406312 134328488 /var/lib/sss/mc/initgroups (deleted)
sssd      16743      root   15r   REG  253,2    10406312 134328488 /var/lib/sss/mc/initgroups (deleted)
sssd_be   16746      root   22r   REG  253,2    10406312 134328488 /var/lib/sss/mc/initgroups (deleted)

after investigation we saw that gdm-session takes ~40G from /var as deleted file

so after we killed the PID --> 13304

we decreased /var from 98% used to 59.4G used

since we are dealing with very important production server

we want to know if we can avoid such of this behaviors that some deleted file as gdm-session can crash the OS by reaching /var to became 100%

  • /var size is 100G

appreciate for any useful suggestion ?

Improve this question
8
  • 1
    why does your "very important production server" run gdm-session?! Why would you want a graphical login manager on a production server!??!?
    – Marcus Müller
    Jan 12, 2022 at 11:53
  • "Why would you want a graphical login manager on a production server" , do you mean that we should stop the gdm service?
    – yael
    Jan 12, 2022 at 11:56
  • 1
    yes. also, uninstall it? Alongside with all the stuff you don't need when you don't need a graphical desktop? You should think about what kind of things you want to run on your "very important production server", if it's very important.
    – Marcus Müller
    Jan 12, 2022 at 11:57
  • 1
    Whatever the solution, you’re going to have to apply it to all your servers anyway. Stopping and removing gdm is the right approach IMO.
    – Stephen Kitt
    Jan 12, 2022 at 12:19
  • 1
    if you have 1200 servers, you have automation, or a gigantic problem, anyways.. So, no, it's not a huge task at all. Any other solution wouldn't be easier than just uninstalling a package. I'm totally confused at what kind of operation you're running.
    – Marcus Müller
    Jan 12, 2022 at 12:20

1 Answer 1

Reset to default
1

You may define a maximum file size limit for a process via prlimit

prlimit --fsize=1G:2G -p 12345

sets soft and hard file size limits for the process with PID 12345 to 1 and 2 gigabytes (or gibi ... not entirely sure), respectively. This may be done even after the process started.

Be aware that this will kill the process if the limit is reached. More in the info pages.

Improve this answer
1
  • I don't know whether I would want to be this radical on sss's initgroups files. In the end, this just tries to hide the underlying problem...
    – Marcus Müller
    Jan 13, 2022 at 17:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.