0

I have a machine running a pgbouncer server and an Open SSH server so clients can enstabilish a tunnel to the server and connect to the database.

Now all the clients keep the connection alive for around 3 minutes, then the connection is closed. As you can see the following image shows you that running the command:

ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc

multiple times underline the fact that the number or sshd processes decreases over a shor period of time.

 sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    158     158     790
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    150     150     750
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    146     146     730
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    140     140     700
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    140     140     700
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    140     140     700
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    140     140     700
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    136     136     680
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    132     132     660
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    132     132     660
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    132     132     660
sosepe@pgbouncer:~$ netstat -nt | grep :22
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    298     298    1490
sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    324     324    1620

But not over a long period of time.

What I mean is that that after some hours running that command again shows how actually the number of processes raises. That's because some of them never get actually closed.

sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
    324     324    1620

And after some hours this is the result:

sosepe@pgbouncer:~$ ps -o pid,user,%mem,command ax | sort -b -k3 -r | grep -o sshd | wc
   1926    1926    9630

And these are the consequences:

 Private  +   Shared  =  RAM used   Program

  4.0 KiB +  22.0 KiB =  26.0 KiB   agetty
132.0 KiB +  27.5 KiB = 159.5 KiB   vnstatd
128.0 KiB +  45.5 KiB = 173.5 KiB   systemd-udevd
136.0 KiB +  65.0 KiB = 201.0 KiB   cron
216.0 KiB +  33.0 KiB = 249.0 KiB   tail
112.0 KiB + 239.5 KiB = 351.5 KiB   systemd-timesyncd
504.0 KiB +  50.0 KiB = 554.0 KiB   lvmetad
192.0 KiB + 481.5 KiB = 673.5 KiB   vsftpd
908.0 KiB +  92.0 KiB =   1.0 MiB   sudo
916.0 KiB + 103.0 KiB =   1.0 MiB   pgbouncer
644.0 KiB + 604.0 KiB =   1.2 MiB   vmtoolsd
  1.3 MiB +  47.0 KiB =   1.3 MiB   rsyslogd
  1.2 MiB + 247.0 KiB =   1.5 MiB   CloudEndure_Age (4)
  1.4 MiB +  81.0 KiB =   1.5 MiB   dbus-daemon
  1.6 MiB + 331.0 KiB =   1.9 MiB   su (4)
  1.2 MiB + 892.0 KiB =   2.1 MiB   VGAuthService
  1.7 MiB + 582.0 KiB =   2.2 MiB   collectd
  2.7 MiB + 268.0 KiB =   3.0 MiB   systemd-logind
  2.9 MiB +  63.5 KiB =   3.0 MiB   bash
  4.3 MiB + 231.5 KiB =   4.5 MiB   systemd-journald
  2.7 MiB +   3.6 MiB =   6.2 MiB   php-fpm7.3 (3)
  4.8 MiB +   1.8 MiB =   6.6 MiB   (sd-pam) (3)
  6.5 MiB +   3.3 MiB =   9.8 MiB   systemd (4)
  5.8 MiB +   4.1 MiB =   9.9 MiB   php-fpm7.2 (3)
  5.7 MiB +   6.5 MiB =  12.2 MiB   php-fpm5.6 (3)
 16.1 MiB + 153.0 KiB =  16.3 MiB   run_linux_migration_scripts_periodically (2)
 17.2 MiB + 166.5 KiB =  17.3 MiB   update_onprem_volumes (2)
 18.5 MiB + 145.5 KiB =  18.6 MiB   tailer (2)
 11.8 MiB +  13.1 MiB =  24.9 MiB   apache2 (11)
159.3 MiB + 180.0 KiB = 159.5 MiB   java
741.6 MiB +   2.1 GiB =   2.8 GiB   sshd (4469)
---------------------------------
                          3.1 GiB
=================================

SSHD ate almost 3 GB of memory, and this goes on until the machine is rebooted.

Any clue about where the problem could be?

Thanks!

P.S.

This is the conf file:

Port 10110

Protocol 1,2

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
UsePrivilegeSeparation yes

SyslogFacility AUTHPRIV
LogLevel INFO

PermitRootLogin no
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes

MACs hmac-md5,hmac-sha1,[email protected],hmac-ripemd160
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc

[email protected],diffie-hellman-group-exchange-sha256,diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
KexAlgorithms [email protected],diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521

PermitEmptyPasswords no
PasswordAuthentication yes

ChallengeResponseAuthentication no

UsePAM yes

GatewayPorts yes

ClientAliveInterval 600

PermitTunnel yes

MaxSessions 50

AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS

Subsystem   sftp    /usr/libexec/openssh/sftp-server
3
  • Add text, not images! Commented Nov 24, 2021 at 15:25
  • Using grep sshd before the sort would be more efficient. I am assuming that you are just tacking on the |wc to get a number, and that normally that isn't there, otherwise there is no point in sorting the lines at all, and grep can count.
    – icarus
    Commented Nov 24, 2021 at 15:26
  • I edited the question and put text instead of images Commented Nov 24, 2021 at 15:50

1 Answer 1

0

Having the same behavior on a server that is running Oracle Linux and Oracle Database, we noticed that the sshd service is consuming all the memory until the memory is depleted and the server requires a restart. We suspect that this only started when we implemented the CIS Benchmark for Linux. In our scenario the sshd service starts using memory as soon as we start the Oracle Database instance. For now we've disabled the sshd service until we find the root cause. We've gone as far as disabling PAM for authentication but to no avail. Hoping someone has found the cause of this issue.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .