Server is slow even though there are enough resources

Question

I'm running about 450 simple webistes using Wordpress Multisite on a VPS with Debian 10. Here are some data about the resources:

root@hr711523385:~# free -mh
              total        used        free      shared  buff/cache   available
Mem:           57Gi       9,2Gi       1,2Gi       246Mi        47Gi        47Gi
Swap:          16Gi       9,9Gi       7,0Gi

root@hr711523385:~# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       40 bits physical, 48 bits virtual
CPU(s):              16
On-line CPU(s) list: 0-15
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           16
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               61
Model name:          Intel Core Processor (Broadwell, IBRS)
Stepping:            2
CPU MHz:             3000.000
BogoMIPS:            6000.00
Virtualization:      VT-x
Hypervisor vendor:   KVM
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            4096K
L3 cache:            16384K
NUMA node0 CPU(s):   0-15
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ibrs ibpb tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat
root@hr711523385:~# 

I'm using Plesk 18.0.48 to manage the VPS. Since several weeks all my websites become very slow, sometimes I get a 504 nginx error. When I open an SSH terminal everything is slow: typing commands, editing files, downloading remote files (238.00 KiB/s!!)

This is what htop gives: Do you see resource issues? RAM and CPU seem OK (to me)! Since the server is only used to run the wordpress network, I think WP is the problem, but I can't find how to begin: is it php-fm? apache? mysqld? ... who is the root of the problem ?

UPDATE

I aked an external sysadmin to dig deeper, he used Telegraf and sent me this : He said the problem is disk i/o (80 write/s) but he don't still know the reasons ! How can I find what generates so much disk i/o ? I think it's mysqld but I'm not sure (every website generates about 10 UPDATE queries, investigation in progress)

Answer 1

"450 Simple Websites using Wordpress Multisite": I am certain that the issue arises from PHP-FPM processes overloading the system. The root cause are most often bad bots that are hammering some domains.

1. Check the web server logs of the domains to find out which one is getting many requests.
2. Run (as root) ps aux | grep php-fpm to find out which PHP-FPM processes exist, which user owns them and then either take directly from the output which domain they are related to or run (also as root): strace -p <pid> where <pid> is the process id of a process and watch what they are doing (you'll find references to the domain in the output).
3. Run (as root) watch "ps aux | sort -nrk 3,3 | head -n 20" to watch the 20 load processes to find out, which ones are most cpu hungry.

The 504 results almost certainly results from unresponsive PHP processes, too. Either there are long-running scripts that are caught in never-ending loops or the max_children setting of PHP-FPM is too low so that no new children can be spawned hence Nginx does not get a response from the website.