0

TL;DR - when there's traffic (around 900) our sites' CPU reaches 100%, slow down and crush.

INTRODUCTION

Our company holds more than 40 content websites, hosted on Digital Ocean, using Wordpress CMS. In the past 3 days, something odd has started to happen, day after another, each time a different website's CPU rises to 100% causing it to crush eventually.

DETAILS

Server

  1. Machine specs - 48GB RAM, 12 cores, Centos
  2. Host - Digital Ocean
  3. Environment Configurations - nginx & php-fpm & mysql - All in one server.

What We Know

  1. It's probably NOT an attack - if it had happened on one website, we'd think it's an attack, but since it's 4 websites, then it's probably something else.

  2. It's NOT the plugins - we tried turning them off & restarting the services, but the CPU was still high. Plus, all the other websites have the same plugins.

  3. We haven't made any changes - All other websites are OK, and we changed nothing for this to happen.

  4. It's an interior process - we use CDN worker to serve cache, so anything that's content wise shouldn't be requested from the database.

What We Tried, But Didn't Help

  1. Changing php-fpm configurations - increasing php-fpm max children num to 200 from 50. Also, setting time limit of proccess to terminate within 20 seconds.

  2. Changning nginx configurations - turning ngnix fastcgi_ignore_client_abort off

REPORTS

htop report

htop report

php fpm error log php-fpm error log

nginx error log nginx error log

mysql process list enter image description here

mysql error log

2019-04-02 21:13:34 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-04-02 21:13:34 0 [Note] /usr/sbin/mysqld (mysqld 5.6.42) starting as process 14287 ...
2019-04-02 21:13:34 14287 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000)
2019-04-02 21:13:34 14287 [Warning] Buffered warning: Changed limits: table_open_cache: 431 (requested 2000)
2019-04-02 21:13:34 14287 [Note] Plugin 'FEDERATED' is disabled.
2019-04-02 21:13:34 14287 [Note] InnoDB: Using atomics to ref count buffer pool pages
2019-04-02 21:13:34 14287 [Note] InnoDB: The InnoDB memory heap is disabled
2019-04-02 21:13:34 14287 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-04-02 21:13:34 14287 [Note] InnoDB: Memory barrier is not used
2019-04-02 21:13:34 14287 [Note] InnoDB: Compressed tables use zlib 1.2.11
2019-04-02 21:13:34 14287 [Note] InnoDB: Using Linux native AIO
2019-04-02 21:13:34 14287 [Note] InnoDB: Using CPU crc32 instructions
2019-04-02 21:13:34 14287 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-04-02 21:13:34 14287 [Note] InnoDB: Completed initialization of buffer pool
2019-04-02 21:13:34 14287 [Note] InnoDB: Highest supported file format is Barracuda.
2019-04-02 21:13:34 14287 [Note] InnoDB: 128 rollback segment(s) are active.
2019-04-02 21:13:34 14287 [Note] InnoDB: Waiting for purge to start
2019-04-02 21:13:34 14287 [Note] InnoDB: 5.6.42 started; log sequence number 79628880679
2019-04-02 21:13:34 14287 [Note] Server hostname (bind-address): '*'; port: 3306
2019-04-02 21:13:34 14287 [Note] IPv6 is available.
2019-04-02 21:13:34 14287 [Note]   - '::' resolves to '::';
2019-04-02 21:13:34 14287 [Note] Server socket created on IP: '::'.
2019-04-02 21:13:34 14287 [Note] Event Scheduler: Loaded 0 events
2019-04-02 21:13:34 14287 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.6.42'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server (GPL)
2019-04-02 21:19:56 14287 [Warning] IP address '85.93.20.38' could not be resolved: Name or service not known
2019-04-02 21:21:20 14287 [Note] /usr/sbin/mysqld: Normal shutdown
2019-04-02 21:21:20 14287 [Note] Giving 2 client threads a chance to die gracefully
2019-04-02 21:21:20 14287 [Note] Event Scheduler: Purging the queue. 0 events
2019-04-02 21:21:20 14287 [Note] Shutting down slave threads
2019-04-02 21:21:22 14287 [Note] Forcefully disconnecting 0 remaining clients
2019-04-02 21:21:22 14287 [Note] Binlog end
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'partition'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_METRICS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMPMEM'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_CMP'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_LOCKS'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'INNODB_TRX'
2019-04-02 21:21:22 14287 [Note] Shutting down plugin 'InnoDB'
2019-04-02 21:21:22 14287 [Note] InnoDB: FTS optimize thread exiting.
2019-04-02 21:21:22 14287 [Note] InnoDB: Starting shutdown...
2019-04-02 21:21:24 14287 [Note] InnoDB: Shutdown completed; log sequence number 79629654999
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'BLACKHOLE'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'ARCHIVE'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'MRG_MYISAM'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'MyISAM'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'MEMORY'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'CSV'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'sha256_password'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'mysql_old_password'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'mysql_native_password'
2019-04-02 21:21:24 14287 [Note] Shutting down plugin 'binlog'
2019-04-02 21:21:24 14287 [Note] /usr/sbin/mysqld: Shutdown complete
190402 21:21:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
190402 21:21:25 mysqld_safe Logging to '/var/log/mysqld.log'.
190402 21:21:25 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2019-04-02 21:21:25 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-04-02 21:21:25 0 [Note] /usr/sbin/mysqld (mysqld 5.6.42) starting as process 14764 ...
2019-04-02 21:21:25 14764 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000)
2019-04-02 21:21:25 14764 [Warning] Buffered warning: Changed limits: table_open_cache: 431 (requested 2000)
  • 3
    Please dont use screenshots for posting logs. Copy-paste them into code blocks in your question. Don't rule out attacks. There are many forms of attack which simply don't care about your site, they are looking for any vulnerable site. Vulnerable sites will attract a lot of traffic from a lot of separate attackers who have all found your site(s). This is very true for the Wordpress Pingback Vulnrability You should check your access logs and especially check for a lot of xmlrpc.php calls. – Philip Couling Apr 3 at 9:12
  • Is your database on thesame machine as your nginx/php processes? You might want a dedicated database server. – Gudgip Apr 3 at 9:54
  • 1
    Ancedotally, this is the second recent instance of php-fpm possibly misbehaving on here that I've seen. – Jeff Schaller Apr 3 at 10:47
  • @JeffSchaller It's also the second instance of Wordpress misbehaving. That link includes wp-content references. – Philip Couling Apr 3 at 10:54
  • Nothing in the mysql logs? – mark Apr 3 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.