0

When I start the httpd (Apache) server on my Amazon Linux 2 it is creating numerous repeated port listeners on port 80 and 443. Every couple minutes it adds another set. The original port 80 and 443 start as root, the others are started by apache user. Basically, if I leave it running it keeps adding listeners until the resources are completely overwhelmed and the server slows to a crawl. After one night running it had 256 listeners on port 80 and 256 listeners on port 443. (The server was running fine until 2 days ago - might have started when I rebooted the server or did a basic yum update. But I am not sure what else to check.) Is this normal behavior? (new to apache, my experience was mostly IIS)

I removed all of my additional site configurations (custom headers), yet even after a restart of httpd I get: (and every few minutes, the last 2 lines repeat with new pid adding to the running listeners)

I cannot tell what is causing the repeated listeners. But every few minutes more get added.

# sudo lsof -i -P -n | grep LISTEN

rpcbind  2717      rpc    8u  IPv4  17600      0t0  TCP *:111 (LISTEN)
rpcbind  2717      rpc   11u  IPv6  17603      0t0  TCP *:111 (LISTEN)
vsftpd   3082     root    4u  IPv6  18895      0t0  TCP *:21 (LISTEN)
master   3176     root   13u  IPv4  19485      0t0  TCP 127.0.0.1:25 (LISTEN)
sshd     3339     root    3u  IPv4  20872      0t0  TCP *:22 (LISTEN)
sshd     3339     root    4u  IPv6  20874      0t0  TCP *:22 (LISTEN)
httpd    4731     root    4u  IPv6  41913      0t0  TCP *:80 (LISTEN)
httpd    4731     root    6u  IPv6  41923      0t0  TCP *:443 (LISTEN)
httpd    4732   apache    4u  IPv6  41913      0t0  TCP *:80 (LISTEN)
httpd    4732   apache    6u  IPv6  41923      0t0  TCP *:443 (LISTEN)
httpd    4733   apache    4u  IPv6  41913      0t0  TCP *:80 (LISTEN)
httpd    4733   apache    6u  IPv6  41923      0t0  TCP *:443 (LISTEN)
httpd    4734   apache    4u  IPv6  41913      0t0  TCP *:80 (LISTEN)
httpd    4734   apache    6u  IPv6  41923      0t0  TCP *:443 (LISTEN)
httpd    4735   apache    4u  IPv6  41913      0t0  TCP *:80 (LISTEN)
httpd    4735   apache    6u  IPv6  41923      0t0  TCP *:443 (LISTEN)
httpd    4736   apache    4u  IPv6  41913      0t0  TCP *:80 (LISTEN)
httpd    4736   apache    6u  IPv6  41923      0t0  TCP *:443 (LISTEN)

and

# ps -aux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.4 125604  4656 ?        Ss   19:39   0:01 /usr/lib/systemd/systemd --switched-root --system --deserialize 22
root         2  0.0  0.0      0     0 ?        S    19:39   0:00 [kthreadd]
root         4  0.0  0.0      0     0 ?        I<   19:39   0:00 [kworker/0:0H]

... more services...

root      3764  0.0  1.4 325392 14632 ?        Ss   19:52   0:00 php-fpm: master process (/etc/php-fpm.conf)
apache    3765  0.3  7.5 463876 76388 ?        S    19:52   0:12 php-fpm: pool www
apache    3766  0.3  8.8 476208 89352 ?        S    19:52   0:10 php-fpm: pool www
apache    3767  0.7  6.2 449888 62856 ?        S    19:52   0:25 php-fpm: pool www
apache    3768  1.1  7.2 459748 72844 ?        S    19:52   0:39 php-fpm: pool www
apache    3769  0.6  8.0 468352 80712 ?        S    19:52   0:20 php-fpm: pool www
apache    3776  0.5  6.3 449896 63620 ?        S    19:52   0:18 php-fpm: pool www
apache    3828  0.4  7.8 466048 78588 ?        S    19:52   0:16 php-fpm: pool www
apache    3830  0.2  6.7 455328 67944 ?        S    19:52   0:09 php-fpm: pool www
apache    3831  0.9  7.6 463524 76724 ?        S    19:52   0:31 php-fpm: pool www
root      4396  0.0  0.0      0     0 ?        I    20:24   0:00 [kworker/0:1]
postfix   4685  0.0  0.5  81752  5672 ?        S    20:36   0:00 pickup -l -t unix -u
root      4713  0.0  0.0      0     0 ?        I    20:41   0:00 [kworker/0:2]
root      4722  0.0  0.0      0     0 ?        I    20:46   0:00 [kworker/0:0]
root      4795  0.4  1.5 278012 15268 ?        Ss   20:49   0:00 /usr/sbin/httpd -DFOREGROUND
apache    4796  0.0  1.0 319136 10276 ?        Sl   20:49   0:00 /usr/sbin/httpd -DFOREGROUND
apache    4797  0.0  1.0 319136 10276 ?        Sl   20:49   0:00 /usr/sbin/httpd -DFOREGROUND
apache    4798  0.0  1.0 515816 10288 ?        Sl   20:49   0:00 /usr/sbin/httpd -DFOREGROUND
apache    4799  0.0  1.0 319136 10276 ?        Sl   20:49   0:00 /usr/sbin/httpd -DFOREGROUND
apache    4800  0.0  1.0 319136 10276 ?        Sl   20:49   0:00 /usr/sbin/httpd -DFOREGROUND
ec2-user  4842  0.0  0.3 164364  3796 pts/0    R+   20:49   0:00 ps -aux

The process for apache and pool keep adding...

apache    4800  0.0  1.0 319136 10276 ?        Sl   20:49   0:00 /usr/sbin/httpd -DFOREGROUND
apache    3831  0.9  7.6 463524 76724 ?        S    19:52   0:31 php-fpm: pool www

Even with all of my website conf files removed from sites-enabled, it still starts with these extra ports running.

Configuration: basically an Amazon LAMP - MariaDB from amazon-linux-extras (with maria db disabled and php upgraded to 7.4. - I have a separate MySQL server) yum installed: php-cli php-pdo php-fpm php-json php-mysqlnd php-gd vsftpd epel certbot python2-certbot-apache php-pear-SOAP.noarch php-soap.x86_64 php-mbstring php-bcmath php-xml php-pclzip.noarch php-pecl-zip pcre-devel gcc zlib zlib-devel

#sudo apachectl -M

 Loaded Modules:
 core_module (static)
 so_module (static)
 http_module (static)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 allowmethods_module (shared)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_anon_module (shared)
 authn_core_module (shared)
 authn_dbd_module (shared)
 authn_dbm_module (shared)
 authn_file_module (shared)
 authn_socache_module (shared)
 authz_core_module (shared)
 authz_dbd_module (shared)
 authz_dbm_module (shared)
 authz_groupfile_module (shared)
 authz_host_module (shared)
 authz_owner_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cache_module (shared)
 cache_disk_module (shared)
 cache_socache_module (shared)
 data_module (shared)
 dbd_module (shared)
 deflate_module (shared)
 dir_module (shared)
 dumpio_module (shared)
 echo_module (shared)
 env_module (shared)
 expires_module (shared)
 ext_filter_module (shared)
 filter_module (shared)
 headers_module (shared)
 include_module (shared)
 info_module (shared)
 log_config_module (shared)
 logio_module (shared)
 macro_module (shared)
 mime_magic_module (shared)
 mime_module (shared)
 negotiation_module (shared)
 remoteip_module (shared)
 reqtimeout_module (shared)
 request_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 slotmem_plain_module (shared)
 slotmem_shm_module (shared)
 socache_dbm_module (shared)
 socache_memcache_module (shared)
 socache_shmcb_module (shared)
 status_module (shared)
 substitute_module (shared)
 suexec_module (shared)
 unique_id_module (shared)
 unixd_module (shared)
 userdir_module (shared)
 version_module (shared)
 vhost_alias_module (shared)
 watchdog_module (shared)
 dav_module (shared)
 dav_fs_module (shared)
 dav_lock_module (shared)
 lua_module (shared)
 mpm_prefork_module (shared)
 proxy_module (shared)
 lbmethod_bybusyness_module (shared)
 lbmethod_byrequests_module (shared)
 lbmethod_bytraffic_module (shared)
 lbmethod_heartbeat_module (shared)
 proxy_ajp_module (shared)
 proxy_balancer_module (shared)
 proxy_connect_module (shared)
 proxy_express_module (shared)
 proxy_fcgi_module (shared)
 proxy_fdpass_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_hcheck_module (shared)
 proxy_scgi_module (shared)
 proxy_uwsgi_module (shared)
 proxy_wstunnel_module (shared)
 ssl_module (shared)
 systemd_module (shared)
 cgi_module (shared)
 http2_module (shared)
 proxy_http2_module (shared)

The httpd.conf is default with 2 exceptions: I changed the DocumentRoot and added a path to my site conf files:

IncludeOptional sites-enabled/*.conf

Finally the www.conf under /etc/php-fpm.d (I trimmed off some comments for this post)

; Start a new pool named 'www'.
; the variable $pool can we used in any directive and will be replaced by the
; pool name ('www' here)
[www]

; Per pool prefix
; It only applies on the following directives:
; - 'access.log'
; - 'slowlog'
; - 'listen' (unixsocket)
; - 'chroot'
; - 'chdir'
; - 'php_values'
; - 'php_admin_values'
; When not set, the global prefix (or @php_fpm_prefix@) applies instead.
; Note: This directive can also be relative to the global prefix.
; Default Value: none
;prefix = /path/to/pools/$pool

; Unix user/group of processes
; Note: The user is mandatory. If the group is not set, the default user's group
;       will be used.
; RPM: apache user chosen to provide access to the same directories as httpd
user = apache
; RPM: Keep a group allowed to write in log dir.
group = apache

; The address on which to accept FastCGI requests.
listen = /run/php-fpm/www.sock

; Set listen(2) backlog.
; Default Value: 511
;listen.backlog = 511

; Set permissions for unix socket, if one is used. In Linux, read/write
; permissions must be set in order to allow connections from a web server.
; Default Values: user and group are set as the running user
;                 mode is set to 0660
;listen.owner = nobody
;listen.group = nobody
;listen.mode = 0660

; When POSIX Access Control Lists are supported you can set them using
; these options, value is a comma separated list of user/group names.
; When set, listen.owner and listen.group are ignored
listen.acl_users = apache,nginx
;listen.acl_groups =

; List of addresses (IPv4/IPv6) of FastCGI clients which are allowed to connect.
; Equivalent to the FCGI_WEB_SERVER_ADDRS environment variable in the original
; PHP FCGI (5.2.2+). Makes sense only with a tcp listening socket. Each address
; must be separated by a comma. If this value is left blank, connections will be
; accepted from any ip address.
; Default Value: any
listen.allowed_clients = 127.0.0.1

; Choose how the process manager will control the number of child processes.
pm = dynamic

; The number of child processes to be created when pm is set to 'static' and the
; maximum number of child processes when pm is set to 'dynamic' or 'ondemand'.
; This value sets the limit on the number of simultaneous requests that will be
; served. Equivalent to the ApacheMaxClients directive with mpm_prefork.
; Equivalent to the PHP_FCGI_CHILDREN environment variable in the original PHP
; CGI. The below defaults are based on a server without much resources. Don't
; forget to tweak pm.* to fit your needs.
; Note: Used when pm is set to 'static', 'dynamic' or 'ondemand'
; Note: This value is mandatory.
pm.max_children = 50

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 5

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 5

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 35

; The number of seconds after which an idle process will be killed.
; Note: Used only when pm is set to 'ondemand'
; Default Value: 10s
;pm.process_idle_timeout = 10s;

; The number of requests each child process should execute before respawning.
; This can be useful to work around memory leaks in 3rd party libraries. For
; endless request processing specify '0'. Equivalent to PHP_FCGI_MAX_REQUESTS.
; Default Value: 0
;pm.max_requests = 500

; The access log format.
; The following syntax is allowed
;  %%: the '%' character
;  %C: %CPU used by the request
;      it can accept the following format:
;      - %{user}C for user CPU only
;      - %{system}C for system CPU only
;      - %{total}C  for user + system CPU (default)
;  %d: time taken to serve the request
;      it can accept the following format:
;      - %{seconds}d (default)
;      - %{miliseconds}d
;      - %{mili}d
;      - %{microseconds}d
;      - %{micro}d
;  %e: an environment variable (same as $_ENV or $_SERVER)
;      it must be associated with embraces to specify the name of the env
;      variable. Some exemples:
;      - server specifics like: %{REQUEST_METHOD}e or %{SERVER_PROTOCOL}e
;      - HTTP headers like: %{HTTP_HOST}e or %{HTTP_USER_AGENT}e
;  %f: script filename
;  %l: content-length of the request (for POST request only)
;  %m: request method
;  %M: peak of memory allocated by PHP
;      it can accept the following format:
;      - %{bytes}M (default)
;      - %{kilobytes}M
;      - %{kilo}M
;      - %{megabytes}M
;      - %{mega}M
;  %n: pool name
;  %o: output header
;      it must be associated with embraces to specify the name of the header:
;      - %{Content-Type}o
;      - %{X-Powered-By}o
;      - %{Transfert-Encoding}o
;      - ....
;  %p: PID of the child that serviced the request
;  %P: PID of the parent of the child that serviced the request
;  %q: the query string
;  %Q: the '?' character if query string exists
;  %r: the request URI (without the query string, see %q and %Q)
;  %R: remote IP address
;  %s: status (response code)
;  %t: server time the request was received
;      it can accept a strftime(3) format:
;      %d/%b/%Y:%H:%M:%S %z (default)
;      The strftime(3) format must be encapsuled in a %{<strftime_format>}t tag
;      e.g. for a ISO8601 formatted timestring, use: %{%Y-%m-%dT%H:%M:%S%z}t
;  %T: time the log has been written (the request has finished)
;      it can accept a strftime(3) format:
;      %d/%b/%Y:%H:%M:%S %z (default)
;      The strftime(3) format must be encapsuled in a %{<strftime_format>}t tag
;      e.g. for a ISO8601 formatted timestring, use: %{%Y-%m-%dT%H:%M:%S%z}t
;  %u: remote user
;
; Default: "%R - %u %t \"%m %r\" %s"
;access.format = "%R - %u %t \"%m %r%Q%q\" %s %f %{mili}d %{kilo}M %C%%"

; The log file for slow requests
; Default Value: not set
; Note: slowlog is mandatory if request_slowlog_timeout is set
slowlog = /var/log/php-fpm/www-slow.log

; The timeout for serving a single request after which a PHP backtrace will be
; dumped to the 'slowlog' file. A value of '0s' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_slowlog_timeout = 0

; The timeout for serving a single request after which the worker process will
; be killed. This option should be used when the 'max_execution_time' ini option
; does not stop script execution for some reason. A value of '0' means 'off'.
; Available units: s(econds)(default), m(inutes), h(ours), or d(ays)
; Default Value: 0
;request_terminate_timeout = 0

; Set open file descriptor rlimit.
; Default Value: system defined value
;rlimit_files = 1024

; Set max core size rlimit.
; Possible Values: 'unlimited' or an integer greater or equal to 0
; Default Value: system defined value
;rlimit_core = 0

; Chroot to this directory at the start. This value must be defined as an
; absolute path. When this value is not set, chroot is not used.
; Note: you can prefix with '$prefix' to chroot to the pool prefix or one
; of its subdirectories. If the pool prefix is not set, the global prefix
; will be used instead.
; Note: chrooting is a great security feature and should be used whenever
;       possible. However, all PHP paths will be relative to the chroot
;       (error_log, sessions.save_path, ...).
; Default Value: not set
;chroot = 

; Chdir to this directory at the start.
; Note: relative path can be used.
; Default Value: current directory or / when chroot
;chdir = /var/www

; Redirect worker stdout and stderr into main error log. If not set, stdout and
; stderr will be redirected to /dev/null according to FastCGI specs.
; Note: on highloaded environement, this can cause some delay in the page
; process time (several ms).
; Default Value: no
;catch_workers_output = yes

; Clear environment in FPM workers
; Prevents arbitrary environment variables from reaching FPM worker processes
; by clearing the environment in workers before env vars specified in this
; pool configuration are added.
; Setting to "no" will make all environment variables available to PHP code
; via getenv(), $_ENV and $_SERVER.
; Default Value: yes
;clear_env = no

; Limits the extensions of the main script FPM will allow to parse. This can
; prevent configuration mistakes on the web server side. You should only limit
; FPM to .php extensions to prevent malicious users to use other extensions to
; exectute php code.
; Note: set an empty value to allow all extensions.
; Default Value: .php
;security.limit_extensions = .php .php3 .php4 .php5 .php7

; Pass environment variables like LD_LIBRARY_PATH. All $VARIABLEs are taken from
; the current environment.
; Default Value: clean env
;env[HOSTNAME] = $HOSTNAME
;env[PATH] = /usr/local/bin:/usr/bin:/bin
;env[TMP] = /tmp
;env[TMPDIR] = /tmp
;env[TEMP] = /tmp

; Additional php.ini defines, specific to this pool of workers. These settings
; overwrite the values previously defined in the php.ini. The directives are the
; same as the PHP SAPI:
;   php_value/php_flag             - you can set classic ini defines which can
;                                    be overwritten from PHP call 'ini_set'. 
;   php_admin_value/php_admin_flag - these directives won't be overwritten by
;                                     PHP call 'ini_set'
; For php_*flag, valid values are on, off, 1, 0, true, false, yes or no.

; Defining 'extension' will load the corresponding shared extension from
; extension_dir. Defining 'disable_functions' or 'disable_classes' will not
; overwrite previously defined php.ini values, but will append the new value
; instead.

; Note: path INI options can be relative and will be expanded with the prefix
; (pool, global or @prefix@)

; Default Value: nothing is defined by default except the values in php.ini and
;                specified at startup with the -d argument
;php_admin_value[sendmail_path] = /usr/sbin/sendmail -t -i -f [email protected]
;php_flag[display_errors] = off
php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on
;php_admin_value[memory_limit] = 128M

; Set the following data paths to directories owned by the FPM process user.
;
; Do not change the ownership of existing system directories, if the process
; user does not have write permission, create dedicated directories for this
; purpose.
;
; See warning about choosing the location of these directories on your system
; at http://php.net/session.save-path
php_value[session.save_handler] = files
php_value[session.save_path]    = /var/lib/php/session
php_value[soap.wsdl_cache_dir]  = /var/lib/php/wsdlcache
;php_value[opcache.file_cache]  = /var/lib/php/opcache

the only error information I found is:

ssl_error_log

[Tue Jun 09 21:19:06.680548 2020] [proxy_fcgi:error] [pid 4877] [client ###.###.###.###:56480] AH01071: Got error 'Primary script unknown'

error_log   (repeated entries for the following)

[Tue Jun 09 20:49:20.046026 2020] [mpm_prefork:notice] [pid 4731] AH00170: caught SIGWINCH, shutting down gracefully
[Tue Jun 09 20:49:21.126867 2020] [suexec:notice] [pid 4795] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Jun 09 20:49:21.168981 2020] [lbmethod_heartbeat:notice] [pid 4795] AH02282: No slotmem from mod_heartmonitor
[Tue Jun 09 20:49:21.169053 2020] [http2:warn] [pid 4795] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are processed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will continue to work, but the HTTP/2 protocol will be inactive.
[Tue Jun 09 20:49:21.171476 2020] [mpm_prefork:notice] [pid 4795] AH00163: Apache/2.4.43 () OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Tue Jun 09 20:49:21.171494 2020] [core:notice] [pid 4795] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

2 Answers 2

1

The Apache server works by creating child processes to serve requests. How these child processes are managed, depends upon the MPM or Multi-Processing Module being used. You can use configuration directives such as MaxRequestWorkers, ThreadsPerChild and ServerLimit to control the number of child process and threads being used to serve requests. See the Apache documentation for more information on MPMs.

The php-fpm module works in a similar way. Based on the configuration you have shown, you have set pm = dynamic and pm.max_children = 50. This allows the module to have up to 50 child processes at a time to serve requests. You can find the config documentation for the php-fpm module here: https://www.php.net/manual/en/install.fpm.configuration.php


Using the ps command, you can verify that the process count only changes between the specified limits. You may also want to use a different view (such as -e) that highlights the parent PID of each process.

Regarding the use of lsof, you should be aware that lsof shows open file descriptors. This includes sockets, but the same file descriptors are also inherited by each child process. This is the reason for the repeated entries corresponding to the same socket. At the network/interface level, it is not possible for more than one process to bind to the same port number at the same time.

1
  • That is exactly what I was looking for, the variables that give me a way to set the limits. thank you! Commented Jun 12, 2020 at 21:38
0

If you can confirm my logic, please do... but I think I figured it out, I don't think it is running multiple port 80's and 443's... I think it is the lsof command that is showing something different.

when I do this instead it shows one of each:

sudo netstat -tulpn

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1961/rpcbind
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2705/sshd
tcp        0      0 0.0.0.0:3000            0.0.0.0:*               LISTEN      2598/node /var/www/
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      2492/master
tcp6       0      0 :::111                  :::*                    LISTEN      1961/rpcbind
tcp6       0      0 :::80                   :::*                    LISTEN      2349/httpd
tcp6       0      0 :::21                   :::*                    LISTEN      2351/vsftpd
tcp6       0      0 :::22                   :::*                    LISTEN      2705/sshd
tcp6       0      0 :::443                  :::*                    LISTEN      2349/httpd
udp        0      0 0.0.0.0:859             0.0.0.0:*                           1961/rpcbind
udp        0      0 0.0.0.0:68              0.0.0.0:*                           2184/dhclient
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1961/rpcbind
udp        0      0 127.0.0.1:323           0.0.0.0:*                           1968/chronyd
udp6       0      0 :::859                  :::*                                1961/rpcbind
udp6       0      0 :::111                  :::*                                1961/rpcbind
udp6       0      0 ::1:323                 :::*                                1968/chronyd
udp6       0      0 fe80::1f:48ff:fe35::546 :::*                                2302/dhclient

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .