Pacemaker apache resource is Failed to access httpd status page after change to HTTPS

Question

I get this error from pacemaker after i change apache from http to https. now my ocf::heartbeat:apache resource is not find status page.

I generate SSL certificate separately for 3 servers.

Everything was working fine when running on http but as soon as I added the (self-signed) SSL certificate pacemaker Apache (ocf::heartbeat:apache): Stopped

And error shows

Failed Actions:
* Apache_start_0 on server3 'unknown error' (1): call=315, status=complete, exitreason='Failed to access httpd status page.',
    last-rc-change='Mon Sep 21 16:22:37 2020', queued=0ms, exec=3456ms
* Apache_start_0 on server1 'unknown error' (1): call=59, status=complete, exitreason='Failed to access httpd status page.',
    last-rc-change='Mon Sep 21 16:22:41 2020', queued=0ms, exec=3421ms
* Apache_start_0 on server2 'unknown error' (1): call=197, status=complete, exitreason='Failed to access httpd status page.',
    last-rc-change='Mon Sep 21 16:22:33 2020', queued=0ms, exec=3451ms

/etc/apache2/sites-available/000-default.conf

<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html
        Redirect "/" "https://10.226.***.***/"

<Location /server-status>
 SetHandler server-status        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html
        Redirect "/" "https://10.226.179.205/"

 Order deny,allow
 Deny from all
 Allow from 127.0.0.1
</Location>
</VirtualHost>

pcs resource debug-monitor --full Apache

Operation monitor for Apache (ocf:heartbeat:apache) returned 1
 >  stderr: + echo
 >  stderr: + printenv
 >  stderr: + sort
 >  stderr: + env=
 >  stderr: AONIX_LM_DIR=/home/TeleUSE/etc
 >  stderr: BXwidgets=/home/BXwidgets
 >  stderr: HA_logfacility=none
 >  stderr: HOME=/root
 >  stderr: LC_ALL=C
 >  stderr: LOGNAME=root
 >  stderr: MAIL=/var/mail/root
 >  stderr: OCF_EXIT_REASON_PREFIX=ocf-exit-reason:
 >  stderr: OCF_RA_VERSION_MAJOR=1
 >  stderr: OCF_RA_VERSION_MINOR=0
 >  stderr: OCF_RESKEY_CRM_meta_class=ocf
 >  stderr: OCF_RESKEY_CRM_meta_id=Apache
 >  stderr: OCF_RESKEY_CRM_meta_migration_threshold=5
 >  stderr: OCF_RESKEY_CRM_meta_provider=heartbeat
 >  stderr: OCF_RESKEY_CRM_meta_resource_stickiness=10
 >  stderr: OCF_RESKEY_CRM_meta_type=apache
 >  stderr: OCF_RESKEY_configfile=/etc/apache2/apache2.conf
 >  stderr: OCF_RESKEY_statusurl=http://localhost/server-status
 >  stderr: OCF_RESOURCE_INSTANCE=Apache
 >  stderr: OCF_RESOURCE_PROVIDER=heartbeat
 >  stderr: OCF_RESOURCE_TYPE=apache
 >  stderr: OCF_ROOT=/usr/lib/ocf
 >  stderr: OCF_TRACE_RA=1
 >  stderr: PATH=/root/.rbenv/shims:/root/.rbenv/bin:/root/.rbenv/shims:/root/.rbenv/bin:/usr/local/bin:/home/TeleUSE/bin:/home/xrt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb
 >  stderr: PCMK_logfacility=none
 >  stderr: PCMK_service=crm_resource
 >  stderr: PWD=/root
 >  stderr: RBENV_SHELL=bash
 >  stderr: SHELL=/bin/bash
 >  stderr: SHLVL=1
 >  stderr: SSH_CLIENT=10.12.116.46 63097 22
 >  stderr: SSH_CONNECTION=10.12.116.46 63097 10.226.179.205 22
 >  stderr: SSH_TTY=/dev/pts/0
 >  stderr: TERM=xterm
 >  stderr: TeleUSE=/home/TeleUSE
 >  stderr: USER=root
 >  stderr: _=/usr/sbin/pcs
 >  stderr: __OCF_TRC_DEST=
 >  stderr: __OCF_TRC_MANAGE=
 >  stderr: + ocf_is_true
 >  stderr: + false
 >  stderr: + . /usr/lib/ocf/lib/heartbeat/apache-conf.sh
 >  stderr: + . /usr/lib/ocf/lib/heartbeat/http-mon.sh
 >  stderr: + bind_address=127.0.0.1
 >  stderr: + curl_ipv6_opts=
 >  stderr: + ocf_is_true
 >  stderr: + false
 >  stderr: + echo
 >  stderr: + grep -qs ::
 >  stderr: + WGETOPTS=-O- -q -L --no-proxy --bind-address=127.0.0.1
 >  stderr: + CURLOPTS=-o - -Ss -L --interface lo
 >  stderr: + HA_VARRUNDIR=/var/run
 >  stderr: + IBMHTTPD=/opt/IBMHTTPServer/bin/httpd
 >  stderr: + HTTPDLIST=/sbin/httpd2 /usr/sbin/httpd2 /usr/sbin/apache2 /sbin/httpd /usr/sbin/httpd /usr/sbin/apache /opt/IBMHTTPServer/bin/httpd
 >  stderr: + MPM=/usr/share/apache2/find_mpm
 >  stderr: + [ -x /usr/share/apache2/find_mpm ]
 >  stderr: + LOCALHOST=http://localhost
 >  stderr: + HTTPDOPTS=-DSTATUS
 >  stderr: + DEFAULT_IBMCONFIG=/opt/IBMHTTPServer/conf/httpd.conf
 >  stderr: + DEFAULT_SUSECONFIG=/etc/apache2/httpd.conf
 >  stderr: + DEFAULT_RHELCONFIG=/etc/httpd/conf/httpd.conf
 >  stderr: + DEFAULT_DEBIANCONFIG=/etc/apache2/apache2.conf
 >  stderr: + basename /usr/lib/ocf/resource.d/heartbeat/apache
 >  stderr: + CMD=apache
 >  stderr: + OCF_REQUIRED_PARAMS=
 >  stderr: + OCF_REQUIRED_BINARIES=
 >  stderr: + ocf_rarun monitor
 >  stderr: + mk_action_func
 >  stderr: + echo apache_monitor
 >  stderr: + tr - _
 >  stderr: + ACTION_FUNC=apache_monitor
 >  stderr: + validate_args
 >  stderr: + is_function apache_monitor
 >  stderr: + command -v apache_monitor
 >  stderr: + test zapache_monitor = zapache_monitor
 >  stderr: + simple_actions
 >  stderr: + check_required_params
 >  stderr: + local v
 >  stderr: + run_function apache_getconfig
 >  stderr: + is_function apache_getconfig
 >  stderr: + command -v apache_getconfig
 >  stderr: + test zapache_getconfig = zapache_getconfig
 >  stderr: + apache_getconfig
 >  stderr: + HTTPD=
 >  stderr: + PORT=
 >  stderr: + STATUSURL=http://localhost/server-status
 >  stderr: + CONFIGFILE=/etc/apache2/apache2.conf
 >  stderr: + OPTIONS=
 >  stderr: + CLIENT=
 >  stderr: + TESTREGEX=</ *html *>
 >  stderr: + TESTURL=
 >  stderr: + TESTREGEX10=
 >  stderr: + TESTCONFFILE=
 >  stderr: + TESTNAME=
 >  stderr: + : /etc/apache2/envvars
 >  stderr: + source_envfiles /etc/apache2/envvars
 >  stderr: + [ -f /etc/apache2/envvars -a -r /etc/apache2/envvars ]
 >  stderr: + . /etc/apache2/envvars
 >  stderr: + unset HOME
 >  stderr: + [  !=  ]
 >  stderr: + SUFFIX=
 >  stderr: + export APACHE_RUN_USER=www-data
 >  stderr: + export APACHE_RUN_GROUP=www-data
 >  stderr: + export APACHE_PID_FILE=/var/run/apache2/apache2.pid
 >  stderr: + export APACHE_RUN_DIR=/var/run/apache2
 >  stderr: + export APACHE_LOCK_DIR=/var/lock/apache2
 >  stderr: + export APACHE_LOG_DIR=/var/log/apache2
 >  stderr: + export LANG=C
 >  stderr: + export LANG
 >  stderr: + [ X = X -o ! -f  -o ! -x  ]
 >  stderr: + find_httpd_prog
 >  stderr: + HTTPD=
 >  stderr: + [ -f /sbin/httpd2 -a -x /sbin/httpd2 ]
 >  stderr: + [ -f /usr/sbin/httpd2 -a -x /usr/sbin/httpd2 ]
 >  stderr: + [ -f /usr/sbin/apache2 -a -x /usr/sbin/apache2 ]
 >  stderr: + HTTPD=/usr/sbin/apache2
 >  stderr: + break
 >  stderr: + [ X != X -a X/usr/sbin/apache2 != X ]
 >  stderr: + detect_default_config
 >  stderr: + [ -f /etc/apache2/httpd.conf ]
 >  stderr: + [ -f /etc/apache2/apache2.conf ]
 >  stderr: + echo /etc/apache2/apache2.conf
 >  stderr: + DefaultConfig=/etc/apache2/apache2.conf
 >  stderr: + CONFIGFILE=/etc/apache2/apache2.conf
 >  stderr: + [ -n /usr/sbin/apache2 ]
 >  stderr: + basename /usr/sbin/apache2
 >  stderr: + httpd_basename=apache2
 >  stderr: + GetParams /etc/apache2/apache2.conf
 >  stderr: + ConfigFile=/etc/apache2/apache2.conf
 >  stderr: + [ ! -f /etc/apache2/apache2.conf ]
 >  stderr: + get_apache_params /etc/apache2/apache2.conf ServerRoot PidFile Port Listen
 >  stderr: + configfile=/etc/apache2/apache2.conf
 >  stderr: + shift 1
 >  stderr: + echo ServerRoot PidFile Port Listen
 >  stderr: + sed s/ /,/g
 >  stderr: + vars=ServerRoot,PidFile,Port,Listen
 >  stderr: + apachecat /etc/apache2/apache2.conf
 >  stderr: + awk -v vars=ServerRoot,PidFile,Port,Listen
 >  stderr:     BEGIN{
 >  stderr:             split(vars,v,",");
 >  stderr:             for( i in v )
 >  stderr:                     vl[i]=tolower(v[i]);
 >  stderr:     }
 >  stderr:     {
 >  stderr:             for( i in v )
 >  stderr:                     if( tolower($1)==vl[i] ) {
 >  stderr:                     print v[i]"="$2
 >  stderr:                     delete vl[i]
 >  stderr:                     break
 >  stderr:             }
 >  stderr:     }
 >  stderr:
 >  stderr: + awk
 >  stderr:     function procline() {
 >  stderr:             split($0,a);
 >  stderr:             if( a[1]~/^[Ii]nclude$/ ) {
 >  stderr:                     includedir=a[2];
 >  stderr:                     gsub("\"","",includedir);
 >  stderr:                     procinclude(includedir);
 >  stderr:             } else {
 >  stderr:                     if( a[1]=="ServerRoot" ) {
 >  stderr:                             rootdir=a[2];
 >  stderr:                             gsub("\"","",rootdir);
 >  stderr:                     }
 >  stderr:                     print;
 >  stderr:             }
 >  stderr:     }
 >  stderr:     function printfile(infile, a) {
 >  stderr:             while( (getline<infile) > 0 ) {
 >  stderr:                     procline();
 >  stderr:             }
 >  stderr:             close(infile);
 >  stderr:     }
 >  stderr:     function allfiles(dir, cmd,f) {
 >  stderr:             cmd="find -L "dir" -type f";
 >  stderr:             while( ( cmd | getline f ) > 0 ) {
 >  stderr:                     printfile(f);
 >  stderr:             }
 >  stderr:             close(cmd);
 >  stderr:     }
 >  stderr:     function listfiles(pattern, cmd,f) {
 >  stderr:             cmd="ls "pattern" 2>/dev/null";
 >  stderr:             while( ( cmd | getline f ) > 0 ) {
 >  stderr:                     printfile(f);
 >  stderr:             }
 >  stderr:             close(cmd);
 >  stderr:     }
 >  stderr:     function procinclude(spec) {
 >  stderr:             if( rootdir!="" && spec!~/^\// ) {
 >  stderr:                     spec=rootdir"/"spec;
 >  stderr:             }
 >  stderr:             if( isdir(spec) ) {
 >  stderr:                     allfiles(spec); # read all files in a directory (and subdirs)
 >  stderr:             } else {
 >  stderr:                     listfiles(spec); # there could be jokers
 >  stderr:             }
 >  stderr:     }
 >  stderr:     function isdir(s) {
 >  stderr:             return !system("test -d \""s"\"");
 >  stderr:     }
 >  stderr:     { procline(); }
 >  stderr:      /etc/apache2/apache2.conf
 >  stderr: + sed s/#.*//;s/[[:blank:]]*$//;s/^[[:blank:]]*//
 >  stderr: + grep -v ^$
 >  stderr: + eval PidFile=${APACHE_PID_FILE}
 >  stderr: + PidFile=/var/run/apache2/apache2.pid
 >  stderr: + CheckPort
 >  stderr: + ocf_is_decimal
 >  stderr: + false
 >  stderr: + CheckPort
 >  stderr: + ocfError performing operation: Operation not permitted
_is_decimal
 >  stderr: + false
 >  stderr: + CheckPort 80
 >  stderr: + ocf_is_decimal 80
 >  stderr: + true
 >  stderr: + [ 80 -gt 0 ]
 >  stderr: + PORT=80
 >  stderr: + break
 >  stderr: + echo
 >  stderr: + grep :
 >  stderr: + Listen=localhost:
 >  stderr: + [ Xhttp://localhost/server-status = X ]
 >  stderr: + test /var/run/apache2/apache2.pid
 >  stderr: + return 0
 >  stderr: + validate_env
 >  stderr: + check_required_binaries
 >  stderr: + local v
 >  stderr: + is_function apache_validate_all
 >  stderr: + command -v apache_validate_all
 >  stderr: + test zapache_validate_all = zapache_validate_all
 >  stderr: + local rc
 >  stderr: + LSB_STATUS_STOPPED=3
 >  stderr: + apache_validate_all
 >  stderr: + [ -z /usr/sbin/apache2 ]
 >  stderr: + [ ! -x /usr/sbin/apache2 ]
 >  stderr: + [ ! -f /etc/apache2/apache2.conf ]
 >  stderr: + [ -n  ]
 >  stderr: + [ -n  ]
 >  stderr: + dirname /var/run/apache2/apache2.pid
 >  stderr: + local a
 >  stderr: + local b
 >  stderr: + [ 1 = 1 ]
 >  stderr: + a=/var/run/apache2/apache2.pid
 >  stderr: + [ 1 ]
 >  stderr: + b=/var/run/apache2/apache2.pid
 >  stderr: + [ /var/run/apache2/apache2.pid = /var/run/apache2/apache2.pid ]
 >  stderr: + break
 >  stderr: + b=/var/run/apache2
 >  stderr: + [ -z /var/run/apache2 -o /var/run/apache2/apache2.pid = /var/run/apache2 ]
 >  stderr: + echo /var/run/apache2
 >  stderr: + return 0
 >  stderr: + ocf_mkstatedir root 755 /var/run/apache2
 >  stderr: + local owner
 >  stderr: + local perms
 >  stderr: + local path
 >  stderr: + owner=root
 >  stderr: + perms=755
 >  stderr: + path=/var/run/apache2
 >  stderr: + test -d /var/run/apache2
 >  stderr: + return 0
 >  stderr: + return 0
 >  stderr: + rc=0
 >  stderr: + [ 0 -ne 0 ]
 >  stderr: + ocf_is_probe
 >  stderr: + [ monitor = monitor -a 0 = 0 ]
 >  stderr: + run_probe
 >  stderr: + is_function apache_probe
 >  stderr: + command -v apache_probe
 >  stderr: + test z = zapache_probe
 >  stderr: + shift 1
 >  stderr: + apache_monitor
 >  stderr: + silent_status
 >  stderr: + local pid
 >  stderr: + get_pid
 >  stderr: + [ -f /var/run/apache2/apache2.pid ]
 >  stderr: + cat /var/run/apache2/apache2.pid
 >  stderr: + pid=17552
 >  stderr: + [ -n 17552 ]
 >  stderr: + ProcessRunning 17552
 >  stderr: + local pid=17552
 >  stderr: + [ -d /proc -a -d /proc/1 ]
 >  stderr: + [ -d /proc/17552 ]
 >  stderr: + [ 0 -ne 0 ]
 >  stderr: + findhttpclient
 >  stderr: + [ x != x ]
 >  stderr: + which wget
 >  stderr: + echo wget
 >  stderr: + ourhttpclient=wget
 >  stderr: + [ -z wget ]
 >  stderr: + ocf_check_level 10
 >  stderr: + local lvl prev
 >  stderr: + lvl=0
 >  stderr: + prev=0
 >  stderr: + ocf_is_decimal 0
 >  stderr: + true
 >  stderr: + [ 10 -eq 0 ]
 >  stderr: + [ 10 -gt 0 ]
 >  stderr: + lvl=0
 >  stderr: + break
 >  stderr: + echo 0
 >  stderr: + apache_monitor_basic
 >  stderr: + wget_func http://localhost/server-status
 >  stderr: + auth=
 >  stderr: + cl_opts=-O- -q -L --no-proxy --bind-address=127.0.0.1
 >  stderr: + [ x !=+  x ]
 >  stderr: grep+ wget -Ei -O- </ *html *> -q
 >  stderr:  -L --no-proxy --bind-address=127.0.0.1 http://localhost/server-status
 >  stderr: + attempt_index_monitor_request
 >  stderr: + local indexpage=
 >  stderr: + [ -n  ]
 >  stderr: + [ -n  ]
 >  stderr: + [ -n  ]
 >  stderr: + [ -n http://localhost/server-status ]
 >  stderr: + return 1
 >  stderr: + [ 1 -eq 0 ]
 >  stderr: + ocf_is_probe
 >  stderr: + [ monitor = monitor -a 0 = 0 ]
 >  stderr: + return 1

pcs config

 Resource: MasterVip (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=10.226.***.*** nic=lo cidr_netmask=32 iflabel=pgrepvip
  Meta Attrs: target-role=Started
  Operations: start interval=0s timeout=20s (MasterVip-start-interval-0s)
              stop interval=0s timeout=20s (MasterVip-stop-interval-0s)
              monitor interval=90s (MasterVip-monitor-interval-90s)

 Resource: Apache (class=ocf provider=heartbeat type=apache)
  Attributes: configfile=/etc/apache2/apache2.conf statusurl=http://localhost/server-status
  Operations: start interval=0s timeout=40s (Apache-start-interval-0s)
              stop interval=0s timeout=60s (Apache-stop-interval-0s)
              monitor interval=1min (Apache-monitor-interval-1min)

I don't know how to fix this. if anyone knows please help me.

kobi · Accepted Answer · 2020-09-22 11:36:23Z

It seems, this resource-agent uses wget (or curl) for statusurl verification. Both commands fail if a self-signed certificate is used.

I ran into the same problem after using a self-signed certificate for my tomcat https connector. Only solution I found to date was to either add the parameter --no-check-certificate to the wget call in the resource-agent file (ocf/resource.d/heartbeat/tomcat):

isrunning_tomcat()
{
    $WGET --no-check-certificate --tries=20 -O /dev/null $RESOURCE_STATUSURL >/dev/null 2>&1
}

or to add it directly to the statusurl of the pcs resource:

statusurl="--no-check-certificate https://example-host:8443/somewebapp"

In the apache resource-agent file (ocf/resource.d/heartbeat/apache) you can specify which http client to use for verification:

<parameter name="client">
    <longdesc lang="en">
        Client to use to query to Apache. If not specified, the RA will
        try to find one on the system. Currently, wget and curl are
        supported. For example, you can set this parameter to "curl" if
        you prefer that to wget.
    </longdesc>
    <shortdesc lang="en">http client</shortdesc>
    <content type="string" default="wget"/>
</parameter>

Maybe you can specify wget with parameter --no-check-certificate or curl with parameter -k for verification in the resource-agent file.

Or do like I did and kind of inject it to the statusurl.

I have one more question. should i add server status section on virtual host :80 or virtual host :443? — Karippery, Sep 22 '20 at 12:30
I assume the wget command fails wether you use the --no-check-certificate param or not? What does the command return if you test it on your status url? (wget https://<server-ip>/server-status) — kobi, Sep 22 '20 at 13:48
it return WARNING: The certificate of '10.226.***.***' is not trusted. WARNING: The certificate of '10.226.***.***' hasn't got a known issuer for more details github.com/ClusterLabs/resource-agents/issues/1556 — Karippery, Sep 22 '20 at 15:11
As long as you use the --no-check-certificate param, the cert warnings can be ignored. Base problem is following: HTTP request sent, awaiting response... 403 Forbidden 2020-09-22 15:29:53 ERROR 403: Forbidden. So the access to your server-status url has been denied. Can you check if it works when you dont use the access control? — kobi, Sep 23 '20 at 5:37

Stack Exchange Network

current community

your communities

more stack exchange communities

Pacemaker apache resource is Failed to access httpd status page after change to HTTPS

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged apache-httpd cluster pacemaker high-availability heartbeat or ask your own question.

Hot Network Questions

Pacemaker apache resource is Failed to access httpd status page after change to HTTPS

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged apache-httpd cluster pacemaker high-availability heartbeat or ask your own question.

Related

Hot Network Questions