0

I get this error from pacemaker after i change apache from http to https. now my ocf::heartbeat:apache resource is not find status page.

I generate SSL certificate separately for 3 servers.

Everything was working fine when running on http but as soon as I added the (self-signed) SSL certificate pacemaker Apache (ocf::heartbeat:apache): Stopped

And error shows

Failed Actions:
* Apache_start_0 on server3 'unknown error' (1): call=315, status=complete, exitreason='Failed to access httpd status page.',
    last-rc-change='Mon Sep 21 16:22:37 2020', queued=0ms, exec=3456ms
* Apache_start_0 on server1 'unknown error' (1): call=59, status=complete, exitreason='Failed to access httpd status page.',
    last-rc-change='Mon Sep 21 16:22:41 2020', queued=0ms, exec=3421ms
* Apache_start_0 on server2 'unknown error' (1): call=197, status=complete, exitreason='Failed to access httpd status page.',
    last-rc-change='Mon Sep 21 16:22:33 2020', queued=0ms, exec=3451ms

/etc/apache2/sites-available/000-default.conf

<VirtualHost *:80>
        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html
        Redirect "/" "https://10.226.***.***/"

<Location /server-status>
 SetHandler server-status        ServerAdmin webmaster@localhost
        DocumentRoot /var/www/html
        Redirect "/" "https://10.226.179.205/"

 Order deny,allow
 Deny from all
 Allow from 127.0.0.1
</Location>
</VirtualHost>

pcs resource debug-monitor --full Apache

Operation monitor for Apache (ocf:heartbeat:apache) returned 1
 >  stderr: + echo
 >  stderr: + printenv
 >  stderr: + sort
 >  stderr: + env=
 >  stderr: AONIX_LM_DIR=/home/TeleUSE/etc
 >  stderr: BXwidgets=/home/BXwidgets
 >  stderr: HA_logfacility=none
 >  stderr: HOME=/root
 >  stderr: LC_ALL=C
 >  stderr: LOGNAME=root
 >  stderr: MAIL=/var/mail/root
 >  stderr: OCF_EXIT_REASON_PREFIX=ocf-exit-reason:
 >  stderr: OCF_RA_VERSION_MAJOR=1
 >  stderr: OCF_RA_VERSION_MINOR=0
 >  stderr: OCF_RESKEY_CRM_meta_class=ocf
 >  stderr: OCF_RESKEY_CRM_meta_id=Apache
 >  stderr: OCF_RESKEY_CRM_meta_migration_threshold=5
 >  stderr: OCF_RESKEY_CRM_meta_provider=heartbeat
 >  stderr: OCF_RESKEY_CRM_meta_resource_stickiness=10
 >  stderr: OCF_RESKEY_CRM_meta_type=apache
 >  stderr: OCF_RESKEY_configfile=/etc/apache2/apache2.conf
 >  stderr: OCF_RESKEY_statusurl=http://localhost/server-status
 >  stderr: OCF_RESOURCE_INSTANCE=Apache
 >  stderr: OCF_RESOURCE_PROVIDER=heartbeat
 >  stderr: OCF_RESOURCE_TYPE=apache
 >  stderr: OCF_ROOT=/usr/lib/ocf
 >  stderr: OCF_TRACE_RA=1
 >  stderr: PATH=/root/.rbenv/shims:/root/.rbenv/bin:/root/.rbenv/shims:/root/.rbenv/bin:/usr/local/bin:/home/TeleUSE/bin:/home/xrt/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/ucb
 >  stderr: PCMK_logfacility=none
 >  stderr: PCMK_service=crm_resource
 >  stderr: PWD=/root
 >  stderr: RBENV_SHELL=bash
 >  stderr: SHELL=/bin/bash
 >  stderr: SHLVL=1
 >  stderr: SSH_CLIENT=10.12.116.46 63097 22
 >  stderr: SSH_CONNECTION=10.12.116.46 63097 10.226.179.205 22
 >  stderr: SSH_TTY=/dev/pts/0
 >  stderr: TERM=xterm
 >  stderr: TeleUSE=/home/TeleUSE
 >  stderr: USER=root
 >  stderr: _=/usr/sbin/pcs
 >  stderr: __OCF_TRC_DEST=
 >  stderr: __OCF_TRC_MANAGE=
 >  stderr: + ocf_is_true
 >  stderr: + false
 >  stderr: + . /usr/lib/ocf/lib/heartbeat/apache-conf.sh
 >  stderr: + . /usr/lib/ocf/lib/heartbeat/http-mon.sh
 >  stderr: + bind_address=127.0.0.1
 >  stderr: + curl_ipv6_opts=
 >  stderr: + ocf_is_true
 >  stderr: + false
 >  stderr: + echo
 >  stderr: + grep -qs ::
 >  stderr: + WGETOPTS=-O- -q -L --no-proxy --bind-address=127.0.0.1
 >  stderr: + CURLOPTS=-o - -Ss -L --interface lo
 >  stderr: + HA_VARRUNDIR=/var/run
 >  stderr: + IBMHTTPD=/opt/IBMHTTPServer/bin/httpd
 >  stderr: + HTTPDLIST=/sbin/httpd2 /usr/sbin/httpd2 /usr/sbin/apache2 /sbin/httpd /usr/sbin/httpd /usr/sbin/apache /opt/IBMHTTPServer/bin/httpd
 >  stderr: + MPM=/usr/share/apache2/find_mpm
 >  stderr: + [ -x /usr/share/apache2/find_mpm ]
 >  stderr: + LOCALHOST=http://localhost
 >  stderr: + HTTPDOPTS=-DSTATUS
 >  stderr: + DEFAULT_IBMCONFIG=/opt/IBMHTTPServer/conf/httpd.conf
 >  stderr: + DEFAULT_SUSECONFIG=/etc/apache2/httpd.conf
 >  stderr: + DEFAULT_RHELCONFIG=/etc/httpd/conf/httpd.conf
 >  stderr: + DEFAULT_DEBIANCONFIG=/etc/apache2/apache2.conf
 >  stderr: + basename /usr/lib/ocf/resource.d/heartbeat/apache
 >  stderr: + CMD=apache
 >  stderr: + OCF_REQUIRED_PARAMS=
 >  stderr: + OCF_REQUIRED_BINARIES=
 >  stderr: + ocf_rarun monitor
 >  stderr: + mk_action_func
 >  stderr: + echo apache_monitor
 >  stderr: + tr - _
 >  stderr: + ACTION_FUNC=apache_monitor
 >  stderr: + validate_args
 >  stderr: + is_function apache_monitor
 >  stderr: + command -v apache_monitor
 >  stderr: + test zapache_monitor = zapache_monitor
 >  stderr: + simple_actions
 >  stderr: + check_required_params
 >  stderr: + local v
 >  stderr: + run_function apache_getconfig
 >  stderr: + is_function apache_getconfig
 >  stderr: + command -v apache_getconfig
 >  stderr: + test zapache_getconfig = zapache_getconfig
 >  stderr: + apache_getconfig
 >  stderr: + HTTPD=
 >  stderr: + PORT=
 >  stderr: + STATUSURL=http://localhost/server-status
 >  stderr: + CONFIGFILE=/etc/apache2/apache2.conf
 >  stderr: + OPTIONS=
 >  stderr: + CLIENT=
 >  stderr: + TESTREGEX=</ *html *>
 >  stderr: + TESTURL=
 >  stderr: + TESTREGEX10=
 >  stderr: + TESTCONFFILE=
 >  stderr: + TESTNAME=
 >  stderr: + : /etc/apache2/envvars
 >  stderr: + source_envfiles /etc/apache2/envvars
 >  stderr: + [ -f /etc/apache2/envvars -a -r /etc/apache2/envvars ]
 >  stderr: + . /etc/apache2/envvars
 >  stderr: + unset HOME
 >  stderr: + [  !=  ]
 >  stderr: + SUFFIX=
 >  stderr: + export APACHE_RUN_USER=www-data
 >  stderr: + export APACHE_RUN_GROUP=www-data
 >  stderr: + export APACHE_PID_FILE=/var/run/apache2/apache2.pid
 >  stderr: + export APACHE_RUN_DIR=/var/run/apache2
 >  stderr: + export APACHE_LOCK_DIR=/var/lock/apache2
 >  stderr: + export APACHE_LOG_DIR=/var/log/apache2
 >  stderr: + export LANG=C
 >  stderr: + export LANG
 >  stderr: + [ X = X -o ! -f  -o ! -x  ]
 >  stderr: + find_httpd_prog
 >  stderr: + HTTPD=
 >  stderr: + [ -f /sbin/httpd2 -a -x /sbin/httpd2 ]
 >  stderr: + [ -f /usr/sbin/httpd2 -a -x /usr/sbin/httpd2 ]
 >  stderr: + [ -f /usr/sbin/apache2 -a -x /usr/sbin/apache2 ]
 >  stderr: + HTTPD=/usr/sbin/apache2
 >  stderr: + break
 >  stderr: + [ X != X -a X/usr/sbin/apache2 != X ]
 >  stderr: + detect_default_config
 >  stderr: + [ -f /etc/apache2/httpd.conf ]
 >  stderr: + [ -f /etc/apache2/apache2.conf ]
 >  stderr: + echo /etc/apache2/apache2.conf
 >  stderr: + DefaultConfig=/etc/apache2/apache2.conf
 >  stderr: + CONFIGFILE=/etc/apache2/apache2.conf
 >  stderr: + [ -n /usr/sbin/apache2 ]
 >  stderr: + basename /usr/sbin/apache2
 >  stderr: + httpd_basename=apache2
 >  stderr: + GetParams /etc/apache2/apache2.conf
 >  stderr: + ConfigFile=/etc/apache2/apache2.conf
 >  stderr: + [ ! -f /etc/apache2/apache2.conf ]
 >  stderr: + get_apache_params /etc/apache2/apache2.conf ServerRoot PidFile Port Listen
 >  stderr: + configfile=/etc/apache2/apache2.conf
 >  stderr: + shift 1
 >  stderr: + echo ServerRoot PidFile Port Listen
 >  stderr: + sed s/ /,/g
 >  stderr: + vars=ServerRoot,PidFile,Port,Listen
 >  stderr: + apachecat /etc/apache2/apache2.conf
 >  stderr: + awk -v vars=ServerRoot,PidFile,Port,Listen
 >  stderr:     BEGIN{
 >  stderr:             split(vars,v,",");
 >  stderr:             for( i in v )
 >  stderr:                     vl[i]=tolower(v[i]);
 >  stderr:     }
 >  stderr:     {
 >  stderr:             for( i in v )
 >  stderr:                     if( tolower($1)==vl[i] ) {
 >  stderr:                     print v[i]"="$2
 >  stderr:                     delete vl[i]
 >  stderr:                     break
 >  stderr:             }
 >  stderr:     }
 >  stderr:
 >  stderr: + awk
 >  stderr:     function procline() {
 >  stderr:             split($0,a);
 >  stderr:             if( a[1]~/^[Ii]nclude$/ ) {
 >  stderr:                     includedir=a[2];
 >  stderr:                     gsub("\"","",includedir);
 >  stderr:                     procinclude(includedir);
 >  stderr:             } else {
 >  stderr:                     if( a[1]=="ServerRoot" ) {
 >  stderr:                             rootdir=a[2];
 >  stderr:                             gsub("\"","",rootdir);
 >  stderr:                     }
 >  stderr:                     print;
 >  stderr:             }
 >  stderr:     }
 >  stderr:     function printfile(infile, a) {
 >  stderr:             while( (getline<infile) > 0 ) {
 >  stderr:                     procline();
 >  stderr:             }
 >  stderr:             close(infile);
 >  stderr:     }
 >  stderr:     function allfiles(dir, cmd,f) {
 >  stderr:             cmd="find -L "dir" -type f";
 >  stderr:             while( ( cmd | getline f ) > 0 ) {
 >  stderr:                     printfile(f);
 >  stderr:             }
 >  stderr:             close(cmd);
 >  stderr:     }
 >  stderr:     function listfiles(pattern, cmd,f) {
 >  stderr:             cmd="ls "pattern" 2>/dev/null";
 >  stderr:             while( ( cmd | getline f ) > 0 ) {
 >  stderr:                     printfile(f);
 >  stderr:             }
 >  stderr:             close(cmd);
 >  stderr:     }
 >  stderr:     function procinclude(spec) {
 >  stderr:             if( rootdir!="" && spec!~/^\// ) {
 >  stderr:                     spec=rootdir"/"spec;
 >  stderr:             }
 >  stderr:             if( isdir(spec) ) {
 >  stderr:                     allfiles(spec); # read all files in a directory (and subdirs)
 >  stderr:             } else {
 >  stderr:                     listfiles(spec); # there could be jokers
 >  stderr:             }
 >  stderr:     }
 >  stderr:     function isdir(s) {
 >  stderr:             return !system("test -d \""s"\"");
 >  stderr:     }
 >  stderr:     { procline(); }
 >  stderr:      /etc/apache2/apache2.conf
 >  stderr: + sed s/#.*//;s/[[:blank:]]*$//;s/^[[:blank:]]*//
 >  stderr: + grep -v ^$
 >  stderr: + eval PidFile=${APACHE_PID_FILE}
 >  stderr: + PidFile=/var/run/apache2/apache2.pid
 >  stderr: + CheckPort
 >  stderr: + ocf_is_decimal
 >  stderr: + false
 >  stderr: + CheckPort
 >  stderr: + ocfError performing operation: Operation not permitted
_is_decimal
 >  stderr: + false
 >  stderr: + CheckPort 80
 >  stderr: + ocf_is_decimal 80
 >  stderr: + true
 >  stderr: + [ 80 -gt 0 ]
 >  stderr: + PORT=80
 >  stderr: + break
 >  stderr: + echo
 >  stderr: + grep :
 >  stderr: + Listen=localhost:
 >  stderr: + [ Xhttp://localhost/server-status = X ]
 >  stderr: + test /var/run/apache2/apache2.pid
 >  stderr: + return 0
 >  stderr: + validate_env
 >  stderr: + check_required_binaries
 >  stderr: + local v
 >  stderr: + is_function apache_validate_all
 >  stderr: + command -v apache_validate_all
 >  stderr: + test zapache_validate_all = zapache_validate_all
 >  stderr: + local rc
 >  stderr: + LSB_STATUS_STOPPED=3
 >  stderr: + apache_validate_all
 >  stderr: + [ -z /usr/sbin/apache2 ]
 >  stderr: + [ ! -x /usr/sbin/apache2 ]
 >  stderr: + [ ! -f /etc/apache2/apache2.conf ]
 >  stderr: + [ -n  ]
 >  stderr: + [ -n  ]
 >  stderr: + dirname /var/run/apache2/apache2.pid
 >  stderr: + local a
 >  stderr: + local b
 >  stderr: + [ 1 = 1 ]
 >  stderr: + a=/var/run/apache2/apache2.pid
 >  stderr: + [ 1 ]
 >  stderr: + b=/var/run/apache2/apache2.pid
 >  stderr: + [ /var/run/apache2/apache2.pid = /var/run/apache2/apache2.pid ]
 >  stderr: + break
 >  stderr: + b=/var/run/apache2
 >  stderr: + [ -z /var/run/apache2 -o /var/run/apache2/apache2.pid = /var/run/apache2 ]
 >  stderr: + echo /var/run/apache2
 >  stderr: + return 0
 >  stderr: + ocf_mkstatedir root 755 /var/run/apache2
 >  stderr: + local owner
 >  stderr: + local perms
 >  stderr: + local path
 >  stderr: + owner=root
 >  stderr: + perms=755
 >  stderr: + path=/var/run/apache2
 >  stderr: + test -d /var/run/apache2
 >  stderr: + return 0
 >  stderr: + return 0
 >  stderr: + rc=0
 >  stderr: + [ 0 -ne 0 ]
 >  stderr: + ocf_is_probe
 >  stderr: + [ monitor = monitor -a 0 = 0 ]
 >  stderr: + run_probe
 >  stderr: + is_function apache_probe
 >  stderr: + command -v apache_probe
 >  stderr: + test z = zapache_probe
 >  stderr: + shift 1
 >  stderr: + apache_monitor
 >  stderr: + silent_status
 >  stderr: + local pid
 >  stderr: + get_pid
 >  stderr: + [ -f /var/run/apache2/apache2.pid ]
 >  stderr: + cat /var/run/apache2/apache2.pid
 >  stderr: + pid=17552
 >  stderr: + [ -n 17552 ]
 >  stderr: + ProcessRunning 17552
 >  stderr: + local pid=17552
 >  stderr: + [ -d /proc -a -d /proc/1 ]
 >  stderr: + [ -d /proc/17552 ]
 >  stderr: + [ 0 -ne 0 ]
 >  stderr: + findhttpclient
 >  stderr: + [ x != x ]
 >  stderr: + which wget
 >  stderr: + echo wget
 >  stderr: + ourhttpclient=wget
 >  stderr: + [ -z wget ]
 >  stderr: + ocf_check_level 10
 >  stderr: + local lvl prev
 >  stderr: + lvl=0
 >  stderr: + prev=0
 >  stderr: + ocf_is_decimal 0
 >  stderr: + true
 >  stderr: + [ 10 -eq 0 ]
 >  stderr: + [ 10 -gt 0 ]
 >  stderr: + lvl=0
 >  stderr: + break
 >  stderr: + echo 0
 >  stderr: + apache_monitor_basic
 >  stderr: + wget_func http://localhost/server-status
 >  stderr: + auth=
 >  stderr: + cl_opts=-O- -q -L --no-proxy --bind-address=127.0.0.1
 >  stderr: + [ x !=+  x ]
 >  stderr: grep+ wget -Ei -O- </ *html *> -q
 >  stderr:  -L --no-proxy --bind-address=127.0.0.1 http://localhost/server-status
 >  stderr: + attempt_index_monitor_request
 >  stderr: + local indexpage=
 >  stderr: + [ -n  ]
 >  stderr: + [ -n  ]
 >  stderr: + [ -n  ]
 >  stderr: + [ -n http://localhost/server-status ]
 >  stderr: + return 1
 >  stderr: + [ 1 -eq 0 ]
 >  stderr: + ocf_is_probe
 >  stderr: + [ monitor = monitor -a 0 = 0 ]
 >  stderr: + return 1

pcs config

 Resource: MasterVip (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=10.226.***.*** nic=lo cidr_netmask=32 iflabel=pgrepvip
  Meta Attrs: target-role=Started
  Operations: start interval=0s timeout=20s (MasterVip-start-interval-0s)
              stop interval=0s timeout=20s (MasterVip-stop-interval-0s)
              monitor interval=90s (MasterVip-monitor-interval-90s)

 Resource: Apache (class=ocf provider=heartbeat type=apache)
  Attributes: configfile=/etc/apache2/apache2.conf statusurl=http://localhost/server-status
  Operations: start interval=0s timeout=40s (Apache-start-interval-0s)
              stop interval=0s timeout=60s (Apache-stop-interval-0s)
              monitor interval=1min (Apache-monitor-interval-1min)

I don't know how to fix this. if anyone knows please help me.

1

It seems, this resource-agent uses wget (or curl) for statusurl verification. Both commands fail if a self-signed certificate is used.

I ran into the same problem after using a self-signed certificate for my tomcat https connector. Only solution I found to date was to either add the parameter --no-check-certificate to the wget call in the resource-agent file (ocf/resource.d/heartbeat/tomcat):

isrunning_tomcat()
{
    $WGET --no-check-certificate --tries=20 -O /dev/null $RESOURCE_STATUSURL >/dev/null 2>&1
}

or to add it directly to the statusurl of the pcs resource:

statusurl="--no-check-certificate https://example-host:8443/somewebapp"

In the apache resource-agent file (ocf/resource.d/heartbeat/apache) you can specify which http client to use for verification:

<parameter name="client">
    <longdesc lang="en">
        Client to use to query to Apache. If not specified, the RA will
        try to find one on the system. Currently, wget and curl are
        supported. For example, you can set this parameter to "curl" if
        you prefer that to wget.
    </longdesc>
    <shortdesc lang="en">http client</shortdesc>
    <content type="string" default="wget"/>
</parameter>

Maybe you can specify wget with parameter --no-check-certificate or curl with parameter -k for verification in the resource-agent file.

Or do like I did and kind of inject it to the statusurl.

6
  • I have one more question. should i add server status section on virtual host :80 or virtual host :443? – Karippery Sep 22 '20 at 12:30
  • i tried both. it doesn't work – Karippery Sep 22 '20 at 12:53
  • I assume the wget command fails wether you use the --no-check-certificate param or not? What does the command return if you test it on your status url? (wget https://<server-ip>/server-status) – kobi Sep 22 '20 at 13:48
  • it return WARNING: The certificate of '10.226.***.***' is not trusted. WARNING: The certificate of '10.226.***.***' hasn't got a known issuer for more details github.com/ClusterLabs/resource-agents/issues/1556 – Karippery Sep 22 '20 at 15:11
  • As long as you use the --no-check-certificate param, the cert warnings can be ignored. Base problem is following: HTTP request sent, awaiting response... 403 Forbidden 2020-09-22 15:29:53 ERROR 403: Forbidden. So the access to your server-status url has been denied. Can you check if it works when you dont use the access control? – kobi Sep 23 '20 at 5:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.