Centos7 http will not start normally. httpd works, systemctl start httpd does not

Question

I can start apache directly via httpd, but I cannot start it via systemctl start httpd. I would prefer the daemon method so I can enable it to automatically start.

Anyone run into this issue? This is on a fresh CentOS7 VM.

systemctl start http

Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

systemctl status httpd.service

    ● httpd.service - The Apache HTTP Server
       Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
       Active: failed (Result: exit-code) since Tue 2018-03-20 17:20:54 EDT; 37s ago
         Docs: man:httpd(8)
               man:apachectl(8)
      Process: 7025 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
      Process: 7024 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
     Main PID: 7024 (code=exited, status=1/FAILURE)

    Mar 20 17:20:54 test.local.com systemd[1]: Starting The Apache HTTP Server...
    Mar 20 17:20:54 test.local.com systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
    Mar 20 17:20:54 test.local.com kill[7025]: kill: cannot find process ""
    Mar 20 17:20:54 test.local.com systemd[1]: httpd.service: control process exited, code=exited status=1
    Mar 20 17:20:54 test.local.com systemd[1]: Failed to start The Apache HTTP Server.
    Mar 20 17:20:54 test.local.com systemd[1]: Unit httpd.service entered failed state.
    Mar 20 17:20:54 test.local.com systemd[1]: httpd.service failed.

journalctl -xe

Pastebin

/etc/httpd/logs/error_log

Pastebin

The only change made to the default config:

/etc/httpd/conf/httpd.conf
IncludeOptional sites-enabled/*.conf

/etc/httpd/sites-enabled/local.com.conf
<VirtualHost *:80>
    ServerName test.local.com
    ServerAlias local.com
    Redirect / https://local.com
</VirtualHost>

<VirtualHost _default_:443>
    ServerName test.local.com
    ServerAlias local.com
    ServerAdmin testemail@local.com

    DocumentRoot /var/www/local.com/public_html

    ErrorLog /var/www/local.com/error.log
    CustomLog /var/www/local.com/access.log common

    SSLEngine On
    SSLCertificateFile /etc/ssl/certs/www/local.com.crt
    SSLCertificateKeyFile /etc/ssl/certs/www/local.com.key
</VirtualHost>

Only ports opened:

firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-dmc --reload

Here is the entire exact process I did from a fresh CentOS7 installation:

Fresh CentOS 7 installation (VM)

yum upgrade -y

yum search http
yum install -y httpd httpd-devel mod_ssl openssl

systemctl start httpd

firewall-cmd --zone=public --add-port=80/tcp --permanent
firewall-cmd --zone=public --add-port=443/tcp --permanent
firewall-cmd --reload

    Browse to 192.168.1.241
        Apache is live!

yum search mariadb
yum install -y mariadb-server

systemctl start mariadb
mysql_secure_installation

    mysql -uroot -p
        Login to mysql server works!

yum search php

yum install -y php php-cli php-dba php-devel php-fpm php-mysql php-process php-pspell php-xml

systemctl restart httpd

    Browse to 192.168.1.241/info.php
        PHP is live!

mkdir /etc/httpd/sites-enabled
echo "IncludeOptional sites-enabled/*.conf" >> /etc/httpd/conf/httpd.conf

/etc/httpd/sites-enabled/local.com.conf
    <VirtualHost *:80>
        ServerName test.local.com
        ServerAlias local.com
        Redirect permenent / https://local.com
    </VirtualHost>

    <VirtualHost _default_:443>
        ServerName test.local.com
        ServerAlias local.com
        ServerAdmin testemail@local.com

        DocumentRoot /var/www/local.com/public_html

        ErrorLog /var/www/local.com/error.log
        CustomLog /var/www/local.com/access.log combined

        SSLEngine On
        SSLCertificateFile /etc/ssl/certs/www/local.com.crt
        SSLCertificateKeyFile /etc/ssl/certs/www/local.com.key
    </VirtualHost>

mkdir -p /var/www/local.com/public_html

chown -R apache:apache /var/www/local.com/public_html
chmod -R 755 /var/www

cd /etc/ssl/certs/www
openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout local.com.key -out local.com.crt

    Browse to 192.168.1.241
        Unsecure service (self signed ssl) accept
        Site is live!
        I was redirected to https://local.com

NOTE: I added the following to my desktop's (separate PC) /etc/hosts
    192.168.1.241 test.local.com local.com

    This acts as a DNS record for my site

yum install -y epel-release
yum install -y phpmyadmin

edit /etc/httpd/conf.d/phpMyAdmin.conf
    Add under any line with Require ip 127.0.0.1 with
    Require ip 192.168.1.5

    Add under any line with Allow from 127.0.0.1 with
    Allow from 192.168.1.5

systemctl restart httpd # FAILS
kill pid for httpd
httpd # start httpd directly
    Access https://local.com/phpMyAdmin
    Now have access to phpMyAdmin

    Login with root, 12345
    And have mariadb access!

yum install -y awstats

edit /etc/httpd/conf.d/awstats.conf
     Change Require ip and Allow ip same as phpMyAdmin

cp /etc/awstats/awstats.localhost.localdomain.conf /etc/awstats/awstats.local.com.conf

edit /etc/awstats/awstats.local.com.conf
     LogFile="/var/log/httpd/access.log"
     SiteDomain="www.local.com"
     HostAliases="local.com 127.0.0.1"

echo "*/30 * * * * root /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -config=www.local.com -update" >> /etc/crontab

kill httpd pid
httpd

    Browse to https://local.com/awstats/awstats.pl?config=local.com
        Awstats is live!

Answer 1

You're running into trouble with SELinux.

CentOS 7 ships rules that will prevent httpd from writing to files under /var/www, for security reasons.

You are configuring your log files for your VirtualHost to go somewhere under that directory:

    ErrorLog /var/www/local.com/error.log
    CustomLog /var/www/local.com/access.log combined

So when httpd (started by systemd) tries to write to these log files, SELinux will prevent that, which ends up making httpd exit with an error exit code.

You can confirm that using the ausearch command, which checks for entries in the Audit log (which is stored under /var/log/audit/audit.log):

$ sudo ausearch -m avc
type=AVC msg=audit(1234567890.123:234): avc:  denied  { write } for  pid=12345 comm="httpd" name="local.com" dev="sda1" ino=12345678 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:httpd_sys_content_t:s0 tclass=dir

In this message, you'll see that the target of the write is tagged with httpd_sys_content_t. If you use ls -Z on the log files, you'll see they're tagged that way:

$ ls -Z /var/www/local.com/
-rw-r--r--. root   root   unconfined_u:object_r:httpd_sys_content_t:s0 access.log                                                                                         
-rw-r--r--. root   root   unconfined_u:object_r:httpd_sys_content_t:s0 error.log                                                                                          
drwxr-xr-x. apache apache unconfined_u:object_r:httpd_sys_content_t:s0 public_html

The reason why this only affects httpd as launched by systemd and not when you run httpd directly is that your SSH session runs in the "unconfined" domain, so running httpd there doesn't trigger any SELinux transitions... When launched through systemd, it will apply the correct SELinux permissions when starting the daemon.

You can temporarily work around that by using the chcon command to change the SELinux "type" of those files:

$ sudo chcon -t httpd_log_t /var/www/local.com/*.log
$ ls -Z /var/www/local.com/
-rw-r--r--. root   root   unconfined_u:object_r:httpd_log_t:s0 access.log
-rw-r--r--. root   root   unconfined_u:object_r:httpd_log_t:s0 error.log
drwxr-xr-x. apache apache unconfined_u:object_r:httpd_sys_content_t:s0 public_html

At that point, starting httpd through systemctl will work fine...

But that's not a great solution, since the SELinux type will be lost if these files are recreated (for instance, during log rotation) or if your filesystem gets relabeled...

There are ways to make that type more permanent (for example, the semanage fcontext command), but what this SELinux policy is trying to implement here is to prevent mixing up web content with logs, in order to prevent accidentally serving log files, or overwriting web content.

The right answer is to create log files under /var/log/httpd, or subdirectories of that directory. If you do so, the SELinux type will be correct from the start, will be kept correct over any operations, including SELinux relabels, and everything should work as expected.

So, if you can have your logs under /var/log/httpd instead, that should solve this problem!