5

I'm getting audit messages from SELinux saying that it denied Apache doing execmem:

type=AVC msg=audit(05/06/16 19:51:43.058:181060) : avc:  denied  { execmem } for  pid=123456 comm=httpd scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=process 

The PID is one of the Apache PIDs and it keeps cycling between the processes.

From what I've read, execmem for Apache is generally not normal and a Bad Idea™, and that makes sense.

I've tried tracking an origin by tallying timestamps with Apache logs, but if affects various requests across various sites (PHP-based with and without MySQL, Python/mod_wsgi-based, plus internal Apache "OPTION" requests) and I can't find anything consistent.

Rather than try to explain my setup to get people to debug it, what I want to know is how do I work out where the execmem call is coming from so that I can work out whether it is important?

(Note: I know there is an SELinux boolean to set to allow it, but I don't want to do that without first understanding why it is trying it. There is no point in having SELinux if you're going to turn it into a sieve, much like there's no point having a firewall and then opening every port that ever complains without checking whether it is important or not).

  • I've done a bit of investigating and it doesn't seem to be Python/mod_wsgi. I commented out all mod_wsgi lines, reloaded Apache, the WSGI processes were stopped but the errors continued. – IBBoard Jul 1 '16 at 18:45
  • I've finally found the problem through testing on a VM (means I control all requests) and running smaller and smaller test cases. Eventually turned out it was PHP7's PCRE JITing (secure.php.net/manual/en/pcre.configuration.php). Doesn't answer the wider and more general case of "how do I work out where Apache is trying to do execmem from when SELinux complains about it", though! – IBBoard Aug 19 '16 at 19:37
  • There's a Fedora 25 bug report: bugzilla.redhat.com/show_bug.cgi?id=1398474, a RHEL report, and there's an upstream pcre report. Thanks to the folks who had this figured out ahead of time, it helped me a lot when I did an F25 upgrade and saw these denials happening. – Adam Williamson Nov 28 '16 at 21:17
6

I recently came across this problem with SELinux on Amazon Linux with PHP7. I used a combination of Russell Coker's excellent LD_PRELOAD trick (to intercept mmap() calls and trigger an assertion failure) and gdb (to view the call stack as soon as the assertion failure is triggered) to check which function wants to execmem.

I also came to the conclusion that PHP7 PCRE JIT was the culprit. Putting pcre.jit=0 in php.ini fixed it for me.

Detailed steps

  1. log into your machine as root
  2. download the mmap.c source code from https://etbe.coker.com.au/2010/07/12/write-execute-mmap-ld-preload/ into /root/mmap.c
  3. build the code with gcc -shared -g -fPIC mmap.c -o mmap.so
  4. now run Apache through gdb, intercepting mmap() calls: LD_PRELOAD=/root/mmap.so gdb /usr/sbin/httpd
  5. you're thrown into gdb. Because Apache forks child processes, it is important to tell gdb to jump into them by typing set follow-fork-mode child after the (gdb) prompt
  6. now kick off Apache by typing run after the (gdb) prompt
  7. wait patiently until some HTTP request triggers the code that in turn triggers the assertion in mmap, you're then back in gdb.
Program received signal SIGABRT, Aborted.

[Switching to Thread 0x7ffff7fe9840 (LWP 28370)]

0x00007ffff638d5f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

56        return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
  1. Type bt (backtrace) to see the call stack:
(gdb) bt

#0  0x00007ffff638d5f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56

#1  0x00007ffff638ece8 in __GI_abort () at abort.c:90

#2  0x00007ffff6386566 in __assert_fail_base (fmt=0x7ffff64d6ca8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7ffff7bda990 "!(prot & 0x4) || !(prot & 0x2)",

    file=file@entry=0x7ffff7bda985 "mmap.c", line=line@entry=27, function=function@entry=0x7ffff7bda9af  "mmap") at assert.c:92

#3  0x00007ffff6386612 in __GI___assert_fail (assertion=0x7ffff7bda990 "!(prot & 0x4) || !(prot & 0x2)", file=0x7ffff7bda985 "mmap.c", line=27, function=0x7ffff7bda9af  "mmap")

    at assert.c:101

#4  0x00007ffff7bda93e in mmap (addr=0x0, length=65536, prot=7, flags=34, fd=-1, offset=0) at mmap.c:27

#5  0x00007ffff79a6b86 in alloc_chunk (size=65536) at sljit/sljitExecAllocator.c:101

#6  sljit_malloc_exec (size=4440) at sljit/sljitExecAllocator.c:204

#7  sljit_generate_code (compiler=compiler@entry=0x555555b14ad0) at sljit/sljitNativeX86_common.c:296

#8  0x00007ffff79bf74e in _pcre_jit_compile (re=re@entry=0x555555b14650, extra=extra@entry=0x555555b14750) at pcre_jit_compile.c:6434

#9  0x00007ffff79c1fc3 in pcre_study (external_re=external_re@entry=0x555555b14650, options=1, errorptr=errorptr@entry=0x7fffffffa3c8) at pcre_study.c:1354

#10 0x00007fffed2edcbc in pcre_get_compiled_regex_cache (regex=0x7fffd8a04000) at /usr/src/debug/php-7.0.9/ext/pcre/php_pcre.c:487

#11 0x00007fffed2ef21f in php_do_pcre_match (execute_data=0x7fffec419ab0, return_value=0x7fffec419870, global=1) at /usr/src/debug/php-7.0.9/ext/pcre/php_pcre.c:655


  1. the culprit is easily identified. Stack entry 11 gives you a hint to check PCRE and entry 5 is the actual call that goes awry.
  • Just trying this out. It looks promising, but I can't get it working yet. Step 7 never seems to happen and all I get logged are "AH00052: child pid xxxx aborted (6)" (segfaults, apparently) and "mmap: Assertion… failed". No change to gdb when it happens, though. Static content servers cleanly, but pages that should debug just get "secure connection failed". I must have done something wrong somewhere. – IBBoard Aug 30 '16 at 12:58
  • Did you set gdb to follow into forked childs with the set follow-fork-mode child before you entered run? The fact that the mmap() call triggers an assertion failure tells us that the LD_PRELOAD trick worked. – Edward Aug 30 '16 at 13:03
  • Definitely ran that command inside gdb. Repeated several times and no success. However, suggestion in httpd.apache.org/dev/debugging.html to use "run -X" instead of the "follow-fork-mode" seemed to work for me. Stack trace was: #0: raise() … #4: custom mmap(), #5: sljit_generate_code(), #6: _pcre_jit_compile() … #9: php_do_pcre_match(). Thanks! – IBBoard Aug 30 '16 at 13:07
  • You're welcome, and thanks for the -X tip :-) – Edward Aug 30 '16 at 13:19
  • Could it be a runner or other config issue that means "follow-fork-mode child" didn't seem to work as expected for me? I'm using prefork with Webtatic's PHP 7.0. – IBBoard Aug 31 '16 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.