1

I thought I knew something about prelink, but I ran into weirdest issue yesterday. I know I shouldn't use prelink, but I've noticed it is enabled on one of CentOS 6 systems under my administration after it caused a massive change of all binaries, triggering intrusion warnings.

So I restored the system to root cause moment and it goes like this:

prelink -mR -av -q 2> /dev/null | grep Prelink    # clean
prelink -mR -av    2> /dev/null | grep Prelink    # clean
yum install zabbix-agent   # only installs one rpm, no dependencies

[root@www ~]# ls -lt  /lib64/libm-2.12.so /lib64/libc-2.12.so /lib64/ld-2.12.so
-rwxr-xr-x 1 root root 1926520 Feb 16 19:38 /lib64/libc-2.12.so
-rwxr-xr-x 1 root root  599392 Feb 16 19:38 /lib64/libm-2.12.so
-rwxr-xr-x 1 root root  157072 Feb 16 19:37 /lib64/ld-2.12.so
[root@www ~]# ls -lct /lib64/libm-2.12.so /lib64/libc-2.12.so /lib64/ld-2.12.so
-rwxr-xr-x 1 root root  599392 Apr 21 17:51 /lib64/libm-2.12.so
-rwxr-xr-x 1 root root 1926520 Apr 21 17:51 /lib64/libc-2.12.so
-rwxr-xr-x 1 root root  157072 Apr 21 17:51 /lib64/ld-2.12.so
[root@www ~]# md5sum  /lib64/libm-2.12.so /lib64/libc-2.12.so /lib64/ld-2.12.so   # verified - unchanged
348544291616b515c962027644afe879  /lib64/libm-2.12.so
9094a2fcef90994f490554f5514216aa  /lib64/libc-2.12.so
10f3aead091e8bdc85b86a00f6fe2104  /lib64/ld-2.12.so
[root@www ~]# prelink --dry-run -mR -a -v 2> /dev/null | grep Would
Would prelink /usr/lib64/libltdl.so.7.2.1
Would prelink /usr/lib64/libMagickCore.so.5.0.0
Would prelink /usr/lib64/libMagickWand.so.5.0.0
Would prelink /usr/lib64/libcurl.so.4.1.1
Would prelink /usr/lib64/libodbcinst.so.2.0.0
Would prelink /usr/lib64/libodbc.so.2.0.0
Would prelink /usr/sbin/zabbix_agentd

[root@www ~]# rpm -ql zabbix-agent
/etc/init.d/zabbix-agent
/etc/logrotate.d/zabbix-agent
/etc/zabbix/zabbix_agentd.conf
/etc/zabbix/zabbix_agentd.d
/etc/zabbix/zabbix_agentd.d/userparameter_mysql.conf
/usr/sbin/zabbix_agentd
/usr/share/doc/zabbix-agent-3.0.2
/usr/share/doc/zabbix-agent-3.0.2/AUTHORS
/usr/share/doc/zabbix-agent-3.0.2/COPYING
/usr/share/doc/zabbix-agent-3.0.2/ChangeLog
/usr/share/doc/zabbix-agent-3.0.2/NEWS
/usr/share/doc/zabbix-agent-3.0.2/README
/usr/share/man/man8/zabbix_agentd.8.gz
/var/log/zabbix
/var/run/zabbix

Nothing special so far, eh? Now I check what a normal daily prelink would do, and it turns out it wants to modify libm which is weird since full prelink wouldn't modify this often used library. After it does, the next daily prelink naturally modifies hundreds of unrelated binaries in the system.

[root@www ~]# prelink --dry-run -mR -a -q -v 2> /dev/null | grep Would
Would prelink /usr/lib64/libltdl.so.7.2.1
Would prelink /usr/lib64/libodbc.so.2.0.0
Would prelink /lib64/libm-2.12.so
Would prelink /usr/sbin/zabbix_agentd

How is this related to a fact that hex address of libm relocation is very close to that of libltdl?

[root@www ~]# ldd /usr/sbin/zabbix_agentd
        linux-vdso.so.1 =>  (0x00007ffe481da000)
        libssl.so.10 => /usr/lib64/libssl.so.10 (0x0000003459200000)
        libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x0000003458600000)
        libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x0000003456a00000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x0000003459a00000)
        libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x0000003453a00000)
        libodbc.so.2 => /usr/lib64/libodbc.so.2 (0x0000003927000000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f1e1ae0a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003452a00000)
        librt.so.1 => /lib64/librt.so.1 (0x0000003453200000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003454600000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003452600000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x0000003458200000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x0000003457200000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003456600000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x0000003457600000)
        libz.so.1 => /lib64/libz.so.1 (0x0000003453600000)
        libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x0000003458e00000)
        libssl3.so => /usr/lib64/libssl3.so (0x0000003454200000)
        libsmime3.so => /usr/lib64/libsmime3.so (0x0000003456e00000)
        libnss3.so => /usr/lib64/libnss3.so (0x0000003455a00000)
        libnssutil3.so => /usr/lib64/libnssutil3.so (0x0000003455e00000)
        libplds4.so => /lib64/libplds4.so (0x0000003455200000)
        libplc4.so => /lib64/libplc4.so (0x0000003456200000)
        libnspr4.so => /lib64/libnspr4.so (0x0000003455600000)
        libidn.so.11 => /lib64/libidn.so.11 (0x000000345a200000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003452e00000)
        libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x0000003458a00000)
        libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00007f1e1abfa000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003452200000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x0000003457e00000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x0000003457a00000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x0000003454e00000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x0000003453e00000)
        libfreebl3.so => /lib64/libfreebl3.so (0x0000003454a00000)

Update simple test case shows that prelink command acts differently if given a couple of files to operate on, than when only trying to prelink a single file. Why is that?

[root@www ~]# prelink --dry-run -v /usr/sbin/zabbix_agentd 2> /dev/null | grep Would
    Would prelink /lib64/libm-2.12.so
    Would prelink /usr/lib64/libltdl.so.7.2.1
    Would prelink /usr/lib64/libodbc.so.2.0.0
    Would prelink /usr/sbin/zabbix_agentd
[root@www ~]# prelink --dry-run -v /usr/sbin/zabbix_agentd /usr/sbin/era_check 2> /dev/null | grep Would
    Would prelink /usr/lib64/libcurl.so.4.1.1
    Would prelink /usr/lib64/libltdl.so.7.2.1
    Would prelink /usr/lib64/libodbc.so.2.0.0
    Would prelink /usr/sbin/zabbix_agentd

[root@www ~]# rpm -qf /usr/sbin/era_check      # random unrelated binary
device-mapper-persistent-data-0.3.2-1.el6.x86_64

[root@www ~]# ldd /usr/sbin/era_check
    linux-vdso.so.1 =>  (0x00007ffcf6991000)
    libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000030d1e00000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00000030d3e00000)
    libm.so.6 => /lib64/libm.so.6 (0x00000030cfe00000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000030d1200000)
    libc.so.6 => /lib64/libc.so.6 (0x00000030cea00000)
    /lib64/ld-linux-x86-64.so.2 (0x00000030ce600000)
4
  • Fascinating. Can you guarantee with rpm -Va and with rpm -qa --queryformat '%{installtime} (%{installtime:date}) %{name}\n' | sort -n that nothing else on your system is fkng with these libraries? (I'm assuming you don't have a rogue process doing it.)
    – Otheus
    Apr 22, 2016 at 11:56
  • 1
    I've just verified these. The only semi-related output seems to be .......T. /lib/modules/2.6.32-504.16.2.el6.x86_64/modules.softdep Nothing interesting sadly. No rogue process - I've replicated this on a test VM and this is repeatable each time after installing zabbix-agent.
    – kubanczyk
    Apr 22, 2016 at 13:38
  • I would test, but I'm not sure if I yet can install zabbix 3. Still on 2.2. :)
    – Otheus
    Apr 22, 2016 at 14:17
  • 10 years ago, I recall reading about a memory-pointer randomizer for shared libraries, which is essentially there to foil buffer-overflow (and related) attacks from calling library calls via static addresses. The rpm verify function uses a special call to exclude the varying memory pointers from its checksum. Otherwise, shared libraries are changing "all the time", but I don't remember exactly how often.
    – Otheus
    Apr 22, 2016 at 14:21

0

You must log in to answer this question.

Browse other questions tagged .