1

UPDATE: I now know my issue was database corruption, but discerning it was somewhat tricky--apparmor appeared to be the cause for longer than it should've.

I didn't note when first posting that even after putting mysql in complain mode and sending apparmor both stop and teardown commands my syslog still showed the apparmor message...feeding my irrational fear of the protection layer--still not sure how this happened. I finally got mysql separated from apparmor, but it still couldn't lock its own files. Ergo database corruption--dang. My backups worked fine on a new server.

INITIAL POST:

Mysql server is being blocked by (I think) apparmor, but I'm at wits' end to determine why/how. I'm not overly familiar with apparmor.

I know I shouldn't uninstall apparmor--for at least two reasons--but I've used enough profanity (and given this issue too much time) to not consider it. Hopefully I'm merely missing something simple and will learn here.

The failure began today and follows no system changes. MySQL's error log laments permissions

Can't open and lock privilege tables: Table 'servers' is read only

I've been unable to find anyone with this issue who isn't currently moving their default database store. I moved mine as well--two years ago. The apparmor config is unchanged since 2014/04/21:

/files/bak/tmp/ rw,
/files/bak/tmp/* rwk,
/files/bak/mysql/ rw,
/files/bak/mysql/** rwk,

I've verified filesystem permissions:

# find mysql/ -type d -exec chmod 700 {} \;
# find mysql/ -type f -exec chmod 660 {} \;
# chown -R mysql: mysql 

I reloaded apparmor, installed apparmor-utils, pushed mysql to complain

# aa-complain mysql
# apparmor_status
apparmor module is loaded.
5 profiles are loaded.
4 profiles are in enforce mode.
   /sbin/dhclient
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/connman/scripts/dhclient-script
   /usr/sbin/tcpdump
1 profiles are in complain mode.
   /usr/sbin/mysqld
1 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
1 processes are unconfined but have a profile defined.
   /sbin/dhclient (495)

...but viewing syslog still suggests apparmor is blocking mysql after service mysql start:

apparmor="STATUS" operation="profile_replace" name="/usr/sbin/mysqld" pid=13899 comm="apparmor_parser"

Before I found the apparmor issue I tried restoring DBs from backups, which also failed with write permissions:

Can't create/write to file '/files/bak/mysql/dbCooper/wp_cb_contact_form.MYI'

I verified the filesystem is 'rw' (even though the above find -exec would have failed anyway):

mount
/dev/xvdf on /files/bak type ext4 (rw,noatime)

I've even tried stopping apparmor, but the syslog still shows can't open and lock privilege tables after this:

# service apparmor stop
  [...redacted teardown msg...]
# /etc/init.d/apparmor teardown
  * Unloading AppArmor profiles
# service apparmor status
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

Is it possible for mysql to lock database files and fail to unlock them when the daemon crashes? If so how would I clear the lock?

I'm currently running my DB with

mysqld --skip-grant-tables

...so I know the executable can run, and the databases are at least somewhat valid (the sites all appear normal). Am I missing something?

thanks for reading.

  • The syslog line you include does not indicate anything going wrong with AppArmor: it shows the AA profile (the set of rules it applies to /usr/sbin/mysqld) being replaced - which is almost certainly perfectly normal. AppArmor will log into the system log whenever it refuses an operation. – Mark Smith May 7 '16 at 21:08
  • @MarkSmith Yeah thanks. I saw replace and couldn't determine what it meant--with AA in the line I could only assume it was related. Getting AA to completely release mysql was more cumbersome than I expected, but once I'd finally gotten that line completely out of syslog and still had write perms I realized something more sinister was afoot. Thanks again – zedmelon May 10 '16 at 14:48
0

Database corruption.

Occam's Razor prevails. I moved backups to a new server and updated the db location/apparmor config. I cringed as I restarted everything. I'd spent hours convincing myself AppArmor was a cavernously complex and difficult beast, but my reticence was completely without cause--it worked perfectly on the first try.

Amazing how it just works when no files are corrupted--should be an Apple ad.

Now if I could just recover the WordPress site my web developer updated extensively while I was running under --skip-grant-tables but before I realized what the cause was. :-/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.