I've previously written about AppArmor and MySQL, and how to change MySQL's default file locations on systems with AppArmor enabled. Ubuntu and SUSE ship with AppArmor enabled, but some other distributions such as Oracle Linux don't, along with related distrubutions such as Red Hat, CentOS and Fedora. Rather, these other distributions use another mandatory access control system called SELinux.
Here's some technical detail that might come in handy later.
SELinux uses concepts such as types and domains. Types belong to resources such as files and ports; these are the "objects" in SELinux. Domains contain the "subjects" (processes) and object types that are associated with each other in some way, for example because they are all related to MySQL.
Each executable for a service that SELinux understands, including MySQL, has its own type. When the process runs, it is placed within a domain based on its type; members of that domain (normally processes) that wish to access objects (such as files and ports) must be authorised to do so by virtue of the object having a configured type that is approved for that domain.
This configuration is contained within policy files that usually ship with each distribution, but can be created using the appropriate tools.
SELinux (Security-Enhanced, if you're interested) "is a Linux feature that provides the mechanism for supporting access control security policies" according to Wikipedia. More simply, it stops things—like programs—from accessing things—like files and network ports—they shouldn't access. By "shouldn't access" I really mean "haven't been configured to access". For example, MySQL is allowed to write to its data directory in /var/lib/mysql, and read from /etc/my.cnf. It can open a socket on port 3306, but SELinux prevents it from writing to files in /home/jeremy or /sbin or anywhere else that isn't already configured as a MySQL location.
In short, if you try changing MySQL's port to a non-standard one, or try taking backups or configuring data files or log files to anywhere but the usual locations, you'll get some odd access-denied type errors in the MySQL error log. In addition, you'll get messages in /var/log/audit/audit.log (if auditd is running, otherwise /var/log/messages or /var/log/syslog, depending how your system is configured).
What Error do I get?
From the MySQL error log:
130321 11:50:51 mysqld_safe Starting mysqld daemon with databases from /datadir ... 2013-03-21 11:50:52 2119 [Warning] Can't create test file /datadir/boxy.lower-test 2013-03-21 11:50:52 2119 [Warning] Can't create test file /datadir/boxy.lower-test ... 2013-03-21 11:50:52 2119 [ERROR] /usr/sbin/mysqld: Can't create/write to file '/datadir/boxy.pid' (Errcode: 13 - Permission denied) 2013-03-21 11:50:52 2119 [ERROR] Can't start server: can't create PID file: Permission denied 130321 11:50:52 mysqld_safe mysqld from pid file /datadir/boxy.pid ended
Now, I'm very sure the permissions on that folder are correct, so let's have a look in /var/log/audit/audit.log:
...
type=AVC msg=audit(1363866652.030:24): avc: denied { write } for pid=2119
comm="mysqld"name="datadir" dev=dm-0 ino=394
scontext=unconfined_u:system_r:mysqld_t:s0
tcontext=unconfined_u:object_r:default_t:s0 tclass=dir
...A similar error occurs if I try starting MySQL on port 3307, a non-default port:
In the MySQL error log:
2013-03-21 12:12:09 3436 [Note] Server hostname (bind-address): '*'; port: 3307
...
2013-03-21 12:12:09 3436 [ERROR] Can't start server: Bind on TCP/IP port:
Permission denied
2013-03-21 12:12:09 3436 [ERROR] Do you already have another mysqld server
running on port: 3307 ?
2013-03-21 12:12:09 3436 [ERROR] AbortingIn the audit log:
type=AVC msg=audit(1363867929.432:42): avc: denied { name_bind } for pid=3436
comm="mysqld"src=3307
scontext=unconfined_u:system_r:mysqld_t:s0
tcontext=system_u:object_r:port_t:s0 tclass=tcp_socketClearly something going on here. The Access Vector Cache (as seen in the "avc: denied" message) is where SELinux caches permissions for the kernel, so it's definitely SELinux doing the denying.
Just Stop It!
I'm going to start with the hammer and work my way down to the scalpel.
Here's the hammer:
[root@boxy ~]# setenforce 0 [root@boxy ~]# getenforce Permissive
The setenforce 0 command switches off SELinux enforcing until the next reboot, and getenforce shows you the current status. To stop SELinux from enforcing on any reboot, you'll need to change a configuration file:
[root@boxy ~]# cat /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded.SELINUX=enforcing # SELINUXTYPE= can take one of these two values: # targeted - Targeted processes are protected, # mls - Multi Level Security protection. SELINUXTYPE=targeted
Change that enforcing to permissive (or disabled) and you're good to go. The difference:
- enforcing blocks operations that SELinux does not allow
- permissive does not block the operations, but logs them (to /var/log/audit/audit.log)
- disabled switches off SELinux entirely, to the extent that you cannot use setenforce until you change it and reboot.
For example, on a machine with SELinux set to permissive, I can do the following:
[root@boxy ~]# setenforce 1 [root@boxy ~]# getenforce Enforcing
But if it's disabled, this happens:
[root@boxy ~]# setenforce 0 setenforce: SELinux is disabled [root@boxy ~]# setenforce 1 setenforce: SELinux is disabled
That's the hammer.
So, to return to the example that generated the error, I can use the hammer:
[root@boxy ~]# setenforce 0 [root@boxy ~]# service mysql start --datadir=/datadir Starting MySQL. SUCCESS! [root@boxy ~]# service mysql stop Shutting down MySQL.. SUCCESS!
If you're happy with that, you could then edit the configuration file to disable SELinux on next reboot, and thanks for reading. See you next time.
I'm intrigued. How do I configure it?
Obviously, there's a lot more to SELinux than disabling it, and a responsible admin (that's you, right?) wants to know how to use it rather than disable it. I'm not going to get into too much detail here.
We can, however, look at how you can assign SELinux types to objects such as ports and files, so that members of the mysqld_t domain (specifically the mysqld_safe process, launced when you run service mysql start) can access those objects.
So here's the scalpel. First, let's configure SELinux to enable MySQL's use of port 3307:
[root@boxy ~]# semanage port -a -t mysqld_port_t -p tcp 3307
The semanage utility changes various SELinux settings. In this case, it adds (-a) a type (-t mysqld_port_t) to the port mappings for port 3307 using TCP as its protocol (-p tcp). When MySQL (through the mysqld_safe process) tries to access that port, SELinux recognises that the port has a type that is approved for such access by the policy.
We can also allow MySQL to use the /datadir directory:
[root@boxy ~]# semanage fcontext -a -t mysqld_db_t "/datadir(/.*)?" [root@boxy ~]# restorecon -Rv /datadir restorecon reset /datadir context unconfined_u:object_r:default_t:s0->unconfined_u:object_r:mysqld_db_t:s0 restorecon reset /datadir/mysql.sock context system_u:object_r:mysqld_var_run_t:s0->system_u:object_r:mysqld_db_t:s0
In this example, semanage is adding the type mysqld_db_t to the file context map (fcontext) for anything in the /datadir directory and subdirectories ("/datadir(/.*)?", a regular expression). File mappings such as this are contained in the file /etc/selinux/targeted/contexts/files/file_contexts.local; that file must subsequently be read in order to set the appropriate type on the file itself. That's done by the restorecon utility, and at system reboot. If you want to change a file context immediately, but don't need it to survive a reboot, there's a chcon utility that performs that task.
The same principles and statements apply if you wish to use other ports or directories. There are similar types that apply to different kinds of files; I used mysqld_db_t above for database directories, but the standard SELinux policy for MySQL also include:
- mysqld_etc_t for configuration files such as /etc/my.cnf
- mysqld_log_t for logfiles such as those named /var/log/mysql*
- Types for the PID file, tmp files, service startup files in the /etc/init.d directory, and the various executables you're likely to use
Conclusion
This post is already long enough, so I won't get into the deeper topics in SELinux, such as the ability to compile your own policy files and configure new policies for services that SELinux doesn't yet know about. At this stage, you know how to add an SELinux type to an object such as a port or a file so that MySQL can access that object, even if it's not used by default. You also know how to disable SELinux in a couple of ways, but you're not going to do that now, are you? You've got a perfectly good scalpel, after all. Why use a hammer?