There’s been a lot of talk about Docker for running processes in isolated userspace (or the cloud for that matter) lately. Virtualization is a great way to compartmentalise applications and processes however the overhead of virtualization isn’t always worth it – in fact, without directly attached storage IO degradation can seriously impact performance. The solution? Perhaps Docker… With its easy to use CLI as well as the lightweight implementation of cgroups and kernel namespaces.
Without further ado, I present a step-bystep guide on how to build a MariaDB 5.5 Galera Cluster on Ubuntu 14.04. The same guide can probably be applied for MariaDB versions 10+ however I’ve stuck with 5.5 since the latest version of MariaDB Galera Cluster is still in beta.
So we start off with modifying the “ufw” firewall policy to accept forwarded packets and perform a “ufw” service restart for good measure:
[email protected]:~# vi /etc/default/ufw DEFAULT_FORWARD_POLICY="ACCEPT" [email protected]:~# service ufw restart ufw stop/waiting ufw start/running
I’m assuming you already have docker installed – this is available as a package within the Ubuntu repositories and also available in the Docker repositories (see https://docs.docker.com/installation/ubuntulinux/). You’ll also need to have LXC installed (“apt-get install lxc” should suffice) in order to attach to the Linux Containers / Docker Images.
The next step is pulling the Docker / Ubuntu repository in order to customize an image for our purposes
[email protected]:~# docker pull ubuntu Pulling repository ubuntu c4ff7513909d: Pulling dependent layers 3db9c44f4520: Download complete c5881f11ded9: Download complete c4ff7513909d: Download complete 463ff6be4238: Download complete 822a01ae9a15: Download complete 75204fdb260b: Download complete 511136ea3c5a: Download complete bac448df371d: Download complete dfaad36d8984: Download complete 5796a7edb16b: Download complete 1c9383292a8f: Download complete 6cfa4d1f33fb: Download complete f127542f0b61: Download complete af82eb377801: Download complete 93c381d2c255: Download complete 3af9d794ad07: Download complete a5208e800234: Download complete 9fccf650672f: Download complete fae16849ebe2: Download complete b7c6da90134e: Download complete 1186c90e2e28: Download complete 0f4aac48388f: Download complete 47dd6d11a49f: Download complete f6a1afb93adb: Download complete 209ea56fda6d: Download complete f33dbb8bc20e: Download complete 92ac38e49c3e: Download complete 9942dd43ff21: Download complete aa822e26d727: Download complete d92c3c92fa73: Download complete 31db3b10873e: Download complete 0ea0d582fd90: Download complete cc58e55aa5a5: Download complete
After the download is complete, we can check the Ubuntu images available for customization with the following command:
[email protected]:~# docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE ubuntu 14.04.1 c4ff7513909d 12 days ago 225.4 MB ubuntu trusty c4ff7513909d 12 days ago 225.4 MB ubuntu 14.04 c4ff7513909d 12 days ago 225.4 MB ubuntu latest c4ff7513909d 12 days ago 225.4 MB ubuntu utopic 75204fdb260b 12 days ago 230.1 MB ubuntu 14.10 75204fdb260b 12 days ago 230.1 MB ubuntu precise 822a01ae9a15 12 days ago 108.1 MB ubuntu 12.04 822a01ae9a15 12 days ago 108.1 MB ubuntu 12.04.5 822a01ae9a15 12 days ago 108.1 MB ubuntu 12.10 c5881f11ded9 9 weeks ago 172.2 MB ubuntu quantal c5881f11ded9 9 weeks ago 172.2 MB ubuntu 13.04 463ff6be4238 9 weeks ago 169.4 MB ubuntu raring 463ff6be4238 9 weeks ago 169.4 MB ubuntu 13.10 195eb90b5349 9 weeks ago 184.7 MB ubuntu saucy 195eb90b5349 9 weeks ago 184.7 MB ubuntu lucid 3db9c44f4520 4 months ago 183 MB ubuntu 10.04 3db9c44f4520 4 months ago 183 MB
Now that we’ve downloaded our images lets create a custom Dockerfile for our customized MariaDB / Galera Docker image, I’ve added a brief description for each line of the file:
[email protected]:~# vi Dockerfile # # MariaDB Galera 5.5.39/Ubuntu 14.04 64bit FROM ubuntu:14.04 MAINTAINER Pythian Nikolaos Vyzas <[email protected]> # add the universe repo RUN echo "deb https://archive.ubuntu.com/ubuntu trusty main universe" > /etc/apt/sources.list # update apt RUN apt-get -q -y update # install software-properties-common for key management RUN apt-get -q -y install software-properties-common # add the key for Mariadb Ubuntu repos RUN apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db # add the MariaDB repository for 5.5 RUN add-apt-repository 'deb https://ftp.cc.uoc.gr/mirrors/mariadb/repo/5.5/ubuntu trusty main' # update apt again RUN apt-get -q -y update # configure the default root password during installation RUN echo mariadb-galera-server-5.5 mysql-server/root_password password root | debconf-set-selections # confirm the password (as in the usual installation) RUN echo mariadb-galera-server-5.5 mysql-server/root_password_again password root | debconf-set-selections # install the necessary packages RUN LC_ALL=en_US.utf8 DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::='--force-confnew' -qqy install mariadb-galera-server galera mariadb-client # upload the locally created my.cnf (obviously this can go into the default MariaDB path ADD ./my.cnf /etc/mysql/my.cnf # startup the service - this will fail since the nodes haven't been configured on first boot RUN service mysql restart # open the ports required to connect to MySQL and for Galera SST / IST operations EXPOSE 3306 4444 4567 4568 We'll also need our base configuration for MariaDB, I've included the base configuration variable for Galera - obviously there are more however these are good enough for starting up the service:
[email protected]:~# vi my.cnf [mysqld] wsrep_provider=/usr/lib/galera/libgalera_smm.so wsrep_cluster_address=gcomm:// wsrep_sst_method=rsync wsrep_cluster_name=galera_cluster binlog_format=ROW default_storage_engine=InnoDB innodb_autoinc_lock_mode=2 innodb_locks_unsafe_for_binlog=1
So far so good, we have Docker installed and our Dockerfile as well as our “my.cnf” file ready to go. Now its time to build our Docker image, check that the image exists and startup 3x separate Docker images for each of our Galera nodes:
[email protected]:~# docker build -t ubuntu_trusty/mariadb-galera .[email protected]:~# docker images |grep mariadb-galera ubuntu_trusty/mariadb-galera latest afff3aaa9dfb About a minute ago 412.5 MBdocker run --name mariadb1 -i -t -d ubuntu_trusty/mariadb-galera /bin/bash docker run --name mariadb2 -i -t -d ubuntu_trusty/mariadb-galera /bin/bash docker run --name mariadb3 -i -t -d ubuntu_trusty/mariadb-galera /bin/bash
We’ve started up our Docker images, now lets verify that they are in fact up and retrieve the process information we need to connect. We’ll need two pieces of information, the IP-Address and the Docker image name which can be received using the combination the the “docker ps” and the “docker inspect” commands:
}][email protected]:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b51e74933ece ubuntu_trusty/mariadb-galera:latest /bin/bash About an hour ago Up About an hour 3306/tcp, 4444/tcp, 4567/tcp, 4568/tcp mariadb3 03109c7018c0 ubuntu_trusty/mariadb-galera:latest /bin/bash About an hour ago Up About an hour 3306/tcp, 4444/tcp, 4567/tcp, 4568/tcp mariadb2 1db2a9a520f8 ubuntu_trusty/mariadb-galera:latest /bin/bash About an hour ago Up About an hour 3306/tcp, 4444/tcp, 4567/tcp, 4568/tcp mariadb1[email protected]:~# docker ps |cut -d' ' -f1 |grep -v CONTAINER | xargs docker inspect |egrep '"ID"|IPAddress' "ID": "b51e74933ece2f3f457ec87c3a4e7b649149e9cff2a4705bef2a070f7adbafb0", "IPAddress": "172.17.0.3", "ID": "03109c7018c03ddd8448746437346f080a976a74c3fc3d15f0191799ba5aae74", "IPAddress": "172.17.0.4", "ID": "1db2a9a520f85d2cef6e5b387fa7912890ab69fc0918796c1fae9c1dd050078f", "IPAddress": "172.17.0.2",
Time to use lxc-attach to connect to our Docker images using the Docker image name, add the mounts to “/etc/mtab” to keep them MariaDB friendly and customize the “gcomm://” address as we would for a usual Galera configuration (the Docker image name is a generated when the instance fires up so make sure to use your own instance name in the following commands):
[email protected]:~# lxc-attach --name b51e74933ece2f3f457ec87c3a4e7b649149e9cff2a4705bef2a070f7adbafb0 [email protected]:~# cat /proc/mounts > /etc/mtab [email protected]74933ece:~# service mysql restart * Starting MariaDB database mysqld [ OK ] * Checking for corrupt, not cleanly closed and upgrade needing tables. [email protected]:~# vi /etc/mysql/my.cnf #wsrep_cluster_address=gcomm:// wsrep_cluster_address=gcomm://172.17.0.2,172.17.0.3,172.17.0.4 [email protected]:~# exit exit [email protected]:~# lxc-attach --name 03109c7018c03ddd8448746437346f080a976a74c3fc3d15f0191799ba5aae74 [email protected]:~# cat /proc/mounts > /etc/mtab [email protected]:~# vi /etc/mysql/my.cnf #wsrep_cluster_address=gcomm:// wsrep_cluster_address=gcomm://172.17.0.2,172.17.0.3,172.17.0.4 [email protected]:~# service mysql start * Starting MariaDB database server mysqld [ OK ] * Checking for corrupt, not cleanly closed and upgrade needing tables. [email protected]:~# mysql -uroot -proot Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 30 Server version: 5.5.39-MariaDB-1~trusty-wsrep mariadb.org binary distribution, wsrep_25.10.r4014 Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show status like 'wsrep_cluster%'; +--------------------------+--------------------------------------+ | Variable_name | Value | +--------------------------+--------------------------------------+ | wsrep_cluster_conf_id | 2 | | wsrep_cluster_size | 2 | | wsrep_cluster_state_uuid | 42bc375b-2bc0-11e4-851c-1a7627c0624c | | wsrep_cluster_status | Primary | +--------------------------+--------------------------------------+ 4 rows in set (0.00 sec_ MariaDB [(none)]> exit Bye [email protected]:~# exit exit [email protected]:~# lxc-attach --name 1db2a9a520f85d2cef6e5b387fa7912890ab69fc0918796c1fae9c1dd050078f [email protected]:~# cat /proc/mounts > /etc/mtab [email protected]:~# vi /etc/mysql/my.cnf [email protected]:~# service mysql start * Starting MariaDB database server mysqld [ OK ] [email protected]:~# mysql -uroot -proot Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 34 Server version: 5.5.39-MariaDB-1~trusty-wsrep mariadb.org binary distribution, wsrep_25.10.r4014 Copyright (c) 2000, 2014, Oracle, Monty Program Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> show status like 'wsrep_cluster%'; +--------------------------+--------------------------------------+ | Variable_name | Value | +--------------------------+--------------------------------------+ | wsrep_cluster_conf_id | 3 | | wsrep_cluster_size | 3 | | wsrep_cluster_state_uuid | 42bc375b-2bc0-11e4-851c-1a7627c0624c | | wsrep_cluster_status | Primary | +--------------------------+--------------------------------------+ 4 rows in set (0.00 sec) MariaDB [(none)]> exit Bye [email protected]:~# exit exit
Now be honest… Wasn’t that easier than creating multiple virtual machines and configuring the OS for each?
Enjoy your new MariaDB Galera Cluster and happy Dockering!
5 Comments. Leave new
Hi,
first of all, thank you for your tutorial – well done. I followed it step by step and was able to create a working cluster easily.
At least at first glance, after setting up all nodes, i created a new database on node 1 and it was correctly replicated to node 2 and 3. Then i inserted test data and that also got replicated just fine. After that i wanted to try what happens if a node fails and goes down. So i stopped the service on node 2 and dropped the database on node 1.
This is where the problem occurs, when trying to bring node 2 back into the cluster, restarting the service failed – presumable because the node needs to resync to current state.
Since it’s easy with docker to revert to initial state, i freshly restarted the cluster with all 3 nodes and tried adding a 4th node without doing anything on the database before. It worked like a charm, cluster size was 4 after that and changes were correctly replicated. After that, with changes made, i tried to add a 5th node … the same problem like before, starting the service failed.
It seems, that an out-of-sync node is unable to perform SST whereas IST seems to work perfectly. I already turned on logging, the last log entry show the SST request – that’s it.
Did you encounter similar problems? I’m really stuck here, not able to cover failover or scale the cluster afterwards make it unusable for me.
I would really appreciate a response, thanks in advance.
Greetz,
Jens
hi, Jens,
I have similar problem as yours. With any changes made to the cluster, I can not add new node to the cluster. Do you have any solution to this? Thanks.
This issue is related to my.cnf configuration attached. Galera version 25 has an issue with binlog, so those 2 lines need to be commented out as below or removed:
#innodb_autoinc_lock_mode=2
#innodb_locks_unsafe_for_binlog=1
Some may say binlog removal would stop galera from working but in my case it allowed me to resolve theissue.
I managed to add 2 nodes using this tutorial. When I tried to add the third one, maria would crash on the first two nodes. Tried to apply some –wsrep commands, to rebuild the cluster, with no effect. Started MYSQLD in safe mode, found out requests of replication were timing out (110). Tried to start replication manually.
I searched a lot of posts until I came into this bug and I couldn’t find the answer anywhere. Finally I decided to take the trial and error approach and I managed to add 2 more nodes and also start Maria on all of the servers.
It is a great tutorial though, actually on few available right now. It helped me a lot.
@hui kang
Hi there,
in my case it seems to be a problem with rsync, maybe it can be fixed by changing rsync version, but since I already thought changing to xtrabackup I gave that a try and it worked out of the box.
Here is what I did:
– Install Percona xtrabackup:
apt-get -q -y install percona-toolkit percona-xtrabackup netcat-openbsd
– Change sst method in my.cnf:
wsrep_sst_method = xtrabackup
After restarting all cluster nodes everything went fine. And a nice side effect is, that with xtrabackup the donor node allows full access during sync – with rsync it only allows read-only access.
Hope that helps in your case too,
Greetz
Jens
Step 10 : RUN LC_ALL=en_US.utf8 DEBIAN_FRONTEND=noninteractive apt-get -o Dpkg::Options::=’–force-confnew’ -qqy install mariadb-galera-server galera mariadb-client
—> Running in 7b7fae1ced73
E: Unable to correct problems, you have held broken packages.