If you regularly manage enormous data sets, you’ve probably heard about Tumblr’s exciting new toolkit called Jetpants, which automates common processes for MySQL data management, most notably in the area of rebalancing shards for more efficient scaling. In evaluating and using Jetpants in various environments, we identified some interesting installation and usage gotchas, and we’ve documented those here – let us know if you find other issues worth exploring.
Jetpants Installation/Usage Gotchas for Ubuntu 12.04:
Jetpants has to-date only been tested on RHEL/CentOS distributions, though working with the author (Evan Elias of Tumblr, thank you much!) I was able to get it running on Ubuntu 12.04 and Mint Maya/13. There will be more work to do, since RHEL/CentOS report back a service status (“service mysql status” output, see later in this post) differently than Ubuntu/Mint, and there may be other differences as well.
For your Jetpants Console workstation, install Ruby 1.9.3 (also 1.9.1 will be installed), and check /etc/alternatives/ruby points at ruby1.9.3. Also be sure you installed the MySQL client libraries.
# apt-get install ruby1.9.3 rubygems libmysqlclient-dev
# ln -sf /usr/bin/ruby1.9.3 /etc/alternatives/ruby
# ln -sf /usr/bin/gem1.9.3 /etc/alternatives/gem
After that, do gem install Jetpants. You may get scary warnings about inability to convert ASCII to UTF8 characters during documentation installation as below:
# gem install jetpants
Building native extensions. This could take a while…
Fetching: sequel-3.39.0.gem (100%)
Fetching: net-ssh-2.5.2.gem (100%)
Fetching: coderay-1.0.7.gem (100%)
Fetching: slop-3.3.3.gem (100%)
Fetching: method_source-0.8.gem (100%)
Fetching: pry-0.9.10.gem (100%)
Fetching: highline-1.6.15.gem (100%)
Fetching: colored-1.2.gem (100%)
Fetching: jetpants-0.7.4.gem (100%)
Successfully installed mysql2-0.3.11
Successfully installed sequel-3.39.0
Successfully installed net-ssh-2.5.2
Successfully installed coderay-1.0.7
Successfully installed slop-3.3.3
Successfully installed method_source-0.8
Successfully installed pry-0.9.10
Successfully installed highline-1.6.15
Successfully installed colored-1.2
Successfully installed jetpants-0.7.4
10 gems installed
Installing ri documentation for mysql2-0.3.11…
Installing ri documentation for sequel-3.39.0…
Installing ri documentation for net-ssh-2.5.2…
unable to convert “\xE7” from ASCII-8BIT to UTF-8 for lib/net/ssh/authentication/pageant.rb, skipping
Installing ri documentation for coderay-1.0.7…
Installing ri documentation for slop-3.3.3…
Installing ri documentation for method_source-0.8…
Installing ri documentation for pry-0.9.10…
Installing ri documentation for highline-1.6.15…
Installing ri documentation for colored-1.2…
Installing ri documentation for jetpants-0.7.4…
Installing RDoc documentation for mysql2-0.3.11…
Installing RDoc documentation for sequel-3.39.0…
Installing RDoc documentation for net-ssh-2.5.2…
unable to convert “\xE7” from ASCII-8BIT to UTF-8 for lib/net/ssh/authentication/pageant.rb, skipping
Installing RDoc documentation for coderay-1.0.7…
Installing RDoc documentation for slop-3.3.3…
Installing RDoc documentation for method_source-0.8…
Installing RDoc documentation for pry-0.9.10…
Installing RDoc documentation for highline-1.6.15…
Installing RDoc documentation for colored-1.2…
Installing RDoc documentation for jetpants-0.7.4…
Be sure you’ve created /var/www writeable by the user you’ll execute Jetpants as, and be sure you have an /etc/jetpants.yaml that is similar to the template provided on the Jetpants website.
And now you’re ready to run Jetpants!
$ jetpants
Tasks:
jetpants activate_slave # turn a standby slave into an active slave
jetpants clone_slave # clone a standby slave
jetpants console # Jetpants interactive console
jetpants destroy_slave # remove a standby slave from its pool
jetpants help [TASK] # Describe available tasks or one specific task
jetpants pools # display a full summary of every pool in the topology
jetpants pools_compact # display a compact summary (master, name, and size) of every pool in the topology
jetpants promotion # perform a master promotion, changing which node is the master of a pool
jetpants pull_slave # turn an active slave into a standby slave
jetpants rebuild_slave # export and re-import data set on a standby slave
jetpants regen_config # regenerate the application configuration
jetpants shard_cutover # truncate the current last shard range, and add a new shard after it
jetpants shard_offline # mark a shard as offline (not readable or writable)
jetpants shard_online # mark a shard as fully online (readable and writable)
jetpants shard_read_only # mark a shard as read-only
jetpants shard_split # shard split step 1 of 4: spin up child pools with different portions of data set
jetpants shard_split_child_reads # shard split step 2 of 4: move reads to child shards
jetpants shard_split_child_writes # shard split step 3 of 4: move writes to child shards
jetpants shard_split_cleanup # shard split step 4 of 4: clean up data that replicated to wrong shard
jetpants summary # display information about a node in the context of its pool
jetpants weigh_slave # change the weight of an active slave
As mentioned before, Jetpants is currently only well-tested for RHEL/CentOS distributions. RHEL/CentOS report back a service status (“service mysql status” output) differently than Ubuntu/Mint. Following is the requirements of Jetpants and actual output:
/sbin/service mysql status # check if mysql daemon running; output must include the string ‘not running’ if mysql is not running
This is how it works on both Ubuntu 12.04 and Mint Maya/13:
# service mysql stop
mysql stop/waiting
# service mysql status
mysql stop/waiting
# service mysql start
mysql start/running, process 31346
# service mysql status
mysql start/running, process 31346
I decided to check the return code, but it’s always 0 regardless of the service state, so we must parse the output in Jetpants. See the following output:
# service mysql status ; echo $?
mysql stop/waiting
0
# service mysql start
mysql start/running, process 31629
# service mysql status ; echo $?
mysql start/running, process 31629
0
To support Debian-alikes, Jetpants must check for the other string, since the strings being output by “service” makes it impossible to check for a single substring across distros.
Over the coming days, I will be identifying other CentOS/RHEL-specific code and writing patches to make it work with at least also Ubuntu 12.04. In the future, if you’re trying to port Jetpants to another distribution, you’ll be able to use my pull request as a template for the areas that are distribution-specific.
When running Jetpants in EC2, there are some more gotchas (or perhaps they’re just “be sure to follow best-practices”):
1. You must allow root login to your cluster. Generate a privkey/pubkey pair and allow root logins.
a. Edit ~root/.ssh/authorized_keys and get rid of the part where it has the “command=…” that prints an error and logs you out.
b. Edit /etc/cloud/cloud.cfg and set disable_root: 0
c. Edit /etc/ssh/sshd_config and set PermitRootLogin without-password
2. You must be sure your master/slaves replication topology uses only IP addresses. If your slaves have, eg “Master_Host: ec2-5-2-3-8.compute.amazonaws.com” then Jetpants will fail. IP addressses is how you want your cluster configured in production, so do it, but it may surprise you if you’re trying to build a small test cluster.
3. You must allow root login from localhost on all your MySQL instances. In other words, this command must return valid output on your slaves: mysql -e ‘show slave status\G’ – note that you can put root login credentials in ~root/.my.cnf rather than allow passwordless MySQL root logins.
In general, it’s good to remember that Jetpants is a set of utilities for controlling an actual large MySQL cluster, so it will not work so well for tiny little test deploys where you’ve cut corners, and this is how it should be. Here are more Jetpants Environment Requirements/Limitations, from the documentation:
1. Need pigz installed on your cluster. If you use the Palomino Cluster Tool to build your shards, pigz will be installed for you.
2. No auto_increment on sharded tables. Must have ID generator.
3. PK of all sharded tables should start with the shard key. This will vastly improve performance of long-running processes like shard splits.
4. Port :3306 everywhere. One instance per node.
5. (Probably) doesn’t work with MyISAM.
No comments