Oracle RAC on the Cloud, Part 1

I’ve been working on moving a lot of the testing and R&D work I do away from local virtual machines and onto cloud environments, for a few reasons:

• I can avoid carrying around a laptop all the time, and rather log onto the cloud wherever I happen to be
• It’s easy to scale down and scale up capacity as required
• You pay for what you use
• Bandwidth is fast and cheap

One gap so far has been anything involving Oracle RAC. A number of attempts have been made to make it run under Amazon Web Services, notably Jeremy Schneider (blog post link) and Jeremiah Wilton. My experimentation with Amazon cloud environments has hit two major roadblocks. The first is lack of shared storage: the block-level (EBS) storage product can only be mounted my one machine at a time, while RAC is built on the concept of shared disk. The second issue is networking: Oracle RAC expects to manage its own IP addresses, creating a gaggle of VIPs and SCAN IPs. But Amazon and similar cloud providers require all IP addresses to be managed through their own API.

Enter Gandi, stage left

So it was with interest that I read about hosting provider Gandi’s new private VLAN service. They claim to offer layer-2 network services, much like the vSwitches in VMWare, network bridges in Oracle VM, or plain old physical layer-2 switches. By looking like a real switch, they would allow Oracle’s grid infrastructure to manage IP addresses like it expects to.

The issue of shared storage remains; I have yet to find any public cloud provider that offers true shared block storage. For a testing environment, though, we can simulate shared storage by setting up a NFS server that shares its own local disk with the RAC nodes. Highly available it is not, but at least it should let us set up and run the grid infrastructure.

Gandi is a provider I’ve used for domain name registration in the past, but I guess they’ve started IaaS hosting much along the lines of Amazon, including flexible per-hour charges. And unlike Amazon, they offer very flexible resizing of servers: RAM, disk space, networks can be changed dynamically, often without even a reboot.

To try this out, the first step is naturally to sign up with the service. Gandi actually offers a “free trial”: 30,000 free credits to try out the service. (Sounds like a lot, but a credit is actually worth a fraction of a cent). To take advantage, go to the trial page create an account, and you’ll be asked to describe how you’re planning to use the service. In my case, I was approved within an hour, and actually got 60,000 credits (2 servers?).

The setup

So here’s the setup I’m thinking of:

A few notes about the config: server01 will act as shared storage as well as the only access point to the Internet, so all inbound access will be via server01. I’ll use either SSH or SSH port forwarding for access, though a more permanent solution would probably involve a VPN like OpenVPN. (I can put together a VPN walkthrough if there’s enough interest, though there are likely already good ones online). server01 will also host internal DNS and DHCP services for our little network, saving some tedious /etc/hosts configuration. Network-wise, in addition to the built-in globally-routable IP addresses, I’ll be adding two private VLAN networks: the RAC public and RAC private networks. Ideally we could remove the internet-routable IP addresses from rac01 and rac02 completely. And although the management interface permits this, the Gandi boot-up scripts didn’t like the config at all and resulted in a non-bootable VM, so the globally routable IPs will stay for now.

You may also notice that the RAC servers only have 1GB of RAM when RAC officially requires 4. This is purely a cost saving during the initial install, taking advantage of the capability to change sizes later. Each server has two mountpoints: a default 3GB system “/” partition, plus a data disk to store shared data (for server01) and Oracle binaries (for the rest).

On the VLANs, I’ll use 10.100.0.x addresses in RAC-public, and 10.100.1.x addresses in RAC-private.

Firing up the VM

Creating the servers from the GUI, as per the table above.

On the server side, I’m using CentOS 6.4 64-bit as the operating system, the default system disk, and a data disk for the ORACLE_HOME and eventual data. Oracle Linux would be preferable as it’s a certified OS with Oracle 12c, but Gandi does not supply an install image. CentOS is anyways very very similar, except for a few critical differences we’ll get to later. Selecting to use a SSH key security, and pasting in a SSH key I already have. At first I got the error message
“This is not a valid public SSH key” before realizing that the key must be in OpenSSH rather than ssh.com format. Fortunately ssh-keygen can do the key format conversion:

ssh-keygen -i -f id_rsa.pem

The first server created fine, but the VLAN creation errored out with an internal error. Creating a support ticket, which, I heard some hours later, is being sent to the development team for investigation. Good thing this isn’t anything critical! Trying to create a second server, I got an error message that I have run out of “disk quota”. And re-reading the e-mail about the free credits, it looks like they did put a restriction on disk space. So one way or another, you do need to give them some money. I bought their entry-level package of 150k credits for $16.58. After the order went through, not only the server creation worked, but VLANs work too.

Using the “interfaces” tab to create new the RAC-Public and RAC-Private VLANs, and attaching each to all three servers.

Configuring the NFS server

Logging onto the servers via SSH as the root user using the SSH key added during the install, using the public IP listed in the Gandi console. And doing some basic OS-level config.

Setting up a simple static network config on server01 for the local net:

cd /etc/sysconfig/network-scripts
cat > ifcfg-eth1 <<-EOF
DEVICE=eth1
IPADDR=10.100.0.1
NETMASK=255.255.255.0
ONBOOT=yes
NAME=rac-public
EOF
cat > ifcfg-eth2 <<-EOF
DEVICE=eth2
IPADDR=10.100.1.1
NETMASK=255.255.255.0
ONBOOT=yes
NAME=rac-private
EOF
service network restart

Since this server is accessible over the Internet, we need a basic firewall:

iptables -F INPUT
# Allow existing connections
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# Loopback traffic
iptables -A INPUT -i lo -j ACCEPT
# RAC public network
iptables -A INPUT -i eth1 -j ACCEPT
# RAC private network
iptables -A INPUT -i eth2 -j ACCEPT
# SSH incoming
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
# Log and reject everything else
iptables -A INPUT -m limit --limit 2/minute -j LOG --log-prefix "iptables-input: " --log-level info
iptables -A INPUT -j REJECT
service iptables save

If you want to be even more secure, you could restrict the source IPs for SSH access. If you always come from static IP 1.2.3.4, you could replace the SSH incoming line with:

iptables -A INPUT -s 1.2.3.4 -p tcp --dport 22 -j ACCEPT

Getting ready for server01 to be a NFS server. Gandi’s scripts automatically mount the data disk at /srv/datadisk01 (NFS parameters taken from the great guide on oracle-base.com)

mkdir /srv/datadisk01/oradata /srv/datadisk01/dl
cat > /etc/exports <<-EOF
/srv/datadisk01/oradata 10.100.0.0/24(rw,sync,no_wdelay,insecure_locks,no_root_squash)
/srv/datadisk01/dl 10.100.0.0/24(rw,no_root_squash)
EOF
yum -y install nfs-utils
service rpcbind start
service nfs start
chkconfig rpcbind on
chkconfig nfs on

DNS and DHCP

We’ll now set up the DHCP and DNS using the wonderfully easy-to-configure dnsmasq. You may not have heard of dnsmasq before, but if you have a home wireless router, you’re likely using it already for DNS forwarding. It also has the capability to do authoritative DNS, which we’ll use here.

First, we need to create the /etc/hosts file that will both do local hostname resolution and be the source for dnsmasq’e entries:

cat >> /etc/hosts <<-EOF
# RAC public
10.100.0.1      server01-pub
10.100.0.2      rac01-pub
10.100.0.3      rac02-pub
10.100.0.12     rac01-pub-vip
10.100.0.13     rac02-pub-vip
# SCAN IPs
10.100.0.100    rac-cluster
10.100.0.101    rac-cluster
10.100.0.102    rac-cluster
# RAC private
10.100.1.1      server01-priv
10.100.1.2      rac01-priv
10.100.1.3      rac02-priv
EOF

And now we can install dnsmasq itself. It requires a small change to the default configuration to get DHCP running: creating a dummy dynamic DHCP range, and configuring it to assign IPs for rac01/02 public and private networks.

yum -y install dnsmasq
cat >> /etc/dnsmasq.conf <<-EOF
# Dummy DHCP range to enable the DHCP server
dhcp-range=10.99.99.99,10.99.99.102,12h
# Static DHCP entries for the RAC servers; addresses come from /etc/hosts
dhcp-host=rac01-priv
dhcp-host=rac01-pub
dhcp-host=rac02-priv
dhcp-host=rac02-pub
EOF
chkconfig dnsmasq on
service dnsmasq start
netstat -anp | grep dnsmasq
# Make sure it's running: you should see lines for 0.0.0.0:53 and 0.0.0.0:67 here

Downloading Oracle 12c

And we might as well kick off an Oracle software download to run while the rest of the config is done. And why not use the latest and greatest, Oracle 12c? It’s possible to simply download the archives to your local machine and transfer to the cloud server using scp, but those are big files and my local Internet isn’t _that_ fast, so it’s definitely preferable to download from the server directly. The recent blog post by Andre Araujo pointed me to an easy way to do the download on YouTube, at least for Firefox users. Paraphrasing the video, you need to go to the download page, start the download locally, open the FF download manager, right-click “Copy Download Link”, and feed that literal link to wget. No messing with cookies required. The local download can then be cancelled.

The wget command will end up looking something like this:

cd /srv/datadisk01
mkdir dl oradata
cd dl
wget https://download.oracle.com/otn/linux/oracle12c/121010/linuxamd64_12c_database_1of2.zip?AuthParam=(from Firefox copy) &
wget https://download.oracle.com/otn/linux/oracle12c/121010/linuxamd64_12c_database_2of2.zip?AuthParam=(from Firefox copy) &
wget https://download.oracle.com/otn/linux/oracle12c/121010/linuxamd64_12c_grid_1of2.zip?AuthParam=(from Firefox copy) &
wget https://download.oracle.com/otn/linux/oracle12c/121010/linuxamd64_12c_grid_2of2.zip?AuthParam=(from Firefox copy) &

We’ll need parts 1 and 2 of the database download, and parts 1 and 2 of the grid infrastructure download, using the Linux x86-64 platform.

Even on a reasonably fast network like Gandi’s, it still took over an hour to run. So while downloading, we can move onto part 2 (coming soon), where we configure the RAC hosts themselves.

Lessons learned

• Don’t count on the “free trial” at Gandi to actually get usable infrastructure, but $18 won’t break the bank either for this type of infrastructure.
• Once you pony up the money, though, Gandi’s VLAN service does do what it advertises
• Even with a small 2-node cluster, DHCP and DNS make configuration easier and less error-prone
• At one point while setting up networking, I managed to make the network unreachable. And while Gandi provides an emergency console tool, I wasn’t able to get it to work: it showed console messages all right, but no login prompt. So be very careful about any network or bootup configs that could potentially lock you out.