Oracle RAC on the Cloud, Part 2

Posted in: Cloud, Technical Track

In part 1 of this series, we talked about some of the challenges of setting up Oracle RAC on a public cloud provider, and went on to order some VMs from provider Gandi, and finally configuring an NFS server for shared storage. In this post, we move on to configuring the RAC servers themselves, rac01 and rac02.

Network config

After starting up the RAC nodes in the GUI, we can log in via the SSH key we created. The first order of business is to set up networking. The eth1 and eth2 VLAN interfaces by default have no network configuration at all. We’ll set up a configuration to use a DHCP, and sending a hostname to the DHCP server so it knows which IP address to give us. This ties into the dnsmasq configuration we set up in part 1, to automatically assign IP addresses to the eth1 and eth2 private-VLAN network interfaces.

cd /etc/sysconfig/network-scripts
cat > ifcfg-eth1 <<EOF
cat > ifcfg-eth2 <<EOF
service network restart
ip address list

And if all is working properly, we should see eth1 and eth2 each assigned an IP address.

Since this server is accessible over the Internet, we need a basic firewall, just like server01:

iptables -F INPUT
# Allow existing connections
iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# Loopback traffic
iptables -A INPUT -i lo -j ACCEPT
# RAC public network
iptables -A INPUT -i eth1 -j ACCEPT
# RAC private network
iptables -A INPUT -i eth2 -j ACCEPT
# SSH incoming
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
# Log and reject everything else
iptables -A INPUT -m limit --limit 2/minute -j LOG --log-prefix "iptables-input: " --log-level info
iptables -A INPUT -j REJECT
service iptables save

Fun with hostnames

For DNS resolution, we simply point ourselves to the DNS server on server01,

cat > /etc/resolv.conf <<-EOF

One issue I ran into with hostnames: the grid infrastructure install expects its public network to be associated with the hostname of the machine. But in the Gandi setup, the hostname is associated with the Internet-facing IP. Modifying /etc/sysconfig/network-scripts to change the hostname to rac01-pub.

perl -pi -e 's/HOSTNAME=rac01\\n/HOSTNAME=rac01-pub\\n/' /etc/sysconfig/network
hostname rac01-pub

But even after rebooting the hostname is still rac01. More digging showed it to be part of Gandi’s auto-configuration scripts. But conveniently they provide a file, /etc/sysconfig/gandi, where specific configurations can be turned off. There are a few that won’t play well with RAC: the hostname as mentioned above, but also the mountpoint name: the grid infrastructure expects the same mountpoint names for both RAC nodes, so the default /srv/rac01data mountpoints won’t work. And lastly, we have our DNS server, so we don’t want the Gandi configuration to mangle our resolv.conf.

The relevant section of the /etc/sysconfig/gandi looks like this:

# set to 0 to avoid hostname automatic reconfigure
# set to 0 to avoid nameserver automatic reconfigure
# allow mounting the data disk to the mount point using the disk label



Now our config won’t be clobbered on next reboot.

And speaking of the mountpoint, we need to undo what Gandi did and create our own mountpoint on /u01:

umount /srv/rac01data
mkdir /u01
cat >> /etc/fstab <<-EOF
LABEL=rac01data /u01    ext3    rw,nosuid,nodev,noatime 0 0
mount /u01

A bit about the /etc/fstab line: Gandi created our filesystem with a filesystem label, so we can use this to locate the disk even if the actual device node /dev/xvdb changes. The mountpoint options are taken form Gandi’s defaults. Notably, “noatime” avoids doing a disk write every time a file is accessed.

Filessystems and NFS

NFS client config:

mkdir -p /srv/datadisk01/oradata /srv/datadisk01/dl
cat >> /etc/fstab <<-EOF /srv/datadisk01/oradata nfs rw,bg,hard,nointr,tcp,vers=3,timeo=600,rsize=32768,wsize=32768,actimeo=0 0 0 /srv/datadisk01/dl nfs rw,bg,tcp 0 0
yum -y install nfs-utils
service rpcbind start
chkconfig rpcbind on
mount /srv/datadisk01/dl
mount /srv/datadisk01/oradata

And although Gandi gives us a minimal amount of swap, we’ll need 4GB to make the Oracle installer happy. Since the system disk is only 3GB it has to come from the data disk.

dd if=/dev/zero of=/u01/swapfile bs=1048576 count=4096
mkswap /u01/swapfile
swapon /u01/swapfile
# Make it persistent
echo "/u01/swapfile swap swap defaults 0 0" >> /etc/fstab

And Oracle checks for /dev/shm, just in case you want to use memory_target. Adding a config here to make it owned by Oracle to avoid ugly error messages from dbca later on. It you get a “ORA-00600: internal error code, arguments: [SKGMHASH]…” error message, it may very well be permission issues on /dev/shm. The “54321” userid is the numeric ID of the oracle user that the preinstall RPM will create later on.

echo "tmpfs /dev/shm tmpfs user=54321,group=54321,mode=755 0 0" >> /etc/fstab
mount /dev/shm

Oracle prerequisites

Now that storage is set up, we can start looking at Oracle stuff.

Going from the official Oracle Grid Infrastructure install guide

Browsing’s Oracle Linux 6 “latest” repository to find the latest database preinstall RPM. As of this writing, it’s 1.0-8.

yum -y localinstall oracle-rdbms-server-12cR1-preinstall-1.0-8.el6.x86_64.rpm

It complains about a UEK dependency. On this kind of cloud environment we can’t install custom kernels like UEK, so we do need to stay with default kernels. And we’re running CentOS anyway.

flashdbba has an excellent blog post on this subject; he listed a two-line workaround of downloading the full UEK kernel package, and installing it with the –justdb and –nodeps options. I can confirm that it works, but it requires a big UEK package download, plus results in warnings about missing dependencies every time yum is run.

So instead, we’re going to create a dummy RPM package to satisfy the dependency. It won’t have any files or scripts, but will match the name that the preinstall RPM is looking for.

To actually turn this specification into a RPM, we need the the rpm-build package installed as well:

yum -y install rpm-build
mkdir -p ~/rpmbuild/SPECS
cat > ~/rpmbuild/SPECS/dummy-kernel-uek.spec <<-EOF
Summary: Dummy UEK kernel RPM
Name: kernel-uek
Version: 2.6.39
License: GPLv2
Release: 1.dummy
Dummy RPM to satisfy oracle preinstall RPM dependencies
echo OK
rpmbuild -bb dummy-kernel-uek.spec

And if all went well, we should see a kernel-uek package in ~/rpmbuild/RPMS/x86_64

cd ~/rpmbuild/RPMS/x86_64/
yum -y localinstall kernel-uek*.rpm

Back at the original dependency package:

yum -y localinstall oracle-rdbms-server-11cR1-preinstall-*

50m of dependencies. Downloading them all. Setting up the NFS server was so much less work :-)

And we might as well use sudo from the Oracle account rather than using root:

yum -y install sudo

And uncommenting the line

%wheel  ALL=(ALL)       NOPASSWD: ALL

Adding oracle to the “wheel” group so it can run sudo:

usermod -G wheel oracle

Passwordless SSH

Now that we have an Oracle user, the installer will require passwordless SSH to be set up. For now, it’s just from the Oracle user to itself:

su - oracle
mkdir -p .ssh
cd .ssh
ssh-keygen -t rsa -b 2048 -N "" -f id_rsa
cat >> authorized_keys
chmod 0600 authorized_keys

Testing out the SSH, which lets us add the host key.

ssh rac01-pub

And setting a password for Oracle, which requires the passwd tool itself:

sudo yum -y install passwd
sudo passwd oracle

Creating directories for the Oracle install:

sudo mkdir -p /u01/app
sudo chown oracle:oinstall /u01/app
sudo chown oracle:oinstall /srv/datadisk01/oradata

And now it’s time to repeat it all the steps from this post, this time on rac02, replacing rac01 with rac02 where appropriate.

After that, it’s time for part 3

Lessons learned

  • If you’re using Gandi hosting for anything out of the ordinary, familiarize yourself with /etc/sysconfig/gandi
  • Dummy RPMs work very well to circumvent Oracle’s OS compatibility checks


Want to talk with an expert? Schedule a call with our team to get the conversation started.

About the Author

Marc is a passionate and creative problem solver, drawing on deep understanding of the full enterprise application stack to identify the root cause of problems and to deploy sustainable solutions. Marc has a strong background in performance tuning and high availability, developing many of the tools and processes used to monitor and manage critical production databases at Pythian. He is proud to be the very first DataStax Platinum Certified Administrator for Apache Cassandra.

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *