Oracle RAC on Azure

Posted in: Oracle, Technical Track

Microsoft Azure provides an acceptable and affordable platform for a training environment. I am an Oracle DBA, and use it to test functionality, new technologies and features of different Oracle products. Azure supplies a template for Oracle linux and it can be used to run a single database, but when we try to create an Oracle RAC, we hit two major issues.

In the first, the Azure virtual network doesn’t support multicast and, as result, cannot be used for interconnect. The second issue is shared storage. Azure provides shared file storage, and you can access it using SMB-2 protocol, but it isn’t exactly what we need for RAC. How we can solve or workaround those problems? I will share my experience and show how I can setup a RAC on Azure.

For a two node RAC we first need to create at least two virtual machines for the cluster nodes. I’ve chosen Oracle Linux 6.4 from Azure Marketplace. I decided to create the machines with 2 network interfaces where one will be used for public, and another will be used for private interconnect. Here is my blog post how to create a VM with 2 network interfaces. It may not be necessary since you can fork a virtual interface out of your only public network, but I decided to go this way and create cluster nodes with two interfaces.

Here is output for the network from the first node:

[root@oradb5 network-scripts]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0D:3A:11:A3:71
inet addr:10.0.1.11 Bcast:10.0.1.255 Mask:255.255.254.0
inet6 addr: fe80::20d:3aff:fe11:a371/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:776 errors:0 dropped:0 overruns:0 frame:0
TX packets:789 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:96068 (93.8 KiB) TX bytes:127715 (124.7 KiB)

eth1 Link encap:Ethernet HWaddr 00:0D:3A:11:AC:92
inet addr:10.0.2.11 Bcast:10.0.3.255 Mask:255.255.254.0
inet6 addr: fe80::20d:3aff:fe11:ac92/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:722 (722.0 b) TX bytes:1166 (1.1 KiB)

We need to install oracle-rdbms-server-12cR1-preinstall-1.0-14.el6.x86_64 rpm. It will install all required packages and set up kernel and limits for oracle user on our boxes :

yum install oracle-rdbms-server-12cR1-preinstall-1.0-14.el6.x86_64

The next step is to enable multicast support on the network for interconnect. You can read how to enable the multicast support in my other blog. As result you are getting a network interface edge0 which can be used now for our private network. Here is output of the ifconfig after crating virtual interface with support of multicast:

[root@oradb5 ~]# ifconfig
edge0 Link encap:Ethernet HWaddr 9E:1A:D8:0B:94:EF
inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::9c1a:d8ff:fe0b:94ef/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1400 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 b) TX bytes:238 (238.0 b)

eth0 Link encap:Ethernet HWaddr 00:0D:3A:11:A3:71
inet addr:10.0.1.11 Bcast:10.0.1.255 Mask:255.255.254.0
inet6 addr: fe80::20d:3aff:fe11:a371/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:118729 errors:0 dropped:0 overruns:0 frame:0
TX packets:62523 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:143705142 (137.0 MiB) TX bytes:20407664 (19.4 MiB)

eth1 Link encap:Ethernet HWaddr 00:0D:3A:11:AC:92
inet addr:10.0.2.11 Bcast:10.0.3.255 Mask:255.255.254.0
inet6 addr: fe80::20d:3aff:fe11:ac92/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:9 errors:0 dropped:0 overruns:0 frame:0
TX packets:271 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1274 (1.2 KiB) TX bytes:43367 (42.3 KiB)

I’ve used multicast tool from Oracle support document

Grid Infrastructure Startup During Patching, Install or Upgrade May Fail Due to Multicasting Requirement (Doc ID 1212703.1)

The check was successful:

[oracle@oradb5 mcasttest]$ ./mcasttest.pl -n oradb5,oradb6 -i edge0
########### Setup for node oradb5 ##########
Checking node access 'oradb5'
Checking node login 'oradb5'
Checking/Creating Directory /tmp/mcasttest for binary on node 'oradb5'
Distributing mcast2 binary to node 'oradb5'
########### Setup for node oradb6 ##########
Checking node access 'oradb6'
Checking node login 'oradb6'
Checking/Creating Directory /tmp/mcasttest for binary on node 'oradb6'
Distributing mcast2 binary to node 'oradb6'
########### testing Multicast on all nodes ##########

Test for Multicast address 230.0.1.0

Nov 24 16:22:12 | Multicast Succeeded for edge0 using address 230.0.1.0:42000

Test for Multicast address 224.0.0.251

Nov 24 16:22:13 | Multicast Succeeded for edge0 using address 224.0.0.251:42001
[oracle@oradb5 mcasttest]$

So, we have solved the first obstacle and need to get shared storage for our RAC. We have at least a couple of options here, and I believe somebody can advise us on others. We can use NFS based shared storage, or we can use iscsi for that. You may choose something from the Azure Marketplace like SoftNAS or Stonefly or you may decide to create your own solution. In my case I just faired another Oracle Linux VM, added couple of storage disks to it using portal, and then set up NFS server on that machine. Here is the high level description for that:
We create a linux based VM on Azure using Oracle Linux 6.4 template from Marketplace. The size will be dictated by your requirements. I called the machine oradata.
I’ve added a 20 Gb disk to the oradata machine through the Azure portal, and created a partition and filesystem on it:

[root@oradata ~]# fdisk -l
[root@oradata ~]# fdisk /dev/sdc
[root@oradata ~]# mkfs.ext4 /dev/sdc1
[root@oradata ~]# mkdir /share
[root@oradata ~]# mkdir /share/oradata1
[root@oradata ~]# e2label /dev/sdc1 sharedoradata1
[root@oradata ~]# vi /etc/fstab
[root@oradata ~]# mount -a
[root@oradata ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 7.4G 1.4G 5.7G 19% /
tmpfs 1.7G 0 1.7G 0% /dev/shm
/dev/sda1 485M 50M 410M 11% /boot
/dev/sda2 2.0G 67M 1.9G 4% /tmp
/dev/sdc1 20G 4.2G 15G 23% /share/oradata1
/dev/sdb1 60G 180M 56G 1% /mnt/resource
[root@oradata ~]#

Installed necessary utilities using yum:
[root@oradata ~]# yum install nfs-utils
Configured NFS server on the box:

[root@oradata ~]# chkconfig service nfs on
[root@oradata ~]# vi /etc/exports
[root@oradata ~]# cat /etc/exports
/share/oradata1 10.0.0.0/23(rw,sync,no_root_squash)
[root@oradata ~]# service nfs restart
[root@oradata ~]# showmount -e
Export list for oradata:
/share/oradata1 10.0.0.0/23

Configure or stop firewall(You may need to do it on your cluster nodes as well) :

[root@oradata ~]# service iptables stop
iptables: Flushing firewall rules: [ OK ] iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Unloading modules: [ OK ] [root@oradata ~]# chkconfig iptables off
[root@oradata ~]#

On your cluster nodes you need add the mountpoint for your shared storage to /etc/fstab and mount it.

[root@oradb5 ~]# vi /etc/fstab
[root@oradb5 ~]# cat /etc/fstab | grep nfs
oradata:/share/oradata1 /u02/oradata nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,actimeo=0,vers=3,timeo=600 0 0
[root@oradb5 ~]# mount -a
[root@oradb5 ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 7.4G 2.5G 4.6G 36% /
tmpfs 3.5G 0 3.5G 0% /dev/shm
/dev/sda1 485M 69M 391M 15% /boot
/dev/sda2 2.0G 86M 1.8G 5% /tmp
/dev/sdc1 60G 12G 45G 21% /u01/app
/dev/sdb1 281G 191M 267G 1% /mnt/resource
oradata:/share/oradata1
20G 4.2G 15G 23% /u02/oradata
[root@oradb5 ~]# mount | grep /u02/oradata | grep -v grep
oradata:/share/oradata1 on /u02/oradata type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,noac,actimeo=0,vers=3,timeo=600,addr=10.0.1.101)
[root@oradb5 ~]#

Now we have the required storage for OCR and Voting disks, network for public and interconnect, and can install our cluster.
We need to correct /etc/hosts file on both nodes (you may choose to use Azure DNS service instead).

[oracle@oradb5 ~]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.1.11 oradb5
10.0.1.12 oradb6
10.0.1.15 oradb5-vip
10.0.1.16 oradb6-vip
10.0.1.19 oradb-clst-scan
192.168.1.1 oradb5-priv
192.168.1.2 oradb6-priv
10.0.1.101 oradata
[oracle@oradb5 ~]$

You can see I setup the public,VIP and SCAN in the hosts file. Of course it is not acceptable for any production implementation or if you want to have more than one scan. As I’ve already mentioned above you can use DNS for proper installation.
We copy required software to one of the nodes, unpack it and create a response file for installation like:

[oracle@oradb5 ~]$ cat grid.rsp
oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v12.1.0
ORACLE_HOSTNAME=oradb5
INVENTORY_LOCATION=/u01/app/oraInventory
SELECTED_LANGUAGES=en
oracle.install.option=CRS_CONFIG
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/12.1.0/grid
oracle.install.asm.OSDBA=dba
oracle.install.asm.OSOPER=dba
oracle.install.asm.OSASM=dba
oracle.install.crs.config.gpnp.scanName=oradb-clst-scan
oracle.install.crs.config.gpnp.scanPort=1521
oracle.install.crs.config.ClusterType=STANDARD
oracle.install.crs.config.clusterName=oradb-clst
oracle.install.crs.config.gpnp.configureGNS=false
oracle.install.crs.config.autoConfigureClusterNodeVIP=false
oracle.install.crs.config.gpnp.gnsOption=CREATE_NEW_GNS
oracle.install.crs.config.gpnp.gnsClientDataFile=
oracle.install.crs.config.gpnp.gnsSubDomain=
oracle.install.crs.config.gpnp.gnsVIPAddress=
oracle.install.crs.config.clusterNodes=oradb5:oradb5-vip,oradb6:oradb6-vip
oracle.install.crs.config.networkInterfaceList=eth0:10.0.0.0:1,eth1:10.0.2.0:3,edge0:192.168.1.0:2
oracle.install.crs.config.storageOption=FILE_SYSTEM_STORAGE
oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=/u02/oradata/voting/vdsk1,/u02/oradata/voting/vdsk2,/u02/oradata/voting/vdsk3
oracle.install.crs.config.sharedFileSystemStorage.votingDiskRedundancy=NORMAL
oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=/u02/oradata/ocr/ocrf1
oracle.install.crs.config.sharedFileSystemStorage.ocrRedundancy=EXTERNAL
oracle.install.crs.config.useIPMI=false
oracle.install.crs.config.ipmi.bmcUsername=
oracle.install.crs.config.ipmi.bmcPassword=
oracle.install.asm.SYSASMPassword=
oracle.install.asm.diskGroup.name=
oracle.install.asm.diskGroup.redundancy=
oracle.install.asm.diskGroup.AUSize=1
oracle.install.asm.diskGroup.disks=
oracle.install.asm.diskGroup.diskDiscoveryString=
oracle.install.asm.monitorPassword=
oracle.install.asm.ClientDataFile=
oracle.install.crs.config.ignoreDownNodes=false
oracle.install.config.managementOption=NONE
oracle.install.config.omsHost=
oracle.install.config.omsPort=0
oracle.install.config.emAdminUser=
oracle.install.config.emAdminPassword=

The file can be used for silent installation. You may choose instead to use runInstaller in GUI mode.
To run installation in silent mode you just need to go to your unpacked software and run:

[oracle@oradb5 grid]$ ./runInstaller -silent -responseFile /home/oracle/grid.rsp -ignoreSysPrereqs -ignorePrereq
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 415 MB. Actual 1350 MB Passed
Checking swap space: 0 MB available, 150 MB required. Failed <<<>> Ignoring required pre-requisite failures. Continuing…

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2016-02-01_09-41-01AM. Please wait …

You’ve of course noticed that I’ve run the installation ignoring requirements. As a matte of fact, I ran it without ignoring, checked the failing checks, made necessary adjustments for those checks, and then I decided they were important and left other as they were. As example my /etc/resolve.conf file was different due to settings on dhcp server and so on. I advise to apply common sense and your knowledge to decide what checks are important for you and what can be ignored.
Your installation will be completed and all you need to run is a couple of scripts to finish the installation.

As a root user, execute the following script(s):
1. /u01/app/12.1.0/grid/root.sh

Execute /u01/app/12.1.0/grid/root.sh on the following nodes:
[oradb5, oradb6]

Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes.

Successfully Setup Software.
As install user, execute the following script to complete the configuration.
1. /u01/app/12.1.0/grid/cfgtoollogs/configToolAllCommands RESPONSE_FILE=

Note:
1. This script must be run on the same host from where installer was run.
2. This script needs a small password properties file for configuration assistants that require passwords (refer to install guide documentation).

We run the root.sh on each node one by one as user root and execute configToolAllCommands script as user oracle on the node we have run our installation. The response file would be required if we specified password for ASM,ASM monitoring or for DBCA. Here is an example of the file contents:

oracle.assistants.server|S_SYSPASSWORD=welcome1
oracle.assistants.server|S_SYSTEMPASSWORD=welcome1
oracle.assistants.server|S_DBSNMPPASSWORD=welcome1
oracle.assistants.server|S_PDBADMINPASSWORD=welcome1
oracle.assistants.server|S_EMADMINPASSWORD=welcome1
oracle.assistants.server|S_ASMSNMPPASSWORD=welcome1

Change permission for the file to 600 before running the script:

[oracle@oradb5 grid]$ vi /home/oracle/cfgrsp.properties
[oracle@oradb5 grid]$ chmod 600 /home/oracle/cfgrsp.properties

We don’t have any ASM in our installation or BMS console but I will leave the file nevertheless just for reference.
Here is an output what we ran on our system :

[root@oradb5 ~]# /u01/app/12.1.0/grid/root.sh
Check /u01/app/12.1.0/grid/install/root_oradb5_2016-02-01_10-21-07.log for the output of root script
....

[root@oradb6 ~]# /u01/app/12.1.0/grid/root.sh
Check /u01/app/12.1.0/grid/install/root_oradb6_2016-02-01_10-38-50.log for the output of root script
….

[oracle@oradb5 grid]$ /u01/app/12.1.0/grid/cfgtoollogs/configToolAllCommands RESPONSE_FILE=/home/oracle/cfgrsp.properties
Setting the invPtrLoc to /u01/app/12.1.0/grid/oraInst.loc

perform – mode is starting for action: configure
….

Keep in mind the configToolAllCommands should also create the management database in your cluster. If somehow it was failed you can try to recreate it using dbca in silent mode like :

/u01/app/12.1.0/grid/bin/dbca -silent -createDatabase -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType FS -datafileDestination /u02/oradata/ocr/oradb-clst/mgmtdb -datafileJarLocation /u01/app/12.1.0/grid/assistants/dbca/templates -characterset AL32UTF8 -autoGeneratePasswords -skipUserTemplateCheck -oui_internal

The RAC is created and now it can be used for application high availability or for databases tests. You may install a database software on the RAC either using GUI installer or silent mode, but don’t forget to specify cluster nodes during installation. I would also like to mention that I would not recommend installing it as production system, but it is quite suitable for tests or to experiment if you want to verify or troubleshot some RAC specific features.

email

Author

Interested in working with Gleb? Schedule a tech call.

About the Author

Regarded by his peers as an Oracle guru, Gleb is known for being able to resolve any problem related to Oracle. He loves the satisfaction of troubleshooting, and his colleagues even say that seeking Gleb’s advice regarding an issue is more efficient than looking it up. Gleb enjoys the variety of challenges he faces while working at Pythian, rather than working on the same thing every day. His areas of speciality include Oracle RAC, Exadata, RMAN, SQL tuning, high availability, storage, performance tuning, and many more. When he’s not working, running, or cycling, Gleb can be found reading.

19 Comments. Leave new

Joseph Dantoni
February 11, 2016 4:57 pm

This is really cool, like you mentioned, not supported, but great job getting it working.

Reply

Hi Gleb,
I found your blog about Oracle RAC on Azure. May I contact you and discuss some details about your solution ?

Reply

Hi Jaroslav,
Yes, sure. I’ve sent you a private email.

Thanks,
Gleb

Reply
Stanley Fernandes
May 25, 2020 1:01 am

Hi Gleb

This is Stanley and I have been told to implement Oracle extended cluster on Azure Stack.
Basically for Stretched Cluster we have 2 ASM mirror sites and one location to keep the tie breaker voting disk file.
Could you please guide me on this as i need to implement within few days in my office.
Any step by step guide on this would be highly appreciated.
Thanks in Advance

Reply

Ear Gleb,
We found your blog about Oracle RAC on Azure. Could we contact you and discuss some details about your solution ?

Yours sicnerely, Eagle

Reply
Gleb Otochkin
January 16, 2017 9:47 am

Replied privately.

Reply
Alistair Reed
April 28, 2017 5:31 am

Oracle corp strongly criticised a similar approach to yours (but on AWS), essentially claiming the virtualization IO and networking is unreliable, introduces more failure points and is less performant.

Have you found your solution fast enough and robust enough to support a production commercial service with a meaningful sized user base?

Reply
Gleb Otochkin
April 28, 2017 6:05 am

Hi Alistair,
I am familiar with Oracle position and opinion about RAC on AWS. You may see in my summary that I don’t recommend the solution for production but rather for tests and experiments. I have the tested solution using synthetic load and found it stable enough during the tests but, again, it may not be sufficient for your question. The definition of “meaningful sized” can be quite different.
Thanks,
Gleb

Reply

Gleb. Have you tried your RAC solution on AWS?

Reply
Gleb Otochkin
May 11, 2017 9:05 am

Yes, I tried it using more or less the same approach. Also we have another options there to use flashgrid software as a layer for storage and interconnect network.

Thanks,
Gleb

Reply

Thanks for posting your experience. I am finding some of the same issues so it was good to confirm I am not going crazy :)

Reply

How come you didn’t use Oracle ClusterWare to solve your issue for the shared storage?

Reply
Gleb Otochkin
April 10, 2018 3:53 pm

Not sure if I’ve got your question Chad. You always use oracle clusterware (or how it calls now – Grid Infrastructure) for an Oracle RAC. Oracle clusterware itself requires a shared storage to put at least voiting disks and crs there.
In my example I used NFS as a shared storage. You may choose iscsi based shared storage and now, I believe, you can use Flashgrid software for multicast and shared storage.

Thanks,
Gleb

Reply

Hi Gleb,

What do you think is the benefit of installing Oracle RAC which is a cluster on top of another cluster (AWS or Azure)? Is it recommended? If yes, what are the main reasons? Can’t AWS/Azure handle HA or data resilience issues?

Thanks in advance.

Reply
saravanan santhalingam
June 8, 2019 7:15 pm

Gleb Otochkin

Nicely articulated , appreciate your knowledge sharing. after reading this post . i have few clarifications , can you please email me .

Reply
Alifiya Sadikali
June 11, 2019 8:17 am

Hello Saravan, Thank you for your kind comment. Gleb Otochkin is no longer at Pythian but if you have specific questions feel free to leave it in comments and I will try to see if we can have them answered by another expert.

Thanks

Reply
Stanley Fernandes
May 25, 2020 1:03 am

Hi Alifiya

This is Stanley and I have been told to implement Oracle extended cluster on Azure Stack.
Basically for Stretched Cluster we have 2 ASM mirror sites and one location to keep the tie breaker voting disk file.
Could you please guide me on this as i need to implement within few days in my office.
Any step by step guide on this would be highly appreciated.
Thanks in Advance

Reply

We found your blog about Oracle RAC on Azure. Could we contact you and discuss some details about your solution ?
Regards,
Amit

Reply

Hi Amit, if you are still interested in discussing this please reach out to myself personally or to sales@pythian.com – we can certainly help you in this area. Thanks.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *