Oracle Silent Mode, Part 5: Adding a Node to a 10.2 RAC

This fifth post assumes that you want to add a new node to your cluster and database. It describes most of the associated “silent” syntaxes. Even if you don’t leverage RAC’s ability to add or remove nodes to gain in agility, it’s still very likely you’ll come to these techniques when you want to upgrade some of your Servers or Operating Systems. For a complete series agenda up to now, see below:

1. Installation of 10.2 And 11.1 Databases
2. Patches of 10.2 And 11.1 databases
3. Cloning Software and databases
4. Install a 10.2 RAC Database
5. Add a Node to a 10.2 RAC database (this post!)
6. Remove a Node from a 10.2 RAC database
7. Install a 11.1 RAC Database
8. Add a Node to a 11.1 RAC database
9. Remove a Node from a 11.1 RAC database
10. A ton of other stuff you should know

Adding a node involves adding all the components of the RAC in their order of appearance, i.e.: (1) The Clusterware and the associated resources, (2) the database software, (3) the listeners, (4) the ASM instance if necessary, and (5) the database instance. The last is the subject of this post.

For a complete reference to this procedure, see to the corresponding section of the 10.2 documentation.

Make Sure You Can Add The New Node To The Cluster

Before you start with anything else, proceed with the hardware and OS configuration. The new node must access the shared storage, the networks, the packages, parameter, users, etc. There is no difference if you add a node later or at the time of the initial install. You can use Oracle CVU and RDA as described in the corresponding section of the previous post to check that all the nodes can be part of a single cluster.

Adding a Node to the Clusterware

This step is actually pretty easy if the prerequisites are met. It’s also the one that can have the deepest impact on the cluster and the one that can end up badly. A lot of components are impacted by this addition: the voting disk, the cluster registry (OCR), and the inventories from all the nodes. Make sure you know how to revert any changes you are going to make. Make sure also that you have everything you need to revert the changes.

Note
Before you start with the node addition, make sure you’ve backed up the voting disk. There is no need to backup the OCR because it’s done automatically but it’s probably a good idea to locate the backup and make sure it contains the latest changes you’ve made.

You should also make sure that the Clusterware owner, oracle, or crs, or whatever it is, can write into the Clusterware ORACLE_HOME and the Oracle Inventory on the new server. Create the associated directory if necessary.

We’ll assume we want to add a new node rac-server5 to the cluster we’ve build in the previous post. In order to proceed, connect as the Clusterware owner on any of the existing nodes and run the set of commands below:

rac-server1$ cd /u01/app/crs/oui/bin
rac-server1$ ./addNode.sh -silent CLUSTER_NEW_NODES={rac-server5} \
                CLUSTER_NEW_PRIVATE_NODE_NAMES={rac-server5-priv} \
                CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac-server5-vip}

rac-server5, rac-server5-priv, and rac-server5-vip are respectively the public, the interconnect, and the VIP network aliases of the server to be added; all the nodes from the cluster must be able to resolve them. The previous step pushes the Clusterware to the new ORACLE_HOME, relinks it, and updates the inventories. None of the OCR or voting disks are updated at that step. To revert it, you only have to remove the new node from the inventories and delete the Clusterware home content from the new node.

Once the software is on the new node, you have to configure it. In order to proceed there are three scripts to run as root. The first one, orainsRoot.sh, sets up the correct privileges to the Oracle Inventory, and has to be run from the new node. Its location is given by the Oracle Installer when it is run in silent mode:

rac-server5# /u01/oraInventory/orainstRoot.sh

The second script to run has to be run from the node onto which you’ve added the Clusterware, rac-server1 in this example. This script is named rootaddnode.sh and it adds the node and the nodeapps resources to the OCR:

rac-server1# cd /u01/appcrs/install/rootaddnode.sh

Once the resource is created, the root.sh script has to be run on the new node. It will finish the configuration and start the Clusterware:

rac-server5# /u01/app/crs/root.sh

Unlike what happens during the initial installation if the public network is using a private address, there should not be any issue for the VIP, even if it’s on a private networks.

Adding the new node to the ONS configuration

In the case of the Clusterware, ONS configuration is kept in two different files. The ons.config file, located in the ORA_CRS_HOME/opmn/conf stores the local port the ONS binds to. The OCR stores the list of ONS servers each ONS should work with. The node addition doesn’t add the new ONS configuration in the OCR, so you have to add it manually. To do so, check the port the new ONS uses in its local ons.config and as root on one of the nodes, run the racgons command to add it to the OCR:

rac-server5# cat $ORA_CRS_HOME/opmn/conf/ons.config
rac-server5# cd $ORA_CRS_HOME/bin
rac-server5# ./racgons add_config rac-server5:6200

Adding the Database Software to the new node

Once the Clusterware installed and started on the new node, adding the database software is pretty easy and safe to execute. Just make sure you have read-write access to the directory the ORACLE_HOME will be added to (we’ll assume you install the software as oracle):

rac-server5# mkdir -p /usr/app/oracle
rac-server5# chown -R oracle:oinstall /usr/app/oracle

Connect as the software owner on any of the other nodes and run and run the addNode.sh command as below:

rac-server1$ export ORACLE_HOME=/usr/app/oracle/product/10.2.0/db_1
rac-server1$ cd $ORACLE_HOME/oui/bin
rac-server1$ ./addNode.sh -silent CLUSTER_NEW_NODES={rac-server5}

You can then execute the root.sh script as root on the new node:

rac-server1# cd /usr/app/oracle/product/10.2.0/db_1/
rac-server1# ./root.sh

Repeat the set of operations above as many times as necessary to install all the ORACLE_HOMEs you want on the new node. That’s especially true if you use a separate ORACLE_HOME for ASM.

Adding the listener to the new node

Even if it’s part of the nodeapps, the listener can only be added after you’ve installed the database software. Most of its configuration information is stored in the listener.ora file and is easy to change manually. You have, however, to update the associated resource in the cluster registry, and define wich ORACLE_HOME has to be used for that listener. With 10.2, the only way to do it is to use NETCA. You’ll find below the NETCA syntax to use in silent mode to add the listener to the new node. We’ll assume the listener from /u01/app/oracle/product/10.2.0/db_1:

rac-server1$ export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
rac-server1$ export PATH=$ORACLE_HOME/bin:$PATH
rac-server1$ export DISPLAY=:1
rac-server1$ netca /silent /responsefile \
             $ORACLE_HOME/network/install/netca_typ.rsp \
             /nodeinfo rac-server5

The NETCA syntax differs from the other configuration assistant syntaxes. It uses a “/” instead of a “-” for the parameter flags, and it requires the DISPLAY environment variable to be set with a dummy value, even if you don’t have any authorization to use that X display, and even if it doesn’t exist at all. The parameters to use are:

1. /silent to prevent NETCA from using a X display.
2. /responsefile $ORACLE_HOME/network/install/netca_typ.rsp contains the response file that comes with the ORACLE_HOME. You don’t have to modify it.
3. /nodeinfo is the list of nodes to be added.

Adding ASM to the new node

If you use ASM, you need to add the ASM instance to the new node before you add the instance to the database. To proceed, you copy the init.ora file from one of the other ASM instance to the new node. Assuming you use $ORACLE_HOME for your ASM instance, the commands should look like the ones below:

rac-server5$ cd $ORACLE_HOME/dbs
rac-server5$ scp rac-server1:$ORACLE_HOME/dbs/init+ASM1.ora
                 ./init+ASM5.ora

Edit the file and modify it according to your needs. (In many cases, you won’t have to change anything, but it’s worthwhile to check.) Then create the admin directories as defined in the file:

rac-server5$ mkdir -p /u01/app/oracle/admin/+ASM/bdump
rac-server5$ mkdir -p /u01/app/oracle/admin/+ASM/cdump
rac-server5$ mkdir -p /u01/app/oracle/admin/+ASM/udump
rac-server5$ mkdir -p /u01/app/oracle/admin/+ASM/adump

Create a password file for your new ASM instance:

rac-server5$ cd $ORACLE_HOME/dbs
rac-server5$ orapwd file=orapw+ASM5 password=x entries=5

Define the ASM associated resource in the OCR and start it:

rac-server5$ srvctl add asm -n rac-server5 \
                          -i +ASM5 \
                          -o $ORACLE_HOME
rac-server5$ srvctl enable asm -n rac-server5 -i +ASM5
rac-server5$ srvctl start asm -n rac-server5

Make sure the Disk Groups you need for your database are mounted in your new instance by querying V$ASM_DISKGROUP.

Adding An Instance to the New Node with DBCA

The easiest and fastest way to add an instance to a RAC is to use DBCA in silent mode. Unfortunately, it’s not always possible. For example, if the database has Standby Redo Logs, it will fail. The next section of this post digs into the manual addition of an instance. It will help you both to understand what DBCA is doing behind the scenes, and also how to recover after an error from dbca. However, if you can, add the instance with DBCA. The command to run should look like:

rac-server1$ dbca -silent -addInstance  \
  -gdbName ORCL                         \     
  -sysDBAPassword xxx                   \
  -nodelist rac-server5

• -addInstance is used to add an instance.
• -gdbName specifies the database name, as shownin the previous post.
• -sysDBAPassword specifies the SYSDBA (usually from the SYS user) password. It is mandatory.
• -nodelist lists the nodes to be added to the RAC database.

Manually Adding An Instance to the New Node

If you cannot or don’t want use DBCA to add an instance to your RAC database, you can always do it manually. Below is the list of steps involved.

Step 1: Add the instance prefix to the oratab file

echo "ORCL:/u01/app/oracle/product/10.2.0/db_1:N">>/etc/oratab

Step 2: Create the administration directory

You can locate these directories on the other servers and create them on the new server as below:

rac-server5$ mkdir -p /u01/app/oracle/admin/ORCL/adump
rac-server5$ mkdir -p /u01/app/oracle/admin/ORCL/bdump
rac-server5$ mkdir -p /u01/app/oracle/admin/ORCL/cdump
rac-server5$ mkdir -p /u01/app/oracle/admin/ORCL/udump

Step 3: Update the tnsnames.ora files from all the nodes

The easiest way to manage the network files is to have the exact same files on each of the servers. The entries you’d want to have in the tnsnames.ora file are:

1. LISTENER_<server_name> for each of the servers. This alias points to the VIP end of the listener from each of the servers, and is used in the local_listener parameter of each instance.
2. LISTENER_<gDbName> is an alias that points to all the listener VIP ends, and is used by the remote_listener parameter
3. <gDbName> is an alias that points to all the listeners to connect to the database.
4. <Instance_Name> is an alias that points to the local listener, and specify the instance_name parameter to force the connection to a specific instance.

Step 4: Check the instances register in the new node listener

Once you’ve tested the new tnsnames.ora and pushed it to all the servers, you have to force each one of the instances to reload it and to register again. The fastest way is to change the remote_listeners parameter value back and forth as below:

rac-server1$ sqlplus / as sysbda
SQL> alter system set remote_listeners='ORCL' sid='*';
SQL> alter system set remote_listeners='LISTENERS_ORCL' sid='*';
SQL> alter system register;

Once the tns aliases are reloaded, you can check in the new node that the instances are registered:

rac-server5$ lsnrctl status listener_rac-server5

Step 5: Add the Redo Logs for the new thread that will be used by the instance

The way to do this depends on whether you use Oracle Managed Files for the Redo Logs or if you name your files manually. To learn more about your redo logs, you can run the query below:

rac-server1$ sqlplus / as sysdba
SQL> col thread# format 99
SQL> col group# format 999
SQL> col bytes format 999,999,999,999
SQL> col member format a85
SQL> set lines 120
SQL> select l.thread#, f.group#, l.bytes, f.member 
       from v$logfile f, v$log l
      where f.group#=l.group#(+)
      order by f.group#;

Then you can create the log files for the new thread. Refer to the ALTER DATABASE documentation. Here is an example where OMF is being used:

SQL> alter database add logfile thread 5 group 13 size 500M;
SQL> alter database add logfile thread 5 group 14 size 500M;
SQL> alter database add logfile thread 5 group 15 size 500M;

Step 6: Activate the Thread you’ve created

SQL> alter database enable public thread 5;

Step 7: Create an UNDO tablespace for the new instance

The syntax will differ depending on whether or not you use OMF. For more details, refer to the CREATE TABLESPACE documentation. Below is an example in the case of OMF:

SQL> create undo tablespace UNDOTBS5 datafile size 500M;

Step 8: Modify the instance-specific parameters in the spfile

Some parameters depend on the instance. An easy way to see what specific parameter applies to an instance is to connect to it with SQL*Plus and run the query below:

SQL> col name format a30
SQL> col value format a80
SQL> set lines 120
SQL> select name, value 
       from v$spparameter 
      where sid in (select instance_name 
                      from v$instance);

Then you can add the parameters for the new instance. You should at least add undo_tablespace, thread, local_listener, and instance_number, as below:

alter system set instance_number=5 scope=spfile sid='ORCL5';
alter system set thread=5 scope=spfile sid='ORCL5';
alter system set undo_tablespace='UNDOTBS5' scope=spfile sid='ORCL5';
alter system set local_listener='LISTENER_RAC-SERVER5' scope=spfile sid='ORCL5';

Step 9: Create the instance init.ora file on the new server

This file should contain only the spfile parameter. You can simply copy it from another server, as below:

rac-server5$ cd $ORACLE_HOME/dbs
rac-server5$ scp rac-server1:$ORACLE_HOME/dbs/initORCL1.ora initORCL5.ora
rac-server5$ cat initORCL5.ora

Step 10: Create or Link to a Password File on the new node

If you have the opportunity to do it with OCFS, NFS, or any other solution, share the password file across all the instances. In that case just create a link from the new node to that password file with the ln command. If you cannot, use the orapwd command to create a password file in the local ORACLE_HOME.

Step 11: Register the instance in the Clusterware

Connect to any of the nodes and register the new instance as root with the srvctl command:

rac-server5# export ORACLE_HOME=/u01/app/oracle/product/10.2.0/db_1
rac-server5# cd $ORACLE_HOME/bin
rac-server5# ./srvctl add instance -d ORCL \
                -i ORCL5 -n rac-server5

Step 12: Start the new instance

Open a session to monitor the alert_ORCL5.log file with tail -f on rac-server5:

rac-server5$ cd /u01/app/oracle/admin/ORCL/bdump
rac-server5$ touch alert_ORCL5.log
rac-server5$ tail -f alert_ORCL5.log

From another session, start ORCL5:

srvctl start instance -d ORCL -i ORCL5

If you’ve done everything correctly, the instance will start. If it doesn’t start, check the alert.log content or if there is no alert.log, try to start the instance with SQL*Plus.

Step 13: Make sure you can connect to the new instance

This means you can connect to the new instance registers in all the listeners from a client. This assumes that (1) if you uses DNS aliases, the client knows about the new node VIP, and (2) if your application servers are in a DMZ or if there is a firewall in front of your database, you’ve updated the firewall ACL as needed.

Adding the services to the new node

The best way to connect to a RAC is to use services. If you’ve registered those services in the Clusterware, don’t forget to modify those according to the addition of your new node. Below is an example with the OLTP service that would be preferred on all the instances:

rac-server5$ srvctl modify service -d ORCL -s OLTP \
    -n -i "ORCL1,ORCL2,ORCL3,ORCL4,ORCL5"
rac-server5$ srvctl start service -d ORCL -s OLTP \
    -i ORCL5

Additional Notes

After a couple of tries, you’ll find that adding a node to a RAC cluster is pretty easy and fully dynamic. There are, however, two things that may make you think it’s not:

• The necessity of adding the new VIP to the client tns alias. In some situations, this may require a restart of the application. The fact is that the server-side load balancing will redirect the client new connections to the new node even if you don’t add the VIP to the tns alias.
• The need for the cluster_database_instances static parameter to match the number of instances running. Here again, that’s not true. You can have that parameter set to a value and have any number of instances — even more than the parameter indicates, and RAC will work. That parameter only helps tune memory usage, but if it were set to 4, it wouldn’t keep you from adding a fifth or even a 16th instance.

Any comments for now? Well, post 6 is on the way, and it will deal with with the last syntaxes to manage the 10.2 RAC configurations. Stay tuned!