After weeks of not having the time I wanted for this, it’s finally done! Today, I installed RAC on Oracle Enterprise Linux 5 (OEL5), and I can tell you that there’s nothing exceptional about the process.
The only trouble I encountered had nothing directly to do with the installation. In order for the device rights to be assigned when RHEL5 or OEL5 start up, you have to create a file in
/etc/udev/rules.d, but I have already spoken about that in my last post on the subject of raw devices.
Besides that, it’s all quite simple once 10g’s prerequisites are met. After an hour and a half, it was all wrapped up, at least for two nodes. I didn’t see anything revolutionary differences, but nonetheless there are some points worth mentioning.
1. There’s no need to launch VIPCA manually if your public network uses reserved IP addresses. Nonetheless, it’s quite alright to do so, because beneath 80% of examples, Oracle networks are in the
10.x.x.x ranges. Oracle has also fixed the Cluster Verify Utility, which now simply returns a warning instead of an error when this happens:
Interfaces found on subnet "192.168.245.0" that are likely candidates for a private interconnect: sacha eth1:192.168.245.2 WARNING: Could not find a suitable set of interfaces for VIPs.
On the other hand, note that you must have a ping-able GATEWAY at your public address and that the CVU doesn’t seem to validate it. If you don’t have one, the penalty is that your VIP won’t start.
2. No more RAW DEVICES are needed. As I have mentioned before, OCR and Voting Disks can be directly in the partitions (in fact, that’s even the recommended configuration now). It’s the same for ASM disks — for example, you can just change the parameter indicating the path under which ASM will discover disks (
/dev/sdb*, and with DBCA, you can directly adjust your block devices without ASM.
3. There is a command for verifying the state of all the cluster’s nodes.
./crsctl check cluster sacha ONLINE kilian ONLINE
4. There is a command for making a manual backup of OCR. It’s practical to do so before adding a node to the cluster.
./ocrconfig -manualbackup sacha 2007/09/16 08:09:13 /u01/crs/cdata/arkz/backup_20070916_080913.ocr ./ocrconfig -showbackup manual sacha 2007/09/16 08:09:13 /u01/crs/cdata/arkz/backup_20070916_080913.ocr
5. OPROCD replaces hangcheck-timer. I’ve written about this before, and verified it.
ps -ef |grep oproc root 5580 4625 0 07:38 ? 00:00:00 /bin/sh/etc/init.d/init.cssd oprocd root 6142 5580 0 07:38 ? 00:00:00 /u01/crs/bin/oprocd run -t 1000 -m 500 lsmod |grep hangcheck
6. ASMCMD now has a
cp command. This makes it possible to do things such as the working examples below.
. oraenv ORACLE_SID = [oracle] ? +ASM1 The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u01/app/oracle asmcmd ASMCMD> cd DATA/ORCL/CONTROLFILE ASMCMD> pwd +DATA/ORCL/CONTROLFILE ASMCMD> cp Current.260.633428899 +DATA/ORCL/CONTROLFILE/gark.ctl source +DATA/ORCL/CONTROLFILE/Current.260.633428899 target +DATA/ORCL/CONTROLFILE/Gark.ctl copying file(s)... file, +DATA/orcl/controlfile/gark.ctl, copy committed. ASMCMD> ls Current.260.633428899 gark.ctl
lsdsk command allows you to see disks associated with ASM. For example, with the
-d option, you can see which disks belong to a DiskGroup.
oraenv ORACLE_SID = [oracle] ? +ASM1 The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.1.0/db_1 is /u01/app/oracle asmcmd ASMCMD> lsdsk -k -d DATA Total_MB Free_MB OS_MB Name Failgroup Path 1529 864 1529 DATA_0000 DATA_0000 /dev/sdb5 2337 1306 2337 DATA_0001 DATA_0001 /dev/sdb7
A few remarks:
- It’s possible to launch ASMCMD when ASM is stopped. The message that follows,
ASMCMD-08103: failed to connect to ASM; ASMCMD running in non-connected mode, indicates that you will work directly on the heads of the disks.
- For reasons I imagine are linked to the fact that in my case, the
asm_diskstringparameter does not have the default value (
lsdskseems not to work when in non-connected mode.
- ASMCMD’s new commands look very interesting, particularly
remap. Okay, enough ASM stuff for now.
8. You can both add and remove Voting Disks in a working cluster. Let’s start with OCR, which seems to work already. As the root user, you can add and remove an OCR Mirror while the clusterware is working as follows:
cd /u01/crs/bin ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 248724 Used space (kbytes) : 2252 Available space (kbytes) : 246472 ID : 433771587 Device/File Name : /dev/sdb8 Device/File integrity check succeeded Device/File not configured Cluster registry integrity check succeeded ./ocrconfig -replace ocrmirror /dev/sdb9 ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 248724 Used space (kbytes) : 2252 Available space (kbytes) : 246472 ID : 433771587 Device/File Name : /dev/sdb8 Device/File integrity check succeeded Device/File Name : /dev/sdb9 Device/File integrity check succeeded Cluster registry integrity check succeeded ./ocrconfig -replace ocrmirror ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 248724 Used space (kbytes) : 2252 Available space (kbytes) : 246472 ID : 433771587 Device/File Name : /dev/sdb8 Device/File integrity check succeeded Device/File not configured Cluster registry integrity check succeeded
You can do the same thing with Voting Disk while the clusterware is as:
dd if=/dev/sdb10 of=/u01/crs/cdata/arkz/voting.copy bs=4k 62244+0 records in 62244+0 records out 254951424 bytes (255 MB) copied, 10.0618 seconds, 24.2 MB/s ./crsctl query css votedisk 0. 0 /dev/sdb10 Located 1 voting disk(s). ./crsctl add css votedisk /dev/sdb11 Successful addition of voting disk /dev/sdb11. ./crsctl add css votedisk /dev/sdb12 Successful addition of voting disk /dev/sdb12. ./crsctl query css votedisk 0. 0 /dev/sdb10 1. 0 /dev/sdb11 2. 0 /dev/sdb12 Located 3 voting disk(s). # At first glance, it's looks like it's not possible to remove Voting Disk 1: ./crsctl delete css votedisk /dev/sdb10 Failure 8 with Cluster Synchronization Services while deleting voting disk. # Too bad! But we can easily remove the other Voting Disks # and logically, a problem with Voting Disk 1 would not # affect the cluster. # Too bad again! I can't test this with my configuration. ./crsctl delete css votedisk /dev/sdb11 Successful deletion of voting disk /dev/sdb11. ./crsctl delete css votedisk /dev/sdb12 Successful deletion of voting disk /dev/sdb12. ./crsctl query css votedisk 0. 0 /dev/sdb10 Located 1 voting disk(s).
9. You can kill sessions cluster-wide.
# Session 1 sqlplus / as sysdba grant select on gv_$session to scott; # Session 2 sqlplus scott/tiger select sid, serial#, inst_id from gv$session where audsid=sys_context('USERENV','SESSIONID'); SID SERIAL# INST_ID --- ---------- ---------- 67 610 1 # Session 1 alter system kill session '67, 610, @1' immediate; # Session 2 / select sid, serial#, inst_id * ERROR at line 1: ORA-03135: connection lost contact Process ID: 29700 Session ID: 67 Serial number: 610
10. You can launch an AWR script for the whole RAC instead of instance-by-instance. Use
spawrrac.sql as follows:
@?/rdbms/admin/spawrrac Instances in this AWR schema ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Instance DB Id DB Name Count ----------- ------------ -------- 1161209635 ORCL 2 Enter value for dbid: 1161209635 Using 1161209635 for database Id Specify the number of days of snapshots to choose from ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Entering the number of days (n) will result in the most recent (n) days of snapshots being listed. Pressing without specifying a number lists all completed snapshots. Listing the last 31 days of Completed Snapshots Snap Instance DB Name Snap Id End Interval Time Level Count ------------ --------- ----------------- ----- -------- ORCL 1 16 Sep 2007 10:00 1 2 2 16 Sep 2007 11:00 1 2 3 16 Sep 2007 12:00 1 2 Specify the Begin and End Snapshot Ids ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Enter value for begin_snap: 1 Begin Snapshot Id specified: 1 Enter value for end_snap: 2 End Snapshot Id specified: 2 Specify the Report Name ~~~~~~~~~~~~~~~~~~~~~~~ The default report file name is spawrrac_1_2. To use this name, press to continue, otherwise enter an alternative. Enter value for report_name: [...] End of Report ( spawrrac_1_2.lst )
Could you point me to the previous posts regarding etc/udev/rules.d and the fact that no RAW devices are needed anymore.
Thank you for the information in this post
Lets start with the bad news : My original post about raw devices is in French !
What it says is that with RHEL5/OEL5, the /etc/udev/permissions.d directory doesn’t exist anymore and you have to create a file under /etc/udev/rules.d to set the permissions on boot ; you should be able to get the content of the file from the post even if it’s in French !
Regarding Oracle recommendation about “raw vs block” devices you’ll find a couple of informations in different 11g docs :
1- (About 10g first) This section explains that starting with clusterware 10.2 and RHEL4, you can use block (instead of raw) devices to store the OCR and the voting disk.
2- (11g) In this section, the doc says that Oracle recommends using block rather that raw devices on Linux. BTW, this section explains how to configure block devices while storing one file per block device (As you use to do with raw devices).
3- (10g and 11g) This section explains that Oracle recommends using block devices with ASMLib to create ASM disks. This OTN page contains the supported release of ASMLib, including the one for RHEL4 and RHEL5. Well (Shame on me !) this is not what I did ! I’ve changed the asm_diskstring parameter and I’ve bound the ASM DISKS to the block devices. I probably shouldn’t ! I’m pretty sure the problem I’ve encountered with lsdsk will be solved by doing the setup this way : I’ll check soon !
Anyway 1 and 3 show that you don’t need the raw devices anymore (even with 10g and RHEL4). But it doesn’t prevent you to set the permission via the “udev” dynamic device naming.
[…] no permissions.d directory on RHEL5 and OEL5. Last September, GrÃ©gory Guillou from Pythian blogged about installing on Red Hat 5 and referenced a post on his French blog that shows how to setup a RULES file for SCSI block […]
sorry, i don’t understand one thing: can i use directly /dev/sdb* for asm disks with 10gR2? it seems to me that also in 11g documentation Oracle says to use ASMLib but i can’t find ASMLib very useful. Morehover i’ve met problem described in Metalink Note 457369.1 that suggest to use asm_diskstring = /dev/oracleasm/disks/* but devices files under /dev/oracleasm/disks/* have same major e minor numbers of original block devices (/dev/sdb*) so i think that is the same think. So my question is: does oracle support the use of asm_diskstring=/dev/sdb* ?
Compliments for the post.
the answer with Linux is :
a- prefer using ASMLib. In particular is enables to discover the disks on all nodes without partprobe and it is optimized for ASM.
b- If you cannot, use the block device, i.e /dev/sdb* and NOT raw devices
using /dev/sd* devices for ocr and voting disk, i’m getting a “the location /dev/sdb1, entered for the oracle cluster registry (ocr) is not shared across all the nodes in the cluster….”.
This post also https://forums.oracle.com/forums/thread.jspa?threadID=641837&tstart=0&messageID=2474588 says that, for ocr and voting disk, 10g keep using raw dev or cluster fs.
It is correct?
you will get the error “the location /dev/sdb1, entered for the oracle cluster registry (ocr) is not shared across all the nodes in the cluster….”. when you have not shared the device /dev/sdb1 for both nodes. Please check its physical location it should be shared in shared path.
Great Article .. Keep it up
related to your statement that oprocd replaces hangcheck-timer … afaik and according to oracle doc this is not true both for 10g and 11g.
Can you pointed me to the link where this is stated?
Hi, i was getting an error “ocr location is inaccessible” while running root.sh. This is what i did
a. created raw devices, and formated it with fdisk(as specified in one rac installation doc’ but actually i am not sure why they are asking us to partition the disk as it will become ext3 partitioned and not raw devices)
b. granted the required permisssions ocr root:oinstall and voting disk oracle:oinstall
c. but not able to access ocr, should i use block devices else format the ocr and voting disk using ocfs. your advice is greatly appreciated