How to a recreate disk group used by CRS
I will go through detailed steps here on how to recreate a disk group used by CRS. Some things to bear in mind: - I created this cluster based on another existing cluster, so I just followed the same patterns adopted in the existing one. - Oracle Grid Infrastructure version is 12.1.0.2. - SSDCRS disk group exists with External Redundancy and the intention here is to recreate it using Normal redundancy. - When creating this cluster, I used runInstaller on silent mode and there was no option to select Normal or High redundancy with response file or command line on Oracle Grid Infrastructure 12.1.0.2. So creating the cluster with External redundancy disk group for the cluster files and recreating the disk group afterward was the only option I found suitable. This blog post can be split into smaller sections and used to move ASM SPFile, ASM Password File, Voting files or Oracle Cluster Registry. Use it wisely.
Validating and preparing for the change
Important: Pay attention to the OS user you are running each command of this step by step. 0. Before performing any change in the cluster, save an OCR backup: [code] [root@rac-ora01 ~]# . oraenv >>> +ASM1 [root@rac-ora01 ~]# ocrconfig -manualbackup [root@rac-ora01 ~]# ocrconfig -showbackup [/code] 1. Check the current configuration of the OCR: [code] [root@rac-ora01 ~]# ocrcheck [/code] 2. Check the current configuration of Voting files: [code] [root@rac-ora01 ~]# crsctl query css votedisk [/code] 3. Check the current configuration of ASM password file and spfile: [code] [grid@rac-ora01 ~]$ . oraenv >>> +ASM1 [grid@rac-ora01 ~]$ srvctl config asm [grid@rac-ora01 ~]$ asmcmd spget [/code]
Starting the changes
4. Create the disk group SSDDATA with the proper configuration specified by the client: [code] SQL> CREATE DISKGROUP SSDDATA EXTERNAL REDUNDANCY DISK '/dev/mapper/lun_oradisk_SSDDATA0000' NAME SSDDATA0000, '/dev/mapper/lun_oradisk_SSDDATA0001' NAME SSDDATA0001, '/dev/mapper/lun_oradisk_SSDDATA0002' NAME SSDDATA0002, '/dev/mapper/lun_oradisk_SSDDATA0003' NAME SSDDATA0003, '/dev/mapper/lun_oradisk_SSDDATA0004' NAME SSDDATA0004, '/dev/mapper/lun_oradisk_SSDDATA0005' NAME SSDDATA0005, '/dev/mapper/lun_oradisk_SSDDATA0006' NAME SSDDATA0006, '/dev/mapper/lun_oradisk_SSDDATA0007' NAME SSDDATA0007, '/dev/mapper/lun_oradisk_SSDDATA0008' NAME SSDDATA0008, '/dev/mapper/lun_oradisk_SSDDATA0009' NAME SSDDATA0009, '/dev/mapper/lun_oradisk_SSDDATA0010' NAME SSDDATA0010, '/dev/mapper/lun_oradisk_SSDDATA0011' NAME SSDDATA0011, '/dev/mapper/lun_oradisk_SSDDATA0012' NAME SSDDATA0012, '/dev/mapper/lun_oradisk_SSDDATA0013' NAME SSDDATA0013, '/dev/mapper/lun_oradisk_SSDDATA0014' NAME SSDDATA0014, '/dev/mapper/lun_oradisk_SSDDATA0015' NAME SSDDATA0015 ATTRIBUTE 'AU_SIZE' = '4M', 'compatible.rdbms'='10.1', 'compatible.asm'='12.1.0.2'; [/code] 5. Relocate the OCR to the disk group SSDDATA: [code] [root@rac-ora01 ~]# . oraenv >>> +ASM1 [root@rac-ora01 ~]# ocrconfig -add +SSDDATA [root@rac-ora01 ~]# ocrcheck [root@rac-ora01 ~]# ocrconfig -delete +SSDCRS [/code] 6. Relocate Voting file to the disk group SSDDATA: [code] [root@rac-ora01 ~]# crsctl replace votedisk +SSDDATA [root@rac-ora01 ~]# crsctl query css votedisk [/code] 7. Relocate ASM password file to disk group SSDDATA: [code] [grid@rac-ora01 ~]$ asmcmd pwget --asm [grid@rac-ora01 ~]$ srvctl config asm -detail [grid@rac-ora01 ~]$ asmcmd pwmove --asm +SSDCRS/orapwASM +SSDDATA/orapwASM [/code] 8. Relocate spfile to the disk group +SSDDATA [code] [grid@rac-ora01 ~]$ . oraenv >>> +ASM1 [grid@rac-ora01 ~]$ sqlplus / as sysasm SQL> create pfile='/tmp/initasm.ora' from spfile; SQL> create spfile='+SSDDATA' from pfile='/tmp/initasm.ora'; [grid@rac-ora01 ~]$ $ORACLE_HOME/bin/gpnptool get -o- | xmllint --format - | grep -i spfile [/code] 9. Restart the cluster to see if it comes back with the new ASM parameter file: [code] [root@rac-ora01 ~]# . oraenv >>> +ASM1 [root@rac-ora01 ~]# crsctl stop cluster -all [root@rac-ora01 ~]# crsclt start cluster -all [/code] 10. Check the parameter file again by querying the GPnP profile: [code] [grid@rac-ora01 ~]$ $ORACLE_HOME/bin/gpnptool get -o- | xmllint --format - | grep -i spfile [/code] 11. Make sure everything that was on SSDCRS disk group is already in SSDDATA disk group (check everything again if needed) and proceed with dropping the existing SSDCRS disk group (the drop command will raise an error if there is any file inside the disk group, so after you ensure everything was moved or copied to SSDDATA disk group, include the clause "... INCLUDING CONTENTS" to the DROP command below): [code] SQL> DROP DISKGROUP SSDCRS; [/code] 12. Save a new OCR backup: [code] [root@rac-ora01 ~]# . oraenv >>> +ASM1 [root@rac-ora01 ~]# ocrconfig -manualbackup [root@rac-ora01 ~]# ocrconfig -showbackup [/code] 13. Recreate the SSDCRS disk group with high redundancy: [code] CREATE DISKGROUP SSDCRS NORMAL REDUNDANCY FAILGROUP SSDXCRS DISK '/dev/mapper/lun_oradisk_SSDXCRS0000' NAME SSDXCRS0000, '/dev/mapper/lun_oradisk_SSDXCRS0001' NAME SSDXCRS0001, '/dev/mapper/lun_oradisk_SSDXCRS0002' NAME SSDXCRS0002, '/dev/mapper/lun_oradisk_SSDXCRS0003' NAME SSDXCRS0003, '/dev/mapper/lun_oradisk_SSDXCRS0004' NAME SSDXCRS0004, '/dev/mapper/lun_oradisk_SSDXCRS0005' NAME SSDXCRS0005, '/dev/mapper/lun_oradisk_SSDXCRS0006' NAME SSDXCRS0006, '/dev/mapper/lun_oradisk_SSDXCRS0007' NAME SSDXCRS0007, '/dev/mapper/lun_oradisk_SSDXCRS0008' NAME SSDXCRS0008, '/dev/mapper/lun_oradisk_SSDXCRS0009' NAME SSDXCRS0009, '/dev/mapper/lun_oradisk_SSDXCRS0010' NAME SSDXCRS0010, '/dev/mapper/lun_oradisk_SSDXCRS0011' NAME SSDXCRS0011, '/dev/mapper/lun_oradisk_SSDXCRS0012' NAME SSDXCRS0012, '/dev/mapper/lun_oradisk_SSDXCRS0013' NAME SSDXCRS0013, '/dev/mapper/lun_oradisk_SSDXCRS0014' NAME SSDXCRS0014, '/dev/mapper/lun_oradisk_SSDXCRS0015' NAME SSDXCRS0015 FAILGROUP SSDYCRS DISK '/dev/mapper/lun_oradisk_SSDYCRS0040' NAME SSDYCRS0040, '/dev/mapper/lun_oradisk_SSDYCRS0041' NAME SSDYCRS0041, '/dev/mapper/lun_oradisk_SSDYCRS0042' NAME SSDYCRS0042, '/dev/mapper/lun_oradisk_SSDYCRS0043' NAME SSDYCRS0043, '/dev/mapper/lun_oradisk_SSDYCRS0044' NAME SSDYCRS0044, '/dev/mapper/lun_oradisk_SSDYCRS0045' NAME SSDYCRS0045, '/dev/mapper/lun_oradisk_SSDYCRS0046' NAME SSDYCRS0046, '/dev/mapper/lun_oradisk_SSDYCRS0047' NAME SSDYCRS0047, '/dev/mapper/lun_oradisk_SSDYCRS0048' NAME SSDYCRS0048, '/dev/mapper/lun_oradisk_SSDYCRS0049' NAME SSDYCRS0049, '/dev/mapper/lun_oradisk_SSDYCRS0050' NAME SSDYCRS0050, '/dev/mapper/lun_oradisk_SSDYCRS0051' NAME SSDYCRS0051, '/dev/mapper/lun_oradisk_SSDYCRS0052' NAME SSDYCRS0052, '/dev/mapper/lun_oradisk_SSDYCRS0053' NAME SSDYCRS0053, '/dev/mapper/lun_oradisk_SSDYCRS0054' NAME SSDYCRS0054, '/dev/mapper/lun_oradisk_SSDYCRS0055' NAME SSDYCRS0055 FAILGROUP SSDZCRS DISK '/dev/mapper/lun_oradisk_SSDZCRS0080' NAME SSDZCRS0080, '/dev/mapper/lun_oradisk_SSDZCRS0081' NAME SSDZCRS0081, '/dev/mapper/lun_oradisk_SSDZCRS0082' NAME SSDZCRS0082, '/dev/mapper/lun_oradisk_SSDZCRS0083' NAME SSDZCRS0083, '/dev/mapper/lun_oradisk_SSDZCRS0084' NAME SSDZCRS0084, '/dev/mapper/lun_oradisk_SSDZCRS0085' NAME SSDZCRS0085, '/dev/mapper/lun_oradisk_SSDZCRS0086' NAME SSDZCRS0086, '/dev/mapper/lun_oradisk_SSDZCRS0087' NAME SSDZCRS0087, '/dev/mapper/lun_oradisk_SSDZCRS0088' NAME SSDZCRS0088, '/dev/mapper/lun_oradisk_SSDZCRS0089' NAME SSDZCRS0089, '/dev/mapper/lun_oradisk_SSDZCRS0090' NAME SSDZCRS0090, '/dev/mapper/lun_oradisk_SSDZCRS0091' NAME SSDZCRS0091, '/dev/mapper/lun_oradisk_SSDZCRS0092' NAME SSDZCRS0092, '/dev/mapper/lun_oradisk_SSDZCRS0093' NAME SSDZCRS0093, '/dev/mapper/lun_oradisk_SSDZCRS0094' NAME SSDZCRS0094, '/dev/mapper/lun_oradisk_SSDZCRS0095' NAME SSDZCRS0095 ATTRIBUTE 'AU_SIZE' = '1M', 'compatible.rdbms'='12.1.0.2', 'compatible.asm'='12.1.0.2'; [/code] Now move everything back to the SSDCRS disk group. 14. Relocate the OCR to the disk group SSDCRS: [code] [root@rac-ora01 ~]# . oraenv >>> +ASM1 [root@rac-ora01 ~]# ocrconfig -add +SSDCRS [root@rac-ora01 ~]# ocrcheck [root@rac-ora01 ~]# ocrconfig -delete +SSDDATA [/code] 15. Relocate Voting file to the disk group SSDCRS: [code] [root@rac-ora01 ~]# crsctl replace votedisk +SSDCRS [root@rac-ora01 ~]# crsctl query css votedisk [/code] 16. Relocate ASM password file to disk group SSDCRS: [code] [grid@rac-ora01 ~]$ asmcmd pwget --asm [grid@rac-ora01 ~]$ srvctl config asm -detail [grid@rac-ora01 ~]$ asmcmd pwmove --asm +SSDDATA/orapwASM +SSDCRS/orapwASM [/code] 17. Relocate spfile to the disk group +SSDCRS: [code] [grid@rac-ora01 ~]$ . oraenv >>> +ASM1 [grid@rac-ora01 ~]$ sqlplus / as sysasm SQL> create pfile='/tmp/initasm.ora' from spfile; SQL> create spfile='+SSDCRS' from pfile='/tmp/initasm.ora'; [grid@rac-ora01 ~]$ $ORACLE_HOME/bin/gpnptool get -o- | xmllint --format - | grep -i spfile [/code] 18. Restart the cluster to see if it comes back with the new ASM parameter file: [code] [root@rac-ora01 ~]# . oraenv >>> +ASM1 [root@rac-ora01 ~]# crsctl stop cluster -all [root@rac-ora01 ~]# crsclt start cluster -all [/code]
Checking everything after the changes
19. Check the parameter file by querying again the GPnP profile: [code] [grid@rac-ora01 ~]$ $ORACLE_HOME/bin/gpnptool get -o- | xmllint --format - | grep -i spfile [/code] 20. After ensuring all files are in their correct places, save a new OCR backup: [code] [root@rac-ora01 ~]# . oraenv >>> +ASM1 [root@rac-ora01 ~]# ocrconfig -manualbackup [root@rac-ora01 ~]# ocrconfig -showbackup [/code] Fortunately, from 12cR2 onwards, there is an option to create the disk groups with proper redundancy using response files. I hope this will be useful for you and let me know if you have any issues.
"Franky works for Pythian as Senior Oracle Database Consultant. He has extensive knowledge in Oracle Exadata and High Availability technologies and in other databases like MySQL, Cassandra and SQL Server. He is always improving his skills focusing on researching Oracle performance and HA. Franky has been involved in some major implementations of multinode RAC in AIX, Linux and Exadata and multisite DataGuard environments. The guy is OCP 12c, OCE SQL, OCA 11g, OCS Linux 6 and RAC 12c and was nominated Oracle ACE in 2017. He is well known in the Brazilian community for his blog https://loredata.com.br/blog and for all the contribution he provides to the Oracle Brazilian community. Franky is also a frequent writer for OTN and speaker at some Oracle and database conferences around the world. Feel free to contact him in social media."
