Now that Oracle Grid Infrastructure (GI) 19c has been released, it is time to upgrade your GI to 18c!
I am presenting here a well-tested procedure to upgrade your Grid Infrastructure to 18c. It has been designed and applied on many Exadatas and the procedure can also be applied on non-Exadata systems. You will notice that it is very close to upgrade your Grid Infrastructure to 12c.
0/ Preparation
Here are a few things that are good to know and read before starting upgrading a GI to 18c:
- Download the GI 18c image from edelivery: V978971-01.zip.
- Download the RU you want to apply like Patch 28828717: GI RELEASE UPDATE 18.5.0.0.0; As I use to do this kind of maintenance when I patch the Exadata stack, I usually use the GI RU provided in the Exadata bundle like 28980183/Database/18.5.0.0.0/18.5.0.0.190115GIRU/28828717 from the Jan 19 Bundle. You can do either way, it is the same RU.
- GI 18c will be installed on /u01/app/18.1.0.0/grid.
- GI 12.1 is running from /u01/app/12.1.0.2/grid on the systems I work on.
- This procedure has been successfully applied on many Exadata’s half rack and full rack; it also applies to non-Exadata GI system.
- I use the rac-status.sh script to check the status of all the resources of my cluster before and after the maintenance to avoid any unpleasantness
- Check your oratab entries to avoid having them deleted during the upgrade as explained in this post
Versions naming:
It is not a secret to anyone, Oracle’s version naming has always been changed in a not-really-consistent manner across the years making it not really easy to follow. Then this new style of yearly release also came with its new version numbering. Indeed, we used to name our directories with the version number like 12.1.0.2 and then create a new directory like 12.2.0.1 when we would perform an out-of-place upgrade to a new major version.
Nowadays, even if 18c is actually version 12.2.0.2, the database came with an 18.1 version and GI came with an 18.0 version. Each RU will increment the second number of the version: January 19 RU (patch 28828717) being GI version 18.5, and so on.
I had a discussion with some friends about how to name the GI directory knowing that and also that nowadays, Oracle promotes the Rapid Home Provisionning tool for an “always out of place” patch strategy. We decided that we wouldn’t go with RHP nor an out-of-place patching for each RU but for every major release as we were doing before. It would then have made no sense to name our directory /u01/app/18.5.0.0/grid for GI upgrade to 18c + Jan 19 Bundle as April 19 Bundle would upgrade the version to 18.6; we then have decided to name our GI 18c directory with the original version number /u01/app/18.1.0.0/grid which makes it clear to everyone (18.0 looked weird to me, indeed, a “0” version doesn’t make much sense to me). Also, we kept four numbers in the naming to stay consistent with what we were doing before and with the databases homes.
And the day you need to double check what exact patch is installed here, we would rely on something like lspatches, which you would also do in any case, as one-off patches may have been installed on your homes and you would not have renamed the directory.
Having said that, you can name it as you wish :)
1/ Install GI 18c from the gold image
Let’s enjoy this super new feature and quickly install GI 18c from a gold image:
-- Create the target directories for GI 18c sudo su - dcli -g ~/dbs_group -l root "ls -ltr /u01/app/18.1.0.0/grid" dcli -g ~/dbs_group -l root mkdir -p /u01/app/18.1.0.0/grid dcli -g ~/dbs_group -l root chown grid:oinstall /u01/app/18.1.0.0/grid dcli -g ~/dbs_group -l root "ls -altr /u01/app/18.1.0.0/grid" -- Install GI using this gold image : /patches/V978971-01.zip sudo su - grid unzip -q /patches/V978971-01.zip -d /u01/app/18.1.0./grid
2/ Pre requisites
2.1/ Upgrade opatch
As usual, it is recommended to patch opatch before starting any patching activity. If you work with Exadata, you may have a look at this post where I show how to quickly upgrade opatch with dcli.
2.2/ ASM spfile and password file
Check that the ASM passwordfile and the ASM spfile are located under ASM to avoid issues during the upgrade:
[[email protected]]$ asmcmd spget +DATA/mycluster/ASMPARAMETERFILE/registry.253.909449003 [[email protected]]$ asmcmd pwget --asm +DATA/orapwASM [[email protected]]$
If you don’t, you may face the below error and ASM won’t restart after being upgraded:
Verifying Verify that the ASM instance was configured using an existing ASM parameter file. ...FAILED PRCT-1011 : Failed to run "asmcmd". Detailed error: ASMCMD-8001: diskgroup 'u01' does not exist or is not mounted
Please find a quick procedure to move the ASM passwordfile from a FileSystem to ASM:
[[email protected]]$ asmcmd pwcopy /u01/app/12.1.0.2/grid/dbs/orapw+ASM +DBFS_DG/orapwASM [[email protected]]$ asmcmd pwset --asm +DBFS_DG/orapwASM [[email protected]]$ asmcmd pwget --asm
2.3/ Prepare a responsefile such as this one
[[email protected]]$ egrep -v "^#|^$" /tmp/giresponse.rsp | head -10 oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v18.1.0 INVENTORY_LOCATION= oracle.install.option=UPGRADE ORACLE_BASE=/u01/app/oracle oracle.install.asm.OSDBA=dba oracle.install.asm.OSOPER=dba oracle.install.asm.OSASM=dba oracle.install.crs.config.gpnp.scanName= oracle.install.crs.config.gpnp.scanPort= oracle.install.crs.config.ClusterConfiguration= [[email protected]]$
2.4/ System prerequisites
Check these system prerequisites:
-- a 10240 limits for the "soft stack" (if not, set it, log off and log on) [[email protected]]# dcli -g ~/dbs_group -l root grep stack /etc/security/limits.conf | grep soft exadatadb01: * soft stack 10240 exadatadb02: * soft stack 10240 exadatadb03: * soft stack 10240 exadatadb04: * soft stack 10240 [[email protected]]# -- at least 1500 huge pages free [[email protected]]# dcli -g ~/dbs_group -l root grep -i huge /proc/meminfo .... AnonHugePages: 0 kB HugePages_Total: 200000 HugePages_Free: 132171 HugePages_Rsvd: 38338 HugePages_Surp: 0 Hugepagesize: 2048 kB .... [[email protected]]#
2.5/ Run the prerequisites
This step is very important and the logs need to be checked closely for any error:
[[email protected]]$ cd /u01/app/18.1.0.0/grid [[email protected]]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.1.0.2/grid -dest_crshome /u01/app/18.1.0.0/grid -dest_version 18.1.0.0 -fixup -verbose
Below an output which requires a fixup:
CVU operation performed: stage -pre crsinst
Date: Jun 18, 2019 9:47:15 PM
CVU home: /u01/app/18.1.0.0/grid/
User: grid
******************************************************************************************
Following is the list of fixable prerequisites selected to fix in this session
******************************************************************************************
-------------- --------------- ------------- -------------
Check failed. Failed on nodes Reboot Re-Login
required? required?
-------------- --------------- ------------- -------------
Group Membership: asmoper exadb01,exadb02 no no
Execute "/tmp/CVU_18.0.0.0.0_grid/runfixup.sh" as root user on nodes "exadb01,exadb02" to perform the fix up operations manually
Press ENTER key to continue after execution of "/tmp/CVU_18.0.0.0.0_grid/runfixup.sh" has completed on nodes "exadb01,exadb02"
Which I executed on both nodes (do not do that if your a fixup requires a reboot!):
[[email protected] ~]# /tmp/CVU_18.0.0.0.0_grid/runfixup.sh All Fix-up operations were completed successfully. [[email protected] ~]# ssh exadb02 [[email protected] ~]# /tmp/CVU_18.0.0.0.0_grid/runfixup.sh All Fix-up operations were completed successfully. [[email protected] ~]#
Once the fixup script has been executed, restart the prerequisites to be sure that everything is now good:
[[email protected]]$ ./runcluvfy.sh stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.1.0.2/grid -dest_crshome /u01/app/18.1.0.0/grid -dest_version 18.1.0.0 -fixup -verbose . . . Pre-check for cluster services setup was successful. CVU operation performed: stage -pre crsinst Date: Jun 18, 2019 9:53:40 PM CVU home: /u01/app/18.1.0.0/grid/ User: grid [[email protected]]$
3/ Upgrade to GI 18c
Now that all the prerequisites are successful, we can upgrade GI to 18c.
3.0/ A status before starting the upgrade
I strongly recommend keeping a status of all the resources across your cluster before starting the maintenance to avoid any unpleasantness after the maintenance.
[[email protected]]$ ./rac-status.sh -a -w0 | tee status_before_GI_upgrade_to_18 Cluster exadata is a X5-2 Elastic Rack HC 8TB Listener | Port | db01 | db02 | db03 | db04 | Type | ------------------------------------------------------------------------------------------------------------------- LISTENER | TCP:1551 | Online | Online | Online | Online | Listener | LISTENER_ABCD | TCP:1561 | Online | Online | Online | Online | Listener | LISTENER_SCAN1| TCP:1551,1561 | - | - | Online | - | SCAN | LISTENER_SCAN2| TCP:1551,1561 | - | Online | - | - | SCAN | LISTENER_SCAN3| TCP:1551,1561 | Online | - | - | - | SCAN | ------------------------------------------------------------------------------------------------------------------- DB | Service | db01 | db02 | db03 | db04 | ---------------------------------------------------------------------------------------------------- db01 | proddb_1_bkup | Online | - | - | - | | proddb_2_bkup | - | Online | - | - | | proddb_3_bkup | - | - | Online | - | | proddb_4_bkup | - | - | - | Online | db02 | db02svc1_bkup | - | - | Online | - | | db02svc2_bkup | - | - | Online | - | db03 | db03svc1_bkup | Online | - | - | - | | db03svc2_bkup | Online | - | - | - | db04 | db04svc1_bkup | Online | - | - | - | | db04svc2_bkup | - | Online | - | - | | db04svc3_bkup | - | - | Online | - | | db04svc4_bkup | - | - | - | Online | ---------------------------------------------------------------------------------------------------- DB | Version | db01 | db02 | db03 | db04 | DB Type | ------------------------------------------------------------------------------------------------------------------- db01 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) | db02 | 12.1.0.2 (1) | - | - | Open | Open | RAC (P) | db03 | 12.1.0.2 (1) | Open | Open | - | - | RAC (P) | db04 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) | ------------------------------------------------------------------------------------------------------------------- ORACLE_HOME references listed in the Version column : Primary : White and (P) Standby : Red and (S) 1 : /u01/app/oracle/product/12.1.0.2/dbhome_1 [[email protected]]$
3.1/ ASM memory setting
Some recommended memory settings have to be set at ASM instance level:
[[email protected]]$ sqlplus / as sysasm SQL> alter system set sga_max_size = 3G scope=spfile sid='*'; SQL> alter system set sga_target = 3G scope=spfile sid='*'; SQL> alter system set memory_target=0 sid='*' scope=spfile; SQL> alter system set memory_max_target=0 sid='*' scope=spfile /* required workaround */; SQL> alter system reset memory_max_target sid='*' scope=spfile; SQL> alter system set use_large_pages=true sid='*' scope=spfile /* 11.2.0.2 and later(Linux only) */;
3.2/ Reset miscount to default
The miscount parameter is the maximum time, in seconds, that a network heartbeat can be missed before a node eviction occurs. It needs to be reset to default before upgrading. It has to be done as the GI owner.
[[email protected]]$ . oraenv <<< +ASM1 [[email protected]]$ crsctl unset css misscount
3.3/ gridSetup.sh
We will be using a script named gridSetup.sh to initiate the GI upgrade to 18c
Please find below a whole output:
Please find below a whole output:
[[email protected]]$ cd /u01/app/18.1.0.0/grid [[email protected]]$ ./gridSetup.sh -silent -responseFile /tmp/giresponse.rsp -J-Doracle.install.mgmtDB=false -J-Doracle.install.crs.enableRemoteGIMR=false -applyRU /patches/28980183/Database/18.5.0.0.0/18.5.0.0.190115GIRU/28828717 Preparing the home to patch... Applying the patch /patches/28980183/Database/18.5.0.0.0/18.5.0.0.190115GIRU/28828717... Successfully applied the patch. The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2018-11-18_05-11-43PM/installerPatchActions_2018-11-18_05-11-43PM.log Launching Oracle Grid Infrastructure Setup Wizard... . . . You can find the log of this install session at: /u01/app/oraInventory/logs/GridSetupActions2018-11-18_05-11-43PM/gridSetupActions2018-11-18_05-11-43PM.log As a root user, execute the following script(s): 1. /u01/app/18.1.0.0/grid/rootupgrade.sh Execute /u01/app/18.1.0.0/grid/rootupgrade.sh on the following nodes: [exadb01, exadb02, exadb04, exadb03] Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node. Successfully Setup Software. As install user, execute the following command to complete the configuration. /u01/app/18.1.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/giresponse.rsp [-silent] [[email protected]]$
Above are some ignorable warnings about OS groups. Is also described the next step which is to start rootupgrade.sh on each node.
3.4/ rootupgrade.sh
As specified by gridSetup.sh in the previous step, we now need to run rootupgrade.sh on each node knowing that you can start rootupgrade.sh in parallel except for the first and the last node. Below is an example with a half rack (four nodes):
- Start rootupgrade.sh on the node 1
- Start rootupgrade.sh in parallel on the nodes 2 and 3
- Start rootupgrade.sh on the node 4
Note that CRS will be stopped on the node you apply rootupgrade.sh on so your instances will suffer an outage during this operation. Think about rebalancing your services accordingly to avoid any application downtime.
Here is a sample output; note that rootupgrade.sh is very silent, all logs go to the log file specified:
[[email protected]]# /u01/app/18.1.0.0/grid/rootupgrade.sh Check /u01/app/18.1.0.0/grid/install/root_exadb01_2019-06-29_11-14-09-408446489.log for the output of root script [[email protected]]#
An interesting thing to note here after a node is patched is that the softwareversion is now the target one (18.5 and not that here CRS shows 18.0.0.0.0) but the activeversion is still the old one (12.1). The activeversion will be changed to 18.5 when applying rootupgrade.sh on the last node.
[[email protected]]# . oraenv <<< +ASM1 [[email protected]]# crsctl query crs softwareversion Oracle Clusterware version on node [exadb01] is [18.0.0.0.0] [[email protected]]# crsctl query crs activeversion Oracle Clusterware active version on the cluster is [12.1.0.2.0] [[email protected]]#
3.5/ gridSetup.sh -executeConfigTools
Run the gridSetup.sh -executeConfigTools command:
[[email protected]]$ /u01/app/18.1.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/giresponse.rsp -silent Launching Oracle Grid Infrastructure Setup Wizard... You can find the logs of this session at: /u01/app/oraInventory/logs/GridSetupActions2018-11-18_07-11-22PM Successfully Configured Software. [[email protected]]$
3.6/ Check that GI is relinked with RDS
It is worth double checking that the new GI Home is properly relinked with RDS to avoid future performance issues (you may want to read this pdf for more information on what RDS is):
[[email protected]]$ dcli -g ~/dbs_group -l oracle /u01/app/18.1.0.0/grid/bin/skgxpinfo exadatadb01: rds exadatadb02: rds exadatadb03: rds exadatadb04: rds [[email protected]]$
If not, relink the GI Home with RDS:
dcli -g ~/dbs_group -l oracle "ORACLE_HOME=/u01/app/18.1.0.0/grid; make -C /u01/app/18.1.0.0/grid/rdbms/lib -f ins_rdbms.mk ipc_rds ioracle"
3.7/ Check the status of the cluster
Let’s have a look at the status of the cluster and the activeversion:
[[email protected]]$ /u01/app/18.1.0.1/grid/bin/crsctl check cluster -all ************************************************************** exadatadb01: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadatadb02: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadatadb03: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** exadatadb04: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************** [[email protected]]$ dcli -g ~/dbs_group -l oracle /u01/app/18.1.0.0/grid/bin/crsctl query crs activeversion Oracle Clusterware version on node [exadb01] is [18.0.0.0.0] Oracle Clusterware version on node [exadb02] is [18.0.0.0.0] Oracle Clusterware version on node [exadb03] is [18.0.0.0.0] Oracle Clusterware version on node [exadb04] is [18.0.0.0.0] [[email protected]]$
Let’s check the status of all the resources like we did in paragraph 3.0:
[[email protected]]$ ./rac-status.sh -a -w0 | tee status_after_GI_upgrade_to_18 Cluster exadata is a X5-2 Elastic Rack HC 8TB Listener | Port | db01 | db02 | db03 | db04 | Type | ------------------------------------------------------------------------------------------------------------------- LISTENER | TCP:1551 | Online | Online | Online | Online | Listener | LISTENER_ABCD | TCP:1561 | Online | Online | Online | Online | Listener | LISTENER_SCAN1| TCP:1551,1561 | - | Online | - | - | SCAN | LISTENER_SCAN2| TCP:1551,1561 | Online | - | - | - | SCAN | LISTENER_SCAN3| TCP:1551,1561 | - | - | Online | - | SCAN | ------------------------------------------------------------------------------------------------------------------- DB | Service | db01 | db02 | db03 | db04 | ---------------------------------------------------------------------------------------------------- db01 | proddb_1_bkup | Online | - | - | - | | proddb_2_bkup | - | Online | - | - | | proddb_3_bkup | - | - | Online | - | | proddb_4_bkup | - | - | - | Online | db02 | db02svc1_bkup | - | - | Online | - | | db02svc2_bkup | - | - | Online | - | db03 | db03svc1_bkup | Online | - | - | - | | db03svc2_bkup | Online | - | - | - | db04 | db04svc1_bkup | Online | - | - | - | | db04svc2_bkup | - | Online | - | - | | db04svc3_bkup | - | - | Online | - | | db04svc4_bkup | - | - | - | Online | ---------------------------------------------------------------------------------------------------- DB | Version | db01 | db02 | db03 | db04 | DB Type | ------------------------------------------------------------------------------------------------------------------- db01 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) | db02 | 12.1.0.2 (1) | - | - | Open | Open | RAC (P) | db03 | 12.1.0.2 (1) | Open | Open | - | - | RAC (P) | db04 | 12.1.0.2 (1) | Readonly | Readonly | Readonly | Readonly | RAC (S) | ------------------------------------------------------------------------------------------------------------------- ORACLE_HOME references listed in the Version column : Primary : White and (P) Standby : Red and (S) 1 : /u01/app/oracle/product/12.1.0.2/dbhome_1 [[email protected]]$
And check for differences:
[[email protected]]$ diff status_before_GI_upgrade_to_12.2 status_after_GI_upgrade_to_12.2 8,10c8,10 < LISTENER_SCAN1| TCP:1551,1561 | - | - | Online | - | SCAN | < LISTENER_SCAN2| TCP:1551,1561 | - | Online | - | - | SCAN | < LISTENER_SCAN3| TCP:1551,1561 | Online | - | - | - | SCAN | --- > LISTENER_SCAN1| TCP:1551,1561 | - | Online | - | - | SCAN | > LISTENER_SCAN2| TCP:1551,1561 | Online | - | - | - | SCAN | > LISTENER_SCAN3| TCP:1551,1561 | - | - | Online | - | SCAN | [[email protected]]$
We can see here than only the SCAN listeners have been re shuffled by the maintenance which does not matter. You can relocate them but it has no impact whatsoever. It also means that all our instances and services are back as they were before the maintenance. We are then idempotent.
3.8/ Set Flex ASM Cardinality to “ALL”
Starting release 12.2 ASM will be configured as “Flex ASM”. By default, Flex ASM cardinality is set to 3. This means configurations with four or more database nodes in the cluster might only see ASM instances on three nodes. Nodes without an ASM instance running on it will use an ASM instance on a remote node within the cluster. Only when the cardinality is set to “ALL”, will ASM bring up the additional instances required to fulfill the cardinality setting.
[[email protected]]$ srvctl modify asm -count ALL [[email protected]]$
Note that this command provides no output.
3.9/ Update compatible.asm to 18.1
Now that ASM 18c is running, updating the compatible.asm to 18.1 is recommended to be able to enjoy the new features of 18c.
-- Set env and connect [[email protected]]$ . oraenv <<< +ASM1 [[email protected]]$ sqlplus / as sysasm -- List the diskgroups SQL> select name, COMPATIBILITY from v$asm_diskgroup ; -- Set compatible to 12.2 (examples here with some usual DGs) SQL> ALTER DISKGROUP DATA SET ATTRIBUTE 'compatible.asm' = '18.1.0.0.0'; SQL> ALTER DISKGROUP DBFS_DG SET ATTRIBUTE 'compatible.asm' = '18.1.0.0.0'; SQL> ALTER DISKGROUP RECO SET ATTRIBUTE 'compatible.asm' = '18.1.0.0.0'; -- Verify the new settings SQL> select name, COMPATIBILITY from v$asm_diskgroup ;
3.10/ Update the Inventory
To wrap this up, let’s update the Inventory
[[email protected]]$ . oraenv <<< +ASM1 [[email protected]]$ /u01/app/12.2.0.1/grid/oui/bin/runInstaller -ignoreSysPrereqs -updateNodeList ORACLE_HOME=/u01/app/18.1.0.0/grid "CLUSTER_NODES={exadb01,exadb02,exadb03,exadb04}" CRS=true LOCAL_NODE=exadb01
Note: You may also want to update the new GI Home patch in OEM or any other monitoring tool that would require it.
3.11/ /etc/oratab entries
If you have oratab entries that have disappeared after the upgrade, you may have missed the warning in the 0/ Preparation paragraph of this post. You may want to have a look at this post for an explanation of this behavior.
And you’re all done! Enjoy!
No comments