While Upgrading from GI 12c to 19c, +asm1(19c) Cannot Detect +asm2(12c)
A few weeks ago, I was working on a GI upgrade from 12.2 to 19.6. After running the upgrade on Node 1, an issue occurred where +asm1(19c) could not detect +asm2(12c). The information below describes the actions I performed to resolve the problem.
First, I ran the pre-checks. Everything passed successfully.
[grid@vxe-dev-rac-01 bin]$ ./cluvfy stage -pre crsinst -upgrade -rolling -src_crshome /u01/app/12.2.0.1/grid -dest_crshome /u01/app/19.6.0.0/grid -dest_version 19.1.0.0.0 -fixupnoexec -verbose
...
Node Name Required Running? Comment
------------ ------------------------ ------------------------ ----------
reneace02 no yes passed
reneace01 no yes passed
Verifying Node Application Existence ...PASSED
Verifying Check incorrectly sized ASM Disks ...PASSED
Verifying ASM disk group free space ...PASSED
Verifying Network configuration consistency checks ...PASSED
Verifying File system mount options for path GI_HOME ...PASSED
Verifying /boot mount ...PASSED
Verifying OLR Integrity ...PASSED
Verifying Verify that the ASM instance was configured using an existing ASM parameter file. ...PASSED
Verifying User Equivalence ...PASSED
Verifying RPM Package Manager database ...INFORMATION (PRVG-11250)
Verifying Network interface bonding status of private interconnect network interfaces ...PASSED
Verifying /dev/shm mounted as temporary file system ...PASSED
Verifying File system mount options for path /var ...PASSED
Verifying DefaultTasksMax parameter ...PASSED
Verifying zeroconf check ...PASSED
Verifying ASM Filter Driver configuration ...PASSED
Verifying Systemd login manager IPC parameter ...PASSED
Verifying Kernel retpoline support ...PASSED
Pre-check for cluster services setup was successful.
Verifying RPM Package Manager database ...INFORMATION
PRVG-11250 : The check "RPM Package Manager database" was not performed because
it needs 'root' user privileges.
CVU operation performed: stage -pre crsinst
Date: Feb 20, 2020 11:28:43 AM
CVU home: /u01/temp/19.6.0.0/cvu/bin/../
User: gridThen, I created the following response file so that I could run the GI upgrade in silent mode:
[grid@reneace01 grid]$ cat /tmp/gridresponse.rsp | grep -v "#"
oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v19.0.0
INVENTORY_LOCATION=/u01/app/oraInventory
oracle.install.option=UPGRADE
ORACLE_BASE=/u01/app/grid
oracle.install.asm.OSDBA=
oracle.install.asm.OSOPER=
oracle.install.asm.OSASM=
oracle.install.crs.config.scanType=LOCAL_SCAN
oracle.install.crs.config.SCANClientDataFile=
oracle.install.crs.config.gpnp.scanName=
oracle.install.crs.config.gpnp.scanPort=
oracle.install.crs.config.ClusterConfiguration=STANDALONE
oracle.install.crs.config.configureAsExtendedCluster=false
oracle.install.crs.config.memberClusterManifestFile=
oracle.install.crs.config.clusterName=vxe-dev-cluster
oracle.install.crs.config.gpnp.configureGNS=false
oracle.install.crs.config.autoConfigureClusterNodeVIP=false
oracle.install.crs.config.gpnp.gnsOption=CREATE_NEW_GNS
oracle.install.crs.config.gpnp.gnsClientDataFile=
oracle.install.crs.config.gpnp.gnsSubDomain=
oracle.install.crs.config.gpnp.gnsVIPAddress=
oracle.install.crs.config.sites=
oracle.install.crs.config.clusterNodes=reneace01:reneace01-vip,reneace02:reneace02-vip
oracle.install.crs.config.networkInterfaceList=
oracle.install.crs.configureGIMR=false
oracle.install.asm.configureGIMRDataDG=false
oracle.install.crs.config.storageOption=
oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=
oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=
oracle.install.crs.config.useIPMI=false
oracle.install.crs.config.ipmi.bmcUsername=
oracle.install.crs.config.ipmi.bmcPassword=
oracle.install.asm.SYSASMPassword=
oracle.install.asm.diskGroup.name=ORACRS
oracle.install.asm.diskGroup.redundancy=
oracle.install.asm.diskGroup.AUSize=
oracle.install.asm.diskGroup.FailureGroups=
oracle.install.asm.diskGroup.disksWithFailureGroupNames=
oracle.install.asm.diskGroup.disks=
oracle.install.asm.diskGroup.quorumFailureGroupNames=
oracle.install.asm.diskGroup.diskDiscoveryString=
oracle.install.asm.monitorPassword=
oracle.install.asm.gimrDG.name=
oracle.install.asm.gimrDG.redundancy=
oracle.install.asm.gimrDG.AUSize=4
oracle.install.asm.gimrDG.FailureGroups=
oracle.install.asm.gimrDG.disksWithFailureGroupNames=
oracle.install.asm.gimrDG.disks=
oracle.install.asm.gimrDG.quorumFailureGroupNames=
oracle.install.asm.configureAFD=false
oracle.install.crs.configureRHPS=false
oracle.install.crs.config.ignoreDownNodes=false
oracle.install.config.managementOption=NONE
oracle.install.config.omsHost=
oracle.install.config.omsPort=
oracle.install.config.emAdminUser=
oracle.install.config.emAdminPassword=
oracle.install.crs.rootconfig.executeRootScript=false
oracle.install.crs.rootconfig.configMethod=
oracle.install.crs.rootconfig.sudoPath=
oracle.install.crs.rootconfig.sudoUserName=
oracle.install.crs.config.batchinfo=
oracle.install.crs.app.applicationAddress=
oracle.install.crs.deleteNode.nodes=I then executed the upgrade. As shown below, there were no failures after executing the rootupgrade.sh script:
[grid@reneace01 grid]$ pwd
/u01/app/19.6.0.0/grid
[grid@reneace01 grid]$ ./gridSetup.sh -silent -responseFile /tmp/gridresponse.rsp -applyRU /u06/patches/oracle19c/30501910
Preparing the home to patch...
Applying the patch /u06/patches/oracle19c/30501910...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2020-02-26_07-27-27PM/installerPatchActions_2020-02-26_07-27-27PM.log
Launching Oracle Grid Infrastructure Setup Wizard...
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2020-02-26_07-27-27PM/gridSetupActions2020-02-26_07-27-27PM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2020-02-26_07-27-27PM/gridSetupActions2020-02-26_07-27-27PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
/u01/app/19.6.0.0/grid/install/response/grid_2020-02-26_07-27-27PM.rsp
You can find the log of this install session at:
/u01/app/oraInventory/logs/GridSetupActions2020-02-26_07-27-27PM/gridSetupActions2020-02-26_07-27-27PM.log
As a root user, execute the following script(s):
1. /u01/app/19.6.0.0/grid/rootupgrade.sh
Execute /u01/app/19.6.0.0/grid/rootupgrade.sh on the following nodes:
[reneace01, reneace02]
Run the script on the local node first. After successful completion, you can start the script in parallel on all other nodes, except a node you designate as the last node. When all the nodes except the last node are done successfully, run the script on the last node.
Successfully Setup Software with warning(s).
As install user, execute the following command to complete the configuration.
/u01/app/19.6.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/gridresponse.rsp [-silent]
[grid@reneace01 GridSetupActions2020-02-26_07-27-27PM]$ tail -f /u01/app/19.6.0.0/grid/install/root_reneace01.network_2020-02-26_20-09-57-369528654.log
ORACLE_HOME= /u01/app/19.6.0.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.6.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/reneace01/crsconfig/rootcrs_reneace01_2020-02-26_08-10-30PM.log
2020/02/26 20:11:02 CLSRSC-595: Executing upgrade step 1 of 18: 'UpgradeTFA'.
2020/02/26 20:11:02 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2020/02/26 20:11:02 CLSRSC-595: Executing upgrade step 2 of 18: 'ValidateEnv'.
2020/02/26 20:11:09 CLSRSC-595: Executing upgrade step 3 of 18: 'GetOldConfig'.
2020/02/26 20:11:09 CLSRSC-464: Starting retrieval of the cluster configuration data
2020/02/26 20:11:46 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2020/02/26 20:14:49 CLSRSC-692: Checking whether CRS entities are ready for upgrade. This operation may take a few minutes.
2020/02/26 20:17:53 CLSRSC-693: CRS entities validation completed successfully.
2020/02/26 20:18:02 CLSRSC-515: Starting OCR manual backup.
2020/02/26 20:18:18 CLSRSC-516: OCR manual backup successful.
2020/02/26 20:18:28 CLSRSC-486:
At this stage of upgrade, the OCR has changed.
Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.
2020/02/26 20:18:28 CLSRSC-541:
To downgrade the cluster:
1. All nodes that have been upgraded must be downgraded.
2020/02/26 20:18:28 CLSRSC-542:
2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.
2020/02/26 20:18:38 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2020/02/26 20:18:39 CLSRSC-595: Executing upgrade step 4 of 18: 'GenSiteGUIDs'.
2020/02/26 20:18:40 CLSRSC-595: Executing upgrade step 5 of 18: 'UpgPrechecks'.
2020/02/26 20:18:46 CLSRSC-363: User ignored prerequisites during installation
2020/02/26 20:19:02 CLSRSC-595: Executing upgrade step 6 of 18: 'SetupOSD'.
2020/02/26 20:19:02 CLSRSC-595: Executing upgrade step 7 of 18: 'PreUpgrade'.
2020/02/26 20:22:26 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode
2020/02/26 20:22:26 CLSRSC-482: Running command: '/u01/app/12.2.0.1/grid/bin/crsctl start rollingupgrade 19.0.0.0.0'
CRS-1131: The cluster was successfully set to rolling upgrade mode.
2020/02/26 20:22:31 CLSRSC-482: Running command: '/u01/app/19.6.0.0/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/12.2.0.1/grid -oldCRSVersion 12.2.0.1.0 -firstNode true -startRolling false '
ASM configuration upgraded in local node successfully.
2020/02/26 20:25:27 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2020/02/26 20:25:34 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2020/02/26 20:26:08 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
2020/02/26 20:26:11 CLSRSC-595: Executing upgrade step 8 of 18: 'CheckCRSConfig'.
2020/02/26 20:26:12 CLSRSC-595: Executing upgrade step 9 of 18: 'UpgradeOLR'.
2020/02/26 20:26:30 CLSRSC-595: Executing upgrade step 10 of 18: 'ConfigCHMOS'.
2020/02/26 20:26:30 CLSRSC-595: Executing upgrade step 11 of 18: 'UpgradeAFD'.
2020/02/26 20:26:41 CLSRSC-595: Executing upgrade step 12 of 18: 'createOHASD'.
2020/02/26 20:26:51 CLSRSC-595: Executing upgrade step 13 of 18: 'ConfigOHASD'.
2020/02/26 20:26:51 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
2020/02/26 20:29:52 CLSRSC-595: Executing upgrade step 14 of 18: 'InstallACFS'.
2020/02/26 20:33:58 CLSRSC-595: Executing upgrade step 15 of 18: 'InstallKA'.
2020/02/26 20:34:07 CLSRSC-595: Executing upgrade step 16 of 18: 'UpgradeCluster'.
2020/02/26 21:22:11 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2020/02/26 21:22:36 CLSRSC-595: Executing upgrade step 17 of 18: 'UpgradeNode'.
2020/02/26 21:22:41 CLSRSC-474: Initiating upgrade of resource types
2020/02/26 22:07:33 CLSRSC-475: Upgrade of resource types successfully initiated.
2020/02/26 22:07:48 CLSRSC-595: Executing upgrade step 18 of 18: 'PostUpgrade'.
2020/02/26 22:07:58 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeededHowever, my team and I noticed that the +asm1 instance was not detecting the +asm2 instance. While troubleshooting the issue, we discovered that Node 1 and Node 2 had different netmasks:
# Node 1
eth1:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.13.192 netmask 255.255.224.0 broadcast 169.254.31.255
ether 00:50:56:8e:21:77 txqueuelen 1000 (Ethernet)
# Node 2
eth1:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 169.254.60.115 netmask 255.255.0.0 broadcast 169.254.255.255
ether 00:50:56:8e:b4:4a txqueuelen 1000 (Ethernet)After consulting with Oracle, we were able to confirm that this is a known issue with 19c. Note that the documents and bugs below are internal:
- ASM Fails to Start After Running "rootupgrade.sh" on First Node While Upgrading to 19c Grid Infrastructure (Doc ID 2606735.1)
- Bug 30265357 - FAILS TO START ORA.ASM WHEN UPGRADE FROM 12.1.0.2 TO 19.3.0
- Bug 30452852 - DURING UPGRADE FROM 12C TO 19C +ASM1(19C) FAILS TO JOIN ALREADY RUNNING +ASM2(12C)
- Bug 29379299 - HAIP SUBNET FLIPS UPON BOTH LINK DOWN/UP EVENT
To resolve this issue, do the following:
- Shut down Node 1 (crsctl stop crs).
- Run rootupgrade.sh in Node 2.
- After the rootupgrade.sh script completes in Node 2, start up both Node 1 and Node 2.
- To finish the upgrade in Node 1, run /u01/app/19.6.0.0/grid/gridSetup.sh -executeConfigTools -responseFile /tmp/gridresponse.rsp -silent.
Make sure you are aware of this bug if you are preparing to perform a GI upgrade from 12.x to 19.6. Otherwise, the upgrade causes a complete outage (not a rolling upgrade).
Note: This post was originally published at rene-ace.com.
