Silent Out-of-Place Patching For Oracle 19c GI

Posted in: Oracle, Technical Track

I am going through a process of patching an Oracle Grid Infrastructure (GI) from 19.6 to 19.8 (Patch 31305339) and I wanted to try out-of-place patching (OOP). Since I didn’t have access to the GUI (graphical user interface) environment, I proceeded to use the silent option. Below is how it went:

I started with verifying that the GI services were up and running using Fred Dennis’s script and the current patches applied to the GI. This was a sanity check before I began work on the patch.

[grid@node1 ~]$ ./rac_status.sh -a

		Cluster rene-ace-c

        Type      |      Name      |      node1      |      node2      |
  ---------------------------------------------------------------------
   asm            | asm            |      Online     |      Online     |
   asmnetwork     | asmnet1        |      Online     |      Online     |
   chad           | chad           |      Online     |      Online     |
   cvu            | cvu            |      Online     |        -        |
   dg             | DATA           |      Online     |      Online     |
   dg             | RECO           |      Online     |      Online     |
   network        | net1           |      Online     |      Online     |
   ons            | ons            |      Online     |      Online     |
   qosmserver     | qosmserver     |      Online     |        -        |
   vip            | node1          |      Online     |        -        |
   vip            | node2          |        -        |      Online     |
   vip            | scan1          |        -        |      Online     |
   vip            | scan2          |      Online     |        -        |
   vip            | scan3          |      Online     |        -        |
  ---------------------------------------------------------------------
    x  : Resource is disabled
       : Has been restarted less than 24 hours ago
   
      Listener    |      Port      |      node1      |      node2      |     Type     |
  ------------------------------------------------------------------------------------
   ASMNET1LSNR_ASM| TCP:1525       |      Online     |      Online     |   Listener   |
   LISTENER       | TCP:1521       |      Online     |      Online     |   Listener   |
   LISTENER_SCAN1 | TCP:1521       |        -        |      Online     |     SCAN     |
   LISTENER_SCAN2 | TCP:1521       |      Online     |        -        |     SCAN     |
   LISTENER_SCAN3 | TCP:1521       |      Online     |        -        |     SCAN     |
  ------------------------------------------------------------------------------------
   
   
        DB      |    Version    |      node1      |      node2      |    DB Type   |
  ---------------------------------------------------------------------------------------
   renedev        | 19.6.0.0  (1) |         -         |        Open       |  SINGLE (P)  |
   reneqa         | 19.6.0.0  (2) |        Open       |        Open       |    RAC (P)   |

[grid@node1 ~]$ echo $ORACLE_HOME
/u01/app/19.3.0.0/grid
[grid@node1 ~]$ cd $ORACLE_HOME/OPatch
[grid@node1 OPatch]$ ./opatch lspatches
30655595;TOMCAT RELEASE UPDATE 19.0.0.0.0 (30655595)
30557433;Database Release Update : 19.6.0.0.200114 (30557433)
30489632;ACFS RELEASE UPDATE 19.6.0.0.0 (30489632)
30489227;OCW RELEASE UPDATE 19.6.0.0.0 (30489227)

OPatch succeeded.


  

With that done, I made sure the patches for 19.8 were applicable to the 19.6 GI and wouldn’t lead to any conflicts or issues.

## Note.- I trimmed the log for readability
[root@node1 ~]$ . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@node1 ~]$ export PATH=$ORACLE_HOME/OPatch:$PATH
[root@node1 ~]$ opatchauto apply /u01/patches/30501910 -analyze
...
Host:node1
CRS Home:/u01/app/19.6.0.0/grid
Version:19.0.0.0.0

==Following patches were SUCCESSFULLY analyzed to be applied:

Patch: /u01/patches/31305339/31281355
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-07_11-41-25AM_1.log

Patch: /u01/patches/31305339/31304218
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-07_11-41-25AM_1.log

Patch: /u01/patches/31305339/31305087
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-07_11-41-25AM_1.log

Patch: /u01/patches/31305339/31335188
Log: /u01/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2020-10-07_11-41-25AM_1.log

OPatchauto session completed at Wed Oct  7 11:42:46 2020

Time taken to complete the session 2 minutes, 41 seconds

Once I was sure the 19.8 patch wouldn’t cause an issue, I proceeded with the following methodology:

Step one was to unzip the 19.3 GI software to the ORACLE_HOME that will be the 19.8 patched GI. You can download the software from either OTN or edelivery. In my case I used edelivery.

I also updated the OPatch version to the one the 19.8 (31305339) patch needs, which is OPatch version 12.2.0.1.21 for DB 19.x releases (April 2020).

[grid@node1 grid]$ unzip -q /u01/patches/v982068-01.zip -d /u01/app/19.8.0.0/grid
[grid@node1 grid]$ mv /u01/app/19.8.0.0/grid/OPatch /u01/app/19.8.0.0/grid/OPatch.old
[grid@node1 grid]$ unzip -q /u01/patches/p6880880_190000_Linux-x86-64.zip -d /u01/app/19.8.0.0/grid

In these cases I normally unset my Oracle variables, so that whatever I do, I don’t hurt anything currently running. After that I set the ORACLE_HOME to the 19.8 environment.

# a. Unset all Oracle related shell variables
[grid@node1 ~]$ unset ORACLE_BASE
[grid@node1 ~]$ unset ORACLE_HOME
[grid@node1 ~]$ unset ORACLE_SID
[grid@node1 ~]$ unset ORA_CRS_HOME
[grid@node1 ~]$ unset ORA_NLS10
[grid@node1 ~]$ unset TNS_ADMIN
 
#b. The following must return nothing
[grid@node1 ~]$ env | egrep "ORA|TNS" | wc -l
   0

#c. Set the 19.8 ORACLE_HOME
[grid@node1 ~]$ export ORACLE_HOME=/u01/app/19.8.0.0/grid

Now I was ready to run the gridSetup.sh in silent mode. I applied the 19.8 patch passing the response file parameters via command line.

When I did this, I faced the the patch error ZOP-51. I tried looking for a special character in what I was passing, but saw nothing. Even though I verified that the patch binaries where owned by grid:oinstall, and that everything on that location look good to me, I kept running into this error.

[grid@node1 ~]$ cd $ORACLE_HOME
[grid@node1 ~]$ pwd
/u01/app/19.8.0.0/grid
[grid@node1 grid]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent \
>     -ApplyRU /u01/patches/31305339 \
>     -responseFile /u01/app/19.8.0.0/grid/install/response/gridsetup.rsp \
>     INVENTORY_LOCATION=/u01/app/oraInventory \
>     SELECTED_LANGUAGES=en,en_GB \
>     oracle.install.option=CRS_SWONLY \
>     ORACLE_BASE=/u01/app/grid \
>     oracle.install.asm.OSDBA=asmdba \
>     oracle.install.asm.OSOPER=asmoper \
>     oracle.install.asm.OSASM=asmadmin \
>     oracle.install.crs.config.ClusterConfiguration=STANDALONE \
>     oracle.install.crs.config.configureAsExtendedCluster=false \
>     oracle.install.crs.config.clusterName=rene-ace-c \
>     oracle.install.crs.config.gpnp.configureGNS=false \
>     oracle.install.crs.config.autoConfigureClusterNodeVIP=false \
>     oracle.install.crs.config.clusterNodes=node1.localdomain,node2.localdomain
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2020-10-07_11-51-50AM/gridSetupActions2020-10-07_11-51-50AM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2020-10-07_11-51-50AM/gridSetupActions2020-10-07_11-51-50AM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /u01/app/19.8.0.0/grid/install/response/grid_2020-10-07_11-51-50AM.rsp

You can find the log of this install session at:
 /u01/app/oraInventory/logs/GridSetupActions2020-10-07_11-51-50AM/gridSetupActions2020-10-07_11-51-50AM.log
[FATAL] [INS-32500] OPatch tool failed.
   ACTION: Review /u01/app/oraInventory/logs/GridSetupActions2020-10-07_11-51-50AM/gridSetupActions2020-10-07_11-51-50AM.log for more details.
*ADDITIONAL INFORMATION:*
 - Oracle Interim Patch Installer version 12.2.0.1.21
 - Copyright (c) 2020, Oracle Corporation. All rights reserved.
 - 
 - 
 - ZOP-51: The patch location is not valid for apply, because it doesn't have correct metadata, or it points to a patch directory.
 - Please check the arguments and try again.
 - 
 - OPatch failed with error code 135
 - Argument(s) Error... Patch location is not valid for apply
 - 

To address this, I built a response file containing the exact values that I was passing in the command line. Once I did this, I was able to install the CRS without an issue.

[grid@node1 ~]$ cd $ORACLE_HOME
[grid@node1 ~]$ pwd
/u01/app/19.8.0.0/grid
[grid@node1 grid]$ ./gridSetup.sh -ignorePrereq -waitforcompletion -silent -responseFile /home/grid/grid_19.rsp -applyRU /u01/patches/31305339

Preparing the home to patch...
Applying the patch /u01/patches/31305339...
Successfully applied the patch.
The log can be found at: /u01/app/oraInventory/logs/GridSetupActions2020-10-07_12-32-13PM/installerPatchActions_2020-10-07_12-32-13PM.log
Launching Oracle Grid Infrastructure Setup Wizard...

[WARNING] [INS-13014] Target environment does not meet some optional requirements.
   CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/GridSetupActions2020-10-07_12-32-13PM/gridSetupActions2020-10-07_12-32-13PM.log
   ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/GridSetupActions2020-10-07_12-32-13PM/gridSetupActions2020-10-07_12-32-13PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
The response file for this session can be found at:
 /u01/app/19.8.0.0/grid/install/response/grid_2020-10-07_12-32-13PM.rsp

You can find the log of this install session at:
 /u01/app/oraInventory/logs/GridSetupActions2020-10-07_12-32-13PM/gridSetupActions2020-10-07_12-32-13PM.log

As a root user, execute the following script(s):
	1. /u01/app/19.8.0.0/grid/root.sh

Execute /u01/app/19.8.0.0/grid/root.sh on the following nodes: 
[node1, node2]

Successfully Setup Software with warning(s).
...
[root@node1 patches]# /u01/app/19.8.0.0/grid/root.sh
Check /u01/app/19.8.0.0/grid/install/root_node1_2020-10-07_13-01-25-549937823.log for the output of root script
[root@node1 patches]# ssh node2
Last login: Wed Oct  7 12:57:07 2020

[root@node2 ~]# /u01/app/19.8.0.0/grid/root.sh
Check /u01/app/19.8.0.0/grid/install/root_node2_2020-10-07_13-01-32-411313252.log for the output of root script

I next proceeded to switch the GI to the installed 19.8. Keep in mind that running the root.sh will take a while as this is essentially where the GI moves from the 19.6 to 19.8. It’s also important never to run the root.sh concurrently; it has to run on node1 first and then on node2.

[grid@node1 ~]$ cd /u01/app/19.8.0.0/grid
[grid@node1 grid]$ ./gridSetup.sh -switchGridHome -silent
Launching Oracle Grid Infrastructure Setup Wizard...

You can find the log of this install session at:
 /u01/app/oraInventory/logs/cloneActions2020-10-07_01-19-16PM.log

As a root user, execute the following script(s):
	1. /u01/app/19.8.0.0/grid/root.sh

Execute /u01/app/19.8.0.0/grid/root.sh on the following nodes: 
[node1, node2]

Run the scripts on the local node first. After successful completion, run the scripts in sequence on all other nodes.

Successfully Setup Software.
...
[root@node1 patches]# /u01/app/19.8.0.0/grid/root.sh
Check /u01/app/19.8.0.0/grid/install/root_node1_2020-10-07_13-32-59-377526581.log for the output of root script
[root@node1 patches]# ssh node2
Last login: Wed Oct  7 13:33:03 2020

[root@node2 ~]# /u01/app/19.8.0.0/grid/root.sh
Check /u01/app/19.8.0.0/grid/install/root_node2_2020-10-07_13-38-09-323573349.log for the output of root script

After running the root.sh in each node, I proceeded to update the /etc/oratab. I verified that the patches were correct. I also checked that the services that were up prior to the patching exercise remained up after it.

[grid@node1 ~]$ cat /etc/oratab |grep "+ASM"
+ASM1:/u01/app/19.8.0.0/grid:N
[grid@node1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? 
The Oracle base has been set to /u01/app/grid

[grid@node1 OPatch]$ $ORACLE_HOME/OPatch/opatch lspatches
31335188;TOMCAT RELEASE UPDATE 19.0.0.0.0 (31335188)
31305087;OCW RELEASE UPDATE 19.8.0.0.0 (31305087)
31304218;ACFS RELEASE UPDATE 19.8.0.0.0 (31304218)
31281355;Database Release Update : 19.8.0.0.200714 (31281355)

OPatch succeeded.

[grid@node1 ~]$ ./rac_status.sh -a

		Cluster rene-ace-c

        Type      |      Name      |      node1      |      node2      |
  ---------------------------------------------------------------------
   asm            | asm            |      Online     |      Online     |
   asmnetwork     | asmnet1        |      Online     |      Online     |
   chad           | chad           |      Online     |      Online     |
   cvu            | cvu            |      Online     |        -        |
   dg             | DATA           |      Online     |      Online     |
   dg             | RECO           |      Online     |      Online     |
   network        | net1           |      Online     |      Online     |
   ons            | ons            |      Online     |      Online     |
   qosmserver     | qosmserver     |      Online     |        -        |
   vip            | node1          |      Online     |        -        |
   vip            | node2          |        -        |      Online     |
   vip            | scan1          |        -        |      Online     |
   vip            | scan2          |      Online     |        -        |
   vip            | scan3          |      Online     |        -        |
  ---------------------------------------------------------------------
    x  : Resource is disabled
       : Has been restarted less than 24 hours ago
   
      Listener    |      Port      |      node1      |      node2      |     Type     |
  ------------------------------------------------------------------------------------
   ASMNET1LSNR_ASM| TCP:1525       |      Online     |      Online     |   Listener   |
   LISTENER       | TCP:1521       |      Online     |      Online     |   Listener   |
   LISTENER_SCAN1 | TCP:1521       |        -        |      Online     |     SCAN     |
   LISTENER_SCAN2 | TCP:1521       |      Online     |        -        |     SCAN     |
   LISTENER_SCAN3 | TCP:1521       |      Online     |        -        |     SCAN     |
  ------------------------------------------------------------------------------------
   
   
        DB      |    Version    |      node1      |      node2      |    DB Type   |
  ---------------------------------------------------------------------------------------
   renedev        | 19.6.0.0  (1) |         -         |        Open       |  SINGLE (P)  |
   reneqa         | 19.6.0.0  (2) |        Open       |        Open       |    RAC (P)   |

Last but not least, it’s important to remove the 19.6 GI Home from the inventory and delete it. I recommend taking at least a few days to delete it as if you need to rollback this is where you will rollback to.

[grid@node1 ~]$ /u01/app/19.8.0.0/grid/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME="/u01/app/19.6.0.0/grid"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4076 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
You can find the log of this install session at:
 /u01/app/oraInventory/logs/DetachHome2020-10-07_02-26-40PM.log
'DetachHome' was successful.

# As root 
[root@node1 ~]$ cd /u01/app/
[root@node1 ~]$ rm -rf 19.6.0.0

[grid@node2 ~]$ /u01/app/19.8.0.0/grid/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME="/u01/app/19.6.0.0/grid"
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB.   Actual 4073 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
You can find the log of this install session at:
 /u01/app/oraInventory/logs/DetachHome2020-10-07_02-26-54PM.log
'DetachHome' was successful.

# As root 
[root@node2 ~]$ cd /u01/app/
[root@node2 ~]$ rm -rf 19.6.0.0

I hope this blog helps if you’re going to do a silent OOP Patching for your GI. I’m also including the grid rsp file I used so you can use it as reference.

Note: This was originally published on rene-ace.com.

email

Authors

Interested in working with Rene? Schedule a tech call.

About the Author

Currently I am an Oracle ACE ; Speaker at Oracle Open World, Oracle Developers Day, OTN Tour Latin America and APAC region and IOUG Collaborate ; Co-President of ORAMEX (Mexico Oracle User Group); At the moment I am an Oracle Project Engineer at Pythian. In my free time I like to say that I'm Movie Fanatic, Music Lover and bringing the best from México (Mexihtli) to the rest of the world and in the process photographing it ;)

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *