Patching fun: GI and database patches

So, it has been some time since I have patched a database. Patching seemed necessary in a 12c database used for testing, as it was having some crashing issues. As the database and GI home were both un-patched, it seemed worthwhile to just apply the latest set of patches rather than spend (more) time troubleshooting the issue. This is a 12.1.0.2 CDB database on Oracle Linux 6.5, 64 bit. This is a standalone database with Oracle Restart and no RAC.

After downloading the appropriate patches, I unzipped the files to a local directory, installed OPatch and then started in on the patch documentation. After perusing the docs it appeared that patch would be fairly easy with this command:

# opatchauto apply <UNZIPPED_PATCH_LOCATION>/20996835 -ocmrf <ocm response file>

One problem though. What is an OCM response file?

As this was my first time using Opatchauto, I wanted to make this work.  The documentation did include a reference to the following helpful Support Note:

How to Create an OCM Response file to Apply a Patch in Silent Mode – opatch silent (Doc ID 966023.1)

Creating the response file was quite straightforward, since the Oracle user the following command took care of it:

cd /u01/app/oracle/patches/20996835
$ORACLE_HOME/OPatch/ocm/bin/emocmrsp -no_banner -output ./unconfig.rsp

With that out of the way it was now time to try the patch. This was done as root, as in instructed in the patch documentation.

/u01/app/grid/product/12.1.0/grid/OPatch/opatchauto apply /u01/app/oracle/patches/20996835 -ocmrf /u01/app/oracle/patches/20996835/unconfig.rsp

All went well for a few moments, but then (gasp) a failure.  Please excuse the sarcasm; I may not have patched a database recently, yet I’m hardly new to the process.  Patching working properly on the first attempt would have been quite a surprise.

Here is what appeared in the Opatch log file:

2015-09-03_17-30-37 :
Clusterware is either not running or not configured. You have the following 2 options:
1. Configure and start the Clusterware on this node and re-run the tool
2. Run the tool with '-oh <GI_HOME>' to first patch the Grid Home, then invoke tool with '-database <oracle database name>' or '-oh <RAC_HOME>' to patch the RAC home
Parameter Validation: FAILED

Apparently I should not have shutdown GI before starting. Rather than restarting GI, I chose door #2, that is patch GI and DB homes separately.

Seems simple enough. Here’s the command I used:

/u01/app/grid/product/12.1.0/grid/OPatch/opatchauto apply /u01/app/oracle/patches/20996835 -oh /u01/app/grid/product/12.1.0/grid -ocmrf /u01/app/oracle/patches/20996835/unconfig.rsp

Again, all went well for a few moments, and then another failure.  This time a file could not be found. This is from the log file:

Patch 20831113:
onewaycopyAction : Source File "/u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl" does not exists or is not readable
'oracle.crs, 12.1.0.2.0': Cannot copy file from 'dropdb.pl' to '/u01/app/grid/product/12.1.0/grid/crs/install/dropdb.pl'

That seemed rather odd.  The patch is running as root, and the file is definitely readable by root:

[[email protected] 20996835]$ ls -l /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl
-rwx------ 1 oracle asmdba 3541 Jul 12 04:04 /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl

Thinking this might just be a one of error, I changed the permissions on this file to global read:

[[email protected] 20996835]$ chmod +r /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl
[[email protected] 20996835]$ ls -l /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl
-rwxr--r-- 1 oracle asmdba 3541 Jul 12 04:04 /u01/app/oracle/patches/20996835/20831113/files/crs/install/dropdb.pl

… and then re-ran the patch command. Unfortunately yet another failure. Similar to the previous error, but of course this time a different file:

Patch 19769480:
onewaycopyAction : Source File "/u01/app/oracle/patches/20996835/20831110/19769480/files/sqlpatch/sqlpatch" does not exists or is not readable
'oracle.rdbms, 12.1.0.2.0': Cannot copy file from 'sqlpatch' to '/u01/app/grid/product/12.1.0/grid/sqlpatch/sqlpatch'

Checking on the file, again readable by root:

[[email protected] 20996835]$ ls -l /u01/app/oracle/patches/20996835/20831110/19769480/files/sqlpatch/sqlpatch
-rwx--x--- 1 oracle asmdba 2808 Jul 12 04:03 /u01/app/oracle/patches/20996835/20831110/19769480/files/sqlpatch/sqlpatch

Searching the Oracle Support site on this issue led to this note: OPatch fails with error “sqlpatch does not exists or is not readable” (Doc ID 2026594.1)

The solution as per this note is as follows:

Make sure to download the patch as Oracle Software owner and install with same user account.

As a workaround, to fix this error, we have to give the read permission to the file which is issue 

Executing the patch as the Oracle software owner however simply does not work:

[[email protected] ~]$ $ORACLE_HOME/OPatch/opatchauto apply /u01/app/oracle/patches/20996835/ -oh $ORACLE_HOME -ocmrc /u01/app/oracle/patches/20996835/unconfig.rsp
Cannot run as 'grid'. Please run as root user.

Ok, so what is the real solution?

It was apparent that the reason for these errors is that at some point the patching process is doing an su (or something similar) to the software owners account, in this case the grid user.  And grid cannot read that file. The simplest solution seemed to be this: Make all files and directories in the patch directory globally readable.

[[email protected] ~]$ cd /u01/app/oracle/patches/20996835/
[[email protected] 20996835]$ chmod -R +r . 

Yes, it was just that easy.  Too bad the patch files do not have the correct permissions right out of the box. Following this the patch succeeded for the GI home, and then also for the DB home. And yes, I did run datapatch as well.

Has the 12c database stopped crashing? I don’t know yet, but will report back later.

References:

• Quick Reference to Patch Numbers for Database PSU, SPU(CPU), Bundle Patches and Patchsets (Doc ID 1454618.1)
• OPatch fails with error “sqlpatch does not exists or is not readable” (Doc ID 2026594.1)
• How to Create an OCM Response file to Apply a Patch in Silent Mode – opatch silent (Doc ID 966023.1)
• Master Note For OPatch (Doc ID 293369.1)

Patches:

• OPatch:  p6880880_121010_Linux-x86-64.zip
• GI Patch: p20996835_121020_Linux-x86-64.zip

Discover more about our expertise in Oracle.