Recovering from quarterly PSU patching disaster

Applying GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.181016(Patch 28429134) for AIX (64-Bit) was a disaster for Oracle Restart environment, as described in a previous post.

# $GRID_HOME/OPatch/opatch auto $GI_PSU -ocmrf /tmp/ocm.rsp

patch /stage/psu/11g/GI/2018OCT/28429134/27735020/custom/server/27735020  apply successful for home  /u01/app/oracle/product/11.2.0/db_1
patch /stage/psu/11g/GI/2018OCT/28429134/28204707  apply failed  for home  /u01/app/oracle/product/11.2.0/db_1

patch /stage/psu/11g/GI/2018OCT/28429134/27735020  apply successful for home  /u01/11.2.0/grid
patch /stage/psu/11g/GI/2018OCT/28429134/28204707  apply failed  for home  /u01/11.2.0/grid

After failed patching, the following commands failed; hence, a rollback was not a feasible option and we needed to restore GI and DB Home.

# crsctl config has
exec(): 0509-036 Cannot load program crsctl.bin because of the following errors:
rtld: 0712-001 Symbol ztca_Shutdown was referenced
      from module crsctl.bin(), but a runtime definition
      of the symbol was not found.
rtld: 0712-001 Symbol ztpk_SetKeyInfo was referenced
      from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
      of the symbol was not found.
rtld: 0712-001 Symbol ztpk_DestroyKey was referenced
      from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
      of the symbol was not found.
rtld: 0712-001 Symbol ztpk_Sign was referenced
      from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
      of the symbol was not found.
rtld: 0712-001 Symbol ztpk_Verify was referenced
      from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
      of the symbol was not found.
rtld: 0712-002 fatal error: exiting.

$ srvctl config database -d $ORACLE_SID
exec(): 0509-036 Cannot load program getcrshome because of the following errors:
rtld: 0712-001 Symbol ztpk_SetKeyInfo was referenced
      from module /u01/app/oracle/product/11.2.0/db_1/lib/libhasgen11.so(), but a runtime definition
      of the symbol was not found.
rtld: 0712-001 Symbol ztpk_DestroyKey was referenced
      from module /u01/app/oracle/product/11.2.0/db_1/lib/libhasgen11.so(), but a runtime definition
      of the symbol was not found.

Below are the high level steps we implemented to restore, reconfigure, and relink GI and DB Home. These may vary by environment.

Restore GI HOME:
cd $GRID_HOME
cp /backup/GIHome.tar .
tar -xvpf GIHome.tar
$GRID_HOME/crs/install/roothas.pl -unlock
cd $GRID_HOME/rdbms/lib/
ls -l config.o
mv config.o config.o.bkup
$GRID_HOME/bin/relink all

# ./rootadd_rdbms.sh
# ./roothas.pl -patch

Restore DB HOME:
cd $ORACLE_HOME
cp /backup/DBHome.tar .
tar -xvpf DBHome.tar
$ORACLE_HOME/bin/relink all

In conclusion, don’t depend solely on rollback to recover from failed patching. It might be a good idea to back up binaries.

Some useful references:
How To Relink The Oracle Grid Infrastructure Standalone (Restart) Installation Or Oracle Grid Infrastructure RAC/Cluster Installation (11.2 or 12c). ( Doc ID 1536057.1 )
How to backup a Grid Infrastructure installation (Doc ID 1482803.1)
Relinking Oracle Home FAQ ( Frequently Asked Questions) (Doc ID 1467060.1)

It’s nice to see Oracle’s updated Patch 28429134 has been withdrawn.

Master Note for Database Proactive Patch Program (Doc ID 756671.1)
GI PSU 11.2.0.4.181016 Patch 28429134
All except HP-UX PA-RISC (64-Bit), AIX (64-Bit)
ETA: 13-November-2018