Recovering from quarterly PSU patching disaster
Applying GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.181016(Patch 28429134) for AIX (64-Bit) was a disaster for Oracle Restart environment, as described in a previous post.
# $GRID_HOME/OPatch/opatch auto $GI_PSU -ocmrf /tmp/ocm.rsp patch /stage/psu/11g/GI/2018OCT/28429134/27735020/custom/server/27735020 apply successful for home /u01/app/oracle/product/11.2.0/db_1 patch /stage/psu/11g/GI/2018OCT/28429134/28204707 apply failed for home /u01/app/oracle/product/11.2.0/db_1 patch /stage/psu/11g/GI/2018OCT/28429134/27735020 apply successful for home /u01/11.2.0/grid patch /stage/psu/11g/GI/2018OCT/28429134/28204707 apply failed for home /u01/11.2.0/gridAfter failed patching, the following commands failed; hence, a rollback was not a feasible option and we needed to restore GI and DB Home.
# crsctl config has exec(): 0509-036 Cannot load program crsctl.bin because of the following errors: rtld: 0712-001 Symbol ztca_Shutdown was referenced from module crsctl.bin(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol ztpk_SetKeyInfo was referenced from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol ztpk_DestroyKey was referenced from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol ztpk_Sign was referenced from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol ztpk_Verify was referenced from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition of the symbol was not found. rtld: 0712-002 fatal error: exiting. $ srvctl config database -d $ORACLE_SID exec(): 0509-036 Cannot load program getcrshome because of the following errors: rtld: 0712-001 Symbol ztpk_SetKeyInfo was referenced from module /u01/app/oracle/product/11.2.0/db_1/lib/libhasgen11.so(), but a runtime definition of the symbol was not found. rtld: 0712-001 Symbol ztpk_DestroyKey was referenced from module /u01/app/oracle/product/11.2.0/db_1/lib/libhasgen11.so(), but a runtime definition of the symbol was not found.Below are the high level steps we implemented to restore, reconfigure, and relink GI and DB Home. These may vary by environment.
Restore GI HOME: cd $GRID_HOME cp /backup/GIHome.tar . tar -xvpf GIHome.tar $GRID_HOME/crs/install/roothas.pl -unlock cd $GRID_HOME/rdbms/lib/ ls -l config.o mv config.o config.o.bkup $GRID_HOME/bin/relink all # ./rootadd_rdbms.sh # ./roothas.pl -patch Restore DB HOME: cd $ORACLE_HOME cp /backup/DBHome.tar . tar -xvpf DBHome.tar $ORACLE_HOME/bin/relink allIn conclusion, don't depend solely on rollback to recover from failed patching. It might be a good idea to back up binaries. Some useful references: How To Relink The Oracle Grid Infrastructure Standalone (Restart) Installation Or Oracle Grid Infrastructure RAC/Cluster Installation (11.2 or 12c). ( Doc ID 1536057.1 ) How to backup a Grid Infrastructure installation (Doc ID 1482803.1) Relinking Oracle Home FAQ ( Frequently Asked Questions) (Doc ID 1467060.1) It's nice to see Oracle's updated Patch 28429134 has been withdrawn. Master Note for Database Proactive Patch Program (Doc ID 756671.1) GI PSU 11.2.0.4.181016 Patch 28429134 All except HP-UX PA-RISC (64-Bit), AIX (64-Bit) ETA: 13-November-2018
