When is the best time to apply an Oracle quarterly patch? I suppose the answer is, “It depends.”
As it turns out, we started to apply October 2018 Grid Infrastructure PSU for AIX the day after it was released and it was a disaster. There were library corruptions for GI and DB with Oracle Restart configuration. Oracle’s only suggestion is to rollback which was easier said than done, since simple commands such as crsctl config would fail.
Pretty ugly.
# crsctl config has
exec(): 0509-036 Cannot load program crsctl.bin because of the following errors:
rtld: 0712-001 Symbol ztca_Shutdown was referenced
from module crsctl.bin(), but a runtime definition
of the symbol was not found.
rtld: 0712-001 Symbol ztpk_SetKeyInfo was referenced
from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
of the symbol was not found.
rtld: 0712-001 Symbol ztpk_DestroyKey was referenced
from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
of the symbol was not found.
rtld: 0712-001 Symbol ztpk_Sign was referenced
from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
of the symbol was not found.
rtld: 0712-001 Symbol ztpk_Verify was referenced
from module /u01/11.2.0/grid/lib/libhasgen11.so(), but a runtime definition
of the symbol was not found.
rtld: 0712-002 fatal error: exiting.
To make matters worse, there was a breakdown in communication among Oracle’s support engineers. I recall one engineer mentioned there were issues with the current patch release and, unfortunately, this was not documented in the SR.
Another contribution to the failure is that the patch download was corrupted since the checksum did not match and was not checked.
Maybe there are no issues after all.
Fast forward. We tried patching the environment again, only to have it fail for the second time.
Finally, the same Oracle support engineer who did not think there were issues with patch release has now confirmed there are issues.
Lesson learned.
1. Don’t be too hasty to patch.
2. Always get the name of the engineer you are speaking with and ensure the conversation is documented in SR.
3. Trust, but verify, as Oracle engineers are human too.
4. Lastly, have backup. How many backup GI and DB homes before patching? This was the saving grace.
Happy patching and stayed tuned!
In the next post, I will share how we managed to recover from the disaster.
No comments