Opatchauto Failing On “CheckActiveFilesAndExecutables” during Prerequisite Check
Very recently, when applying the 2021 January CPU in a client environment, the following happened:
[root@dbserver01 32226239]# $ORACLE_HOME/OPatch/opatchauto apply OPatchauto session is initiated at Sun Mar 14 03:00:06 2021 System initialization log file is /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchautodb/systemconfig2021-03-14_03-00-08AM.log. Session log file is /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/opatchauto2021-03-14_03-00-13AM.log The id for this session is 1J89 Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19c/db Patch applicability verified successfully on home /u01/app/oracle/product/19c/db Executing patch validation checks on home /u01/app/oracle/product/19c/db Patch validation checks successfully completed on home /u01/app/oracle/product/19c/db Verifying SQL patch applicability on home /u01/app/oracle/product/19c/db SQL patch applicability verified successfully on home /u01/app/oracle/product/19c/db Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19c/grid Patch applicability verified successfully on home /u01/app/oracle/product/19c/grid Executing patch validation checks on home /u01/app/oracle/product/19c/grid Patch validation checks successfully completed on home /u01/app/oracle/product/19c/grid Preparing to bring down database service on home /u01/app/oracle/product/19c/db Successfully prepared home /u01/app/oracle/product/19c/db to bring down database service Bringing down database service on home /u01/app/oracle/product/19c/db Following database has been stopped and will be restarted later during the session: er1pprd,obiee Database service successfully brought down on home /u01/app/oracle/product/19c/db Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/oracle/product/19c/grid Prepatch operation log file location: /u01/app/oracle/product/crsdata/dbserver01/crsconfig/hapatch_2021-03-14_03-06-15AM.log CRS service brought down successfully on home /u01/app/oracle/product/19c/grid Start applying binary patch on home /u01/app/oracle/product/19c/db Failed while applying binary patches on home /u01/app/oracle/product/19c/db Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures: Patch Target : dbserver01->/u01/app/oracle/product/19c/db Type[sidb] Details: [ ---------------------------Patching Failed--------------------------------- Command execution failed during patching in home: /u01/app/oracle/product/19c/db, host: dbserver01. Command failed: /u01/app/oracle/product/19c/db/OPatch/opatchauto apply /ora02/soft/jan21cpu/32126842/32226239 -oh /u01/app/oracle/product/19c/db -target_type oracle_database -binary -invPtrLoc /u01/app/oracle/product/19c/grid/oraInst.loc -jre /u01/app/oracle/product/19c/grid/OPatch/jre -persistresult /u01/app/oracle/product/19c/db/opatchautocfg/db/sessioninfo/sessionresult_dbserver01_sidb_2.ser -analyzedresult /u01/app/oracle/product/19c/db/opatchautocfg/db/sessioninfo/sessionresult_analyze_dbserver01_sidb_2.ser Command failure output: ==Following patches FAILED in apply: Patch: /ora02/soft/jan21cpu/32126842/32226239/32218454 Log: /u01/app/oracle/product/19c/db/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_03-17-58AM_1.log Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed. After fixing the cause of failure Run opatchauto resume ] OPATCHAUTO-68061: The orchestration engine failed. OPATCHAUTO-68061: The orchestration engine failed with return code 1 OPATCHAUTO-68061: Check the log for more details. OPatchAuto failed. OPatchauto session completed at Sun Mar 14 03:19:25 2021 Time taken to complete the session 19 minutes, 19 seconds opatchauto failed with error code 42
Going by parts, let’s see what we have on the referred log:
[Mar 14, 2021 3:33:32 AM] [INFO] Start fuser command /sbin/fuser /u01/app/oracle/product/19c/grid/bin/expdp at Sat Mar 14 03:33:32 PDT 2021 [Mar 14, 2021 3:33:32 AM] [INFO] Finish fuser command /sbin/fuser /u01/app/oracle/product/19c/grid/bin/expdp at Sat Mar 14 03:33:32 PDT 2021 [Mar 14, 2021 3:33:32 AM] [INFO] Following active executables are not used by opatch process : Following active executables are used by opatch process : /u01/app/oracle/product/19c/grid/lib/libclntsh.so.19.1 [Mar 14, 2021 3:33:32 AM] [INFO] Prerequisite check "CheckActiveFilesAndExecutables" failed. The details are: Following active executables are not used by opatch process : Following active executables are used by opatch process : /u01/app/oracle/product/19c/grid/lib/libclntsh.so.19.1 [Mar 14, 2021 3:33:33 AM] [INFO] UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed. [Mar 14, 2021 3:33:33 AM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed. [Mar 14, 2021 3:33:33 AM] [INFO] Finishing UtilSession at Sat Mar 14 03:33:33 PDT 2021 [Mar 14, 2021 3:33:33 AM] [INFO] Log file location: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/oapatch_2021-03-14_03-06-15AM.log
This is an interesting situation.
After some validations to make sure no service was online, the path was writable, and also that Oracle and root had the required privilege and access, I found some relevant Oracle notes:
- 19c Installation Fails with error “libclntsh.so: file format not recognized; treating as linker script” (Doc ID 2631283.1): Pointing to file corruption.
- While Applying a Weblogic Patch, opatch Fails with “Prerequisite check “CheckActiveFilesAndExecutables” failed” Error (Doc ID 2705809.1): Not a DB note and pointing to other processes using the files.
- Opatch failure due to “CheckActiveFilesAndExecutables” as Remote registry service holding files (Doc ID 2462952.1): Remote registry holding the binaries.
- Prerequisite Check “Checkactivefilesandexecutables” Failed (Doc ID 1281644.1): Patch requisite miss on 10g.
- Failed to apply PSU due to CheckActiveFilesAndExecutables check failure (Doc ID 2506432.1): SQL Plus holding the binaries.
- [OCI]: Database System Patching Failed With Error “DCS-10001:Internal Error Encountered: Failure : Failed To Apply” And Opatch Log Shows “Prerequisite check “CheckActiveFilesAndExecutables” failed” (Doc ID 2687607.1): My case is not an OCI and not in RAC.
In other words, there were no matches at all.
However, the final note gave me the hints I needed. From Doc ID 2687607.1, for RAC environments:
/u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -unlock /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -init /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -prepatch /u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -postpatch
So, in my case, a standalone on-premise database (and GI):
/ora01/app/oracle/product/19c/grid/crs/install/roothas.sh -unlock /ora01/app/oracle/product/19c/grid/crs/install/roothas.sh -init /ora01/app/oracle/product/19c/grid/crs/install/roothas.sh -prepatch [ Apply the patch! ] /ora01/app/oracle/product/19c/grid/crs/install/roothas.sh -postpatch
Check the output:
[root@dbserver01 jan21cpu]# /u01/app/oracle/product/19c/grid/crs/install/roothas.sh -unlock Using configuration parameter file: /u01/app/oracle/product/19c/grid/crs/install/crsconfig_params The log of current session can be found at: /u01/app/oracle/product/crsdata/dbserver01/crsconfig/haunlock__2021-03-14_04-00-35AM.log 2021/03/14 04:01:01 CLSRSC-347: Successfully unlock /u01/app/oracle/product/19c/grid [root@dbserver01 jan21cpu]# /u01/app/oracle/product/19c/grid/crs/install/roothas.sh -init Using configuration parameter file: /u01/app/oracle/product/19c/grid/crs/install/crsconfig_params The log of current session can be found at: /u01/app/oracle/product/crsdata/dbserver01/crsconfig/roothas_2021-03-14_04-01-09AM.log [root@dbserver01 jan21cpu]# /u01/app/oracle/product/19c/grid/crs/install/roothas.sh -prepatch Using configuration parameter file: /u01/app/oracle/product/19c/grid/crs/install/crsconfig_params The log of current session can be found at: /u01/app/oracle/product/crsdata/dbserver01/crsconfig/hapatch_2021-03-14_04-01-16AM.log 2021/03/14 04:01:27 CLSRSC-347: Successfully unlock /u01/app/oracle/product/19c/grid 2021/03/14 04:01:27 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.
And now resuming the Opatchauto:
[root@dbserver01 jan21cpu]# cd 32126842/32226239/ [root@dbserver01 32226239]# $ORACLE_HOME/OPatch/opatchauto resume OPatchauto session is initiated at Sun Mar 14 04:02:07 2021 Session log file is /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/opatchauto2021-03-14_04-02-10AM.log Resuming existing session with id 1J89 Start applying binary patch on home /u01/app/oracle/product/19c/db Binary patch applied successfully on home /u01/app/oracle/product/19c/db Start applying binary patch on home /u01/app/oracle/product/19c/grid Binary patch applied successfully on home /u01/app/oracle/product/19c/grid Performing postpatch operations on CRS - starting CRS service on home /u01/app/oracle/product/19c/grid Postpatch operation log file location: /u01/app/oracle/product/crsdata/dbserver01/crsconfig/hapatch_2021-03-14_04-27-58AM.log CRS service started successfully on home /u01/app/oracle/product/19c/grid Preparing home /u01/app/oracle/product/19c/db after database service restarted No step execution required......... Trying to apply SQL patch on home /u01/app/oracle/product/19c/db SQL patch applied successfully on home /u01/app/oracle/product/19c/db OPatchAuto successful. --------------------------------Summary-------------------------------- Patching is completed successfully. Please find the summary as follows: Host:dbserver01 SIDB Home:/u01/app/oracle/product/19c/db Version:19.0.0.0.0 Summary: ==Following patches were SKIPPED: Patch: /ora02/soft/jan21cpu/32126842/32226239/32218663 Reason: This patch is not applicable to this specified target type - "oracle_database" Patch: /ora02/soft/jan21cpu/32126842/32226239/29340594 Reason: This patch is not applicable to this specified target type - "oracle_database" Patch: /ora02/soft/jan21cpu/32126842/32226239/32240590 Reason: This patch is not applicable to this specified target type - "oracle_database" ==Following patches were SUCCESSFULLY applied: Patch: /ora02/soft/jan21cpu/32126842/32226239/32218454 Log: /u01/app/oracle/product/19c/db/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-02-36AM_1.log Patch: /ora02/soft/jan21cpu/32126842/32226239/32222571 Log: /u01/app/oracle/product/19c/db/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-02-36AM_1.log Host:dbserver01 SIHA Home:/u01/app/oracle/product/19c/grid Version:19.0.0.0.0 Summary: ==Following patches were SUCCESSFULLY applied: Patch: /ora02/soft/jan21cpu/32126842/32226239/29340594 Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log Patch: /ora02/soft/jan21cpu/32126842/32226239/32218454 Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log Patch: /ora02/soft/jan21cpu/32126842/32226239/32218663 Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log Patch: /ora02/soft/jan21cpu/32126842/32226239/32222571 Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log Patch: /ora02/soft/jan21cpu/32126842/32226239/32240590 Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log OPatchauto session completed at Sun Mar 14 04:31:48 2021 Time taken to complete the session 29 minutes, 43 seconds
Here’s the relevant point: This has happened to me on several environments and servers over the past few weeks. It always happens for the 2021 January CPU.
My guess is that this might have something to do with this CPU binary’s set or, most likely, with the latest OPatch version:
[oracle@dbserver01 ~]$ $ORACLE_HOME/OPatch/opatch version OPatch Version: 12.2.0.1.24
I hope this information helps you as well!
If you have any questions, or thoughts, please leave them in the comments.
