Weblogic patch rollback issues in Oracle EBS R12.2

Posted in: Technical Track

When you try to rollback a weblogic patch in Oracle EBS R12.2, you might run into issues similar to below:

$ ./bsu.sh -remove -patchlist=YIJF -prod_dir=/u01/install/VISION/fs2/FMW_Home/wlserver_10.3 -verbose
Checking for conflicts..
No conflict(s) detected

Starting removal of Patch ID: YIJF
Restoring /d1/V1223B5B/fs1/FMW_Home/modules/com.bea.core.datasource6.binding_1.10.0.0.jar
from /u01/install/VISION/fs2/FMW_Home/patch_wls1036/backup/backup.jar
Result: Failure
Failure condition follows:
An error occured while removing patches – run with logging to obtain detailed information
java.io.FileNotFoundException: /d1/V1223B5B/fs1/FMW_Home/modules/com.bea.core.datasource6.binding_1.10.0.0.jar (No such file or directory)
Result: Failure
Failure condition follows:
An error occured while removing patches – run with logging to obtain detailed information
java.io.FileNotFoundException: /d1/V1223B5B/fs1/FMW_Home/modules/com.bea.core.datasource6.binding_1.10.0.0.jar (No such file or directory)

If you observe the errors, it’s trying to restore the jar file to a location that is not present in the current filesystem. This error was captured in a VISION instance created from Oracle VM Template. So you can see that the file path seems to be a path used by an Oracle internal system where the VM template is created. There is not much harm caused by the above issue to the instance.

Now consider this hypothetical situation, where you are trying to rollback a weblogic patch in patch fs ( fs2 ) that was previously applied to fs1 in Production Oracle E-Business Suite R12.2. So the patch history in the patch fs ( fs2) will still have file paths of fs1. So when you rollback a patch it will restore the files to RUN FS ( fs1). This can cause a havoc and  a big outage to online users.

As of now bsu utility doesn’t seem to handle issues this kind of situation. So all Oracle EBS Apps DBAs out there, make sure to check the below file for correct paths, before you try to rollback any weblogic patch

$ vi $FMW_HOME/patch_wls1036/registry/patch-backup.xml

A permanent solution to this problem would be to update bsu patch utility to not restore or touch files outside the weblogic home. Hope Oracle Development notices this issue before it effects some customers EBS production instance.

email

Authors

Interested in working with Vasu? Schedule a tech call.

About the Author

Vasu Balla’s colleagues call him “Eagle Eye” for a reason – his diverse technical background enables him to view his clients’ systems from a 360-degree angle, giving him a higher level of understanding. Vasu is well known for being approachable, and he truly enjoys helping people. Even former colleagues reach out to Vasu when they are really stuck on an issue. When he isn’t working, Vasu can be found in the kitchen trying new recipes.

5 Comments. Leave new

Hi,

I have remove the patch you mentioned in preparation for applying patch 17572726 – FCX7.
It did not cause any issues. Here is the output:

./bsu.sh -remove -patchlist=YIJF -prod_dir=$ORACLE_HOME/wlserver_10.3 -verbose
Checking for conflicts…
No conflict(s) detected

Starting removal of Patch ID: YIJF
Restoring /oracle/TEST/fs1/FMW_Home/modules/com.bea.core.datasource6.binding_1.10.0.0.jar from /oracle/TEST/fs1/FMW_Home/patch_wls1036/backup/backup.jar
Restoring /oracle/TEST/fs1/FMW_Home/modules/com.bea.core.datasource_1.10.0.0.jar from /oracle/TEST/fs1/FMW_Home/patch_wls1036/backup/backup.jar
Restoring /oracle/TEST/fs1/FMW_Home/modules/com.bea.core.datasource6_1.10.0.0.jar from /oracle/TEST/fs1/FMW_Home/patch_wls1036/backup/backup.jar
Removing /oracle/TEST/fs1/FMW_Home/patch_wls1036/patch_jars/BUG13729611_103604.jar
Updating /oracle/TEST/fs1/FMW_Home/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path= ../../../patch_jars/AppsAdapter.jar ../../../patch_jars/bpm-infra.jar ../../../patch_jars/DBAdapter.jar ../../../patch_jars/dbws.jar ../../../patch_jars/jca-binding-api.jar ../../../patch_jars/BUG13729611_103604.jar ../../../patch_jars/BUG16083651_1036.jar ../../../patch_jars/com.bea.core.stax2_2.0.0.0_3-0-3.jar ../../../patch_jars/BUG14597598_1036.jar ../../../patch_jars/com.bea.core.management.core_2.9.0.1.jar ../../../patch_jars/BUG14272383_1036.jar ../../../patch_jars/BUG13845626_1036.jar
New manifest value: Class-Path= ../../../patch_jars/AppsAdapter.jar ../../../patch_jars/bpm-infra.jar ../../../patch_jars/DBAdapter.jar ../../../patch_jars/dbws.jar ../../../patch_jars/jca-binding-api.jar ../../../patch_jars/BUG13729611_103604.jar ../../../patch_jars/BUG16083651_1036.jar ../../../patch_jars/com.bea.core.stax2_2.0.0.0_3-0-3.jar ../../../patch_jars/BUG14597598_1036.jar ../../../patch_jars/com.bea.core.management.core_2.9.0.1.jar ../../../patch_jars/BUG14272383_1036.jar ../../../patch_jars/BUG13845626_1036.jar
Result: Success

Regards,
Bashar

Reply

Hi Bashar,

Thanks for stopping by my blog. in your case, you are rollbacking the patch that was applied to same instance and FS previously. Lets consider below situation

1. applied weblogic patch to FS1 and cutover to FS1, and you are live with processes running on FS1
2. time passes by and you applied lot of more apps patches using online patching feature, so you flip between FS1 and FS2 lot time times
3. Now consider a situation where you are trying to rollback a weblogic patch in FS2, while you production services are running out of FS1
4. your FS2 weblogic patch history file will end up with entries of FS1. So when you rollback patch in FS2, it will overwrite files in FS1

I hope you got my point

Vasu

Reply

Hi Vasu,

I got your point.
To tell you the truth, the WebLogic patching is the most annoying thing I’ve ever seen in a product within Oracle EBS.
This problem made things even worse for me!

Regards,
Bashar

Reply

Hi Vasu,

I guess if we need to apply the mandatory patch 17893334 as per MOS: 1594274.1, cloning utilities will start picking up the right path for next executions. However as a part of post patch step, we need to manually check and edit the patch-backup.xml file if required.

Thanks,
Pavan

Reply

I got chance to work with Vasu truely outstanding guy Personally and professionally . I have never seen a person with such troubleshooting skills.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *