When news of the April 2010 PSU for Oracle 11.2 came out, I was excited to see it, since it marked the first non-one-off patch release for the 11.2 database software. I happened to have an 11gR2 test system running on 11gR2 ASM via standalone Grid Infrastructure. I applied PSU 9352237 to the DBMS home and fired it up, only to see the folly of my ways when any ASM file operations like disk resizing (or auto-extending) failed with ORA-1653. This was due to the DBMS component now having a higher version number than the ASM component, which ASM does not allow. The Grid Infrastucture PSU would need to be applied to bring the ASM component up to snuff, but that patch (9343627) was, at that time, only “announced” with no ETA. Alas, the patch was rolled back and we continued testing without it.
Then this week I checked again and saw that PSU 9343627 was released and gave it a whirl. I was a little confused when the README seemed to contain a lot of instructions that always assumed it to be on a clustered, RAC install. My setup was a single-instance Grid Infrastructure installation just to provide ASM. I soon met problem upon problem when going through first this setup step:
# GRID_HOME/crs/install/rootcrs.pl -unlock 2010-05-03 11:40:42: Parsing the host name 2010-05-03 11:40:42: Checking for super user privileges 2010-05-03 11:40:42: User has super user privileges Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params CRS-4013: This command is not supported in a single-node configuration. CRS-4000: Command Stop failed, or completed with errors. You must kill crs processes or reboot the system to properly cleanup the processes started by Oracle clusterware The Oracle Clusterware stack failed to stop. You should stop the stack with 'crsctl stop crs' and rerun the command
I decided to try the PSU anyway, thinking Oracle would certainly have noted in the README if this was a problem for single-node installations. Turns out I was wrong:
$ GRID_HOME/OPatch/opatch napply -local -oh GRID_HOME -id 9343627,9352237 Invoking OPatch 18.104.22.168.2 ... snip ... Running prerequisite checks... Prerequisite check "CheckApplicable" failed. The details are: Patch 9343627: Copy Action: Desctination File "GRID_HOME/bin/oradnssd" is not writeable. 'oracle.crs, 22.214.171.124.0': Cannot copy file from 'oradnssd' to 'GRID_HOME/bin/oradnssd' UtilSession failed: Prerequisite check "CheckApplicable" failed. OPatch failed with error code 73
I created an SR with Oracle support, and after reviewing my case agreed that the instructions were not right for single-instance Grid Infrastructure installations. They will be updating the README, but in the meantime I’d like to share the revised instructions that got me through two successful GI/DBMS PSU applications today.
First a few naming conventions to avoid confusion:
- DBMS_HOME – The Oracle DBMS installation directory.
- GRID_HOME – The Oracle Grid Infrastructure installation directory.
I’ve replaced all references to the directories in my examples with DBMS_HOME and GRID_HOME. Whenever you see those strings, substitute the actual path to your DBMS install or GRID install, whichever the case may be.
Download the PSU and OPatch 11.2
First, you’ll obviously need to download patch 9343627 that contains the PSU for your platform. In my case that’s Linux x86_64. As of this writing, I believe it’s only available for Linux x86 and x86_64. Note that patch 9343627 also includes patch 9352237. You do not have to, and should not, download that PSU separately. Everything you need is in the 9343627 download.
Second, choose a working directory to unzip that file. In my case, this is
/home/oracle/software/11gR2/psu/gi, which looked like this after unzipping the PSU zip file:
$ ls 9343627 9352237 p9343627_112010_Linux-x86-64.zip
Unlike normal one-off patches, where you would descend into the individual patch-number-named directories, you’ll stay at this level.
Note also that you’ll need OPatch 11.2 or higher. See MOS Doc ID 6880880 and get the latest version of OPatch for your platform. Then just unzip the file in the DBMS_HOME and GRID_HOME directories, overwriting the old OPatch directory.
First I’m assuming that you have shut down all DBMS instances on this server. We will shut down ASM later in this process, so be sure to shutdown the DBMS instances now.
Now we’ll start the pre-patch steps. First we stop the database on this server, saving the state of the database configuration in /tmp/oracle_home.stat:
$ DBMS_HOME/bin/srvctl stop home -o DBMS_HOME -s /tmp/oracle_home.stat
Repeat this for any additional Oracle database home installs on the server.
Next we save the database home configuration. This only needs to be done if we’re also going to patch the DBMS home in addition to the GI home. This is done from the directory where we unzipped the PSU:
$ ./9343627/custom/server/9343627/custom/scripts/prepatch.sh -dbhome DBMS_HOME ./9343627/custom/server/9343627/custom/scripts/prepatch.sh completed successfully.
After that, we shut down ASM as one would normally do it:
$ . oraenv +ASM $ sqlplus / as sysasm SQL> shutdown immediate; SQL> quit
Then, as root, we unlock and shutdown HAS (not CRS, as the original README would have you do):
# GRID_HOME/crs/install/roothas.pl -unlock 2010-05-04 17:14:08: Checking for super user privileges 2010-05-04 17:14:08: User has super user privileges 2010-05-04 17:14:08: Parsing the host name Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'localhost' CRS-2673: Attempting to stop 'ora.cssd' on 'localhost' CRS-2677: Stop of 'ora.cssd' on 'localhost' succeeded CRS-2673: Attempting to stop 'ora.diskmon' on 'localhost' CRS-2677: Stop of 'ora.diskmon' on 'localhost' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'localhost' has completed CRS-4133: Oracle High Availability Services has been stopped. Successfully unlock GRID_HOME
Now we’re ready to do some patching.
Applying PSU 9343627 and 9352237 to Oracle 11.2 Grid Infrastructure
As I mentioned earlier, the PSU contains both 9343627 and 9352237, and it is advised that you apply both. I did so successfully with these commands:
$ GRID_HOME/OPatch/opatch napply -oh GRID_HOME -id 9343627 $ GRID_HOME/OPatch/opatch napply -oh GRID_HOME -id 9352237
For each one, you’ll go through the normal opatch [y|n] confirmation prompts, and also prompted for an email address. I entered in my MOS login email, and left the password blank (it is optional). The rest was automatic and finished without error for both cases.
Applying PSU 9343627 and 9352237 to Oracle 11.2 DBMS
Here is something slightly different when applying the 9343627 patch to the DBMS_HOME, we specify a subdirectory after the “napply” parameter:
$ DBMS_HOME/OPatch/opatch napply 9343627/custom/server/ -oh DBMS_HOME -id 9343627
For patch 9352237, we use the regular syntax:
$ DBMS_HOME/OPatch/opatch napply -oh DBMS_HOME -id 9352237
Note that I use OPatch from DBMS_HOME when patching DBMS_HOME, and from GRID_HOME when patching GRID_HOME. I can’t say if it is actually necessary, but I like to keep things tidy. This is why I advised installing the new OPatch into both homes earlier.
As of this moment, both homes are patched, but we aren’t done yet!
First we run the postpatch.sh script included in patch 9343627 to reset some file permissions on the DBMS_HOME:
$ ./9343627/custom/server/9343627/custom/scripts/postpatch.sh -dbhome DBMS_HOME
Now we re-lock the GRID_HOME and restart the GI stack. As root:
# GRID_HOME/crs/install/roothas.pl -patch 2010-05-04 17:33:41: Checking for super user privileges 2010-05-04 17:33:41: User has super user privileges 2010-05-04 17:33:41: Parsing the host name Using configuration parameter file: GRID_HOME/crs/install/crsconfig_params CRS-4123: Oracle High Availability Services has been started.
Verify that the services we need are started (again as root):
# crsctl check has CRS-4638: Oracle High Availability Services is online # crsctl check css CRS-4529: Cluster Synchronization Services is online
If you don’t see CSS started, then you aren’t alone. On both of my servers, CSS was not set to auto-start from OHASD. This was fixed by running these commands as the GRID_HOME owner (oracle):
$ crsctl modify resource "ora.cssd" -attr "AUTO_START=1" $ crsctl modify resource "ora.diskmon" -attr "AUTO_START=1"
Then, as root:
# crsctl stop has CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'localhost' CRS-2673: Attempting to stop 'ora.cssd' on 'localhost' CRS-2677: Stop of 'ora.cssd' on 'localhost' succeeded CRS-2673: Attempting to stop 'ora.diskmon' on 'localhost' CRS-2677: Stop of 'ora.diskmon' on 'localhost' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'localhost' has completed CRS-4133: Oracle High Availability Services has been stopped. # crsctl start has CRS-4123: Oracle High Availability Services has been started. # crsctl check has CRS-4638: Oracle High Availability Services is online # crsctl check css CRS-4529: Cluster Synchronization Services is online
Moving on in the post-patch steps, we start the DBMS_HOME from the state file we created earlier:
$ DBMS_HOME/bin/srvctl start home -o DBMS_HOME -s /tmp/oracle_home.stat
At this point, we are done. You can start ASM and the DBMS instances. For some affirmation, run this for both GRID_HOME and DBMS_HOME (just using GRID_HOME in this example)
$ GRID_HOME/OPatch/opatch lsinventory Invoking OPatch 126.96.36.199.2 Oracle Interim Patch Installer version 188.8.131.52.2 Copyright (c) 2010, Oracle Corporation. All rights reserved. Oracle Home : GRID_HOME Central Inventory : /opt/oracle/app/oraInventory from : /etc/oraInst.loc OPatch version : 184.108.40.206.2 OUI version : 220.127.116.11.0 OUI location : GRID_HOME/oui Log file location : GRID_HOME/cfgtoollogs/opatch/opatch2010-05-04_18-36-23PM.log Patch history file: GRID_HOME/cfgtoollogs/opatch/opatch_history.txt Lsinventory Output file location : GRID_HOME/cfgtoollogs/opatch/lsinv/lsinventory2010-05-04_18-36-23PM.txt -------------------------------------------------------------------------------- Installed Top-level Products (1): Oracle Grid Infrastructure 18.104.22.168.0 There are 1 products installed in this Oracle Home. Interim patches (2) : Patch 9352237 : applied on Tue May 04 18:22:41 EDT 2010 Unique Patch ID: 12381846 Created on 25 Mar 2010, 00:05:17 hrs PST8PDT Bugs fixed: 8661168, 8769239, 8898852, 8801119, 9054253, 8706590, 8725286, 8974548 8778277, 8780372, 8769569, 9027691, 9454036, 9454037, 9454038, 8761974 7705591, 8496830, 8702892, 8639114, 8723477, 8729793, 8919682, 8818983 9001453, 8475069, 9328668, 8891929, 8798317, 8820324, 8733749, 8702535 8565708, 9036013, 8735201, 8684517, 8870559, 8773383, 8933870, 8812705 8405205, 8822365, 8813366, 8761260, 8790767, 8795418, 8913269, 8897784 8760714, 8717461, 8671349, 8775569, 8898589, 8861700, 8607693, 8642202 8780281, 9369797, 8780711, 8784929, 8834636, 9015983, 8891037, 8828328 8570322, 8832205, 8665189, 8717031, 8685253, 8718952, 8799099, 8633358 9032717, 9321701, 8588519, 8783738, 8796511, 8782971, 8756598, 9454385 8856497, 8703064, 9066116, 9007102, 8721315, 8818175, 8674263, 9352237 8753903, 8720447, 9057443, 8790561, 8733225, 9197917, 8928276, 8991997, 8837736 Patch 9343627 : applied on Tue May 04 18:18:31 EDT 2010 Unique Patch ID: 12381846 Created on 15 Apr 2010, 11:28:38 hrs PST8PDT Bugs fixed: 9343627, 9262748, 9262722 -------------------------------------------------------------------------------- OPatch succeeded.
As I said, Oracle support will be updating the README for PSU 9343627, but in the meantime I hope this guide helps you as much as it helped me. I’d like to thank my Pythian colleague Alex Gorbachev for his help in diagnosing some of the GI problems after the first broken patching, and also Esteban B. at Oracle Support for working closely with us to get a new single-node action plan.
Thanks Don, this has been a great help. We ran into exactly the same issues when trying to apply 11.2 April 2010 PSU to single-instance grid infrastructure on one of our key user test databases. We have had an SR open with Oracle for 5.5 days, but it was your blog that provided the solution.
Patrick, glad I could help!
Fantastic workaround, thank you.
That was awesome.
But my v$version still say 22.214.171.124
think i need to run the catbundle.sql…need to look that up.
that got cut off
it says 126.96.36.199.0 sorry….
Thanks! I had been trying to get this PSU installed for 2 days and i was about to give up until i found your post.
Just one question though, do i need to run the catbundle.pl script after the installation as the original read me requires for each database?
[…] leave a comment » Note: This post originally appeared at The Pythian Group blog. […]
can anybody give me a link to psu demo patch pls no surveys though
Reading the Patch 12311357 – 188.8.131.52.2 GI Patch Set Update (Includes Database PSU 184.108.40.206.2) readme document.
I don’t understand why should I unzip the GI PSU 220.127.116.11.2 patch in a shared location since I have installed the cluster in different Grid Infrastructure homes.
Is it ok to unzip the patch in each cluster node and update the nodes separately?
Extract of the read me file:
“The patch application requires explicit user actions to run ‘opatch auto’ command on each node of Oracle clusterware. So, it is recommended that you download and unzip the GI PSU 18.104.22.168.2 patch in a shared location to be able to access it from any node in the cluster and then as the Grid home owner execute the unzip command.”
You can unzip the patch in different locations. I have completed patching on RAC and it works
I’m afraid I can’t answer your question as I have no experience with clusters. Perhaps one of my colleagues will be able to answer this.
Thank you. Who is able to answer me?
Thanks, that was very helpful!
Thanks, mate – life saver. I’ve just used your example here to complete a similar apply process for CPU patch 9706490, where Oracle’s README.txt is still detailing step for RAC installations, and not single node GI.
Looks like this blog entry is still relevant for 22.214.171.124 GI PSU 4 12827731. Thanks Don.
Excellent post Don, Oracle have updated their readme but it’s hardly clear. Since when has the GI standalone home been referred to as a “Restart home” very confusing…
“If this is an Oracle Restart Home, as the root user execute:
# /crs/install/roothas.pl -unlock”
Thanks Mate!! It helped me big time thus saving my important time for the same. I ran into the same issue and it was your blog that saved my day.
Thanks mate! Life saver post!
Thanks Bro!!! it worked for me … In case you have any notes for setting up 11gr2 RAC on vmware please share the link would love to check it out…
Thanks Once again…
Thank you Sooooooooooo much. Its really helped me alot for my activity .
No words .. Thanks