Changes to DBCA patch application behaviour causes PDB cloning to fail
Background
A test upgrade from 11g to 12c and conversion to a container and pluggable database recently pointed out some important 12c behavior differences with respect to the DBCA and whether or not it automatically applies PSUs installed in the Oracle Home. The original objective was to take an existing 11.2.0.4 database and upgrade it to 12.1.0.2 and convert it to a PDB. From a high level the procedure was:- Install the Oracle 12.1.0.2 software and apply the latest PSU (in this case the JAN2016 PSU).
- Create a new CDB to house the upgraded database.
- Upgrade the 11.2.0.4 database to 12.1.0.2 in-place using the DBUA.
- Convert the upgraded 12c database to a PDB (via the clone through DB link method).
Root Cause: DBCA no longer automatically applies PSUs
It's obvious from the ORA-04045 error that the issue is related to patching. But the question is still what was missed in the process since the 12c Oracle Home was fully patched before creating or upgrading any databases? The problem is that DBAs have maybe become complacent with respect to PSU applications after creating databases. With Oracle Database 11g whenever we created a database via the DBCA, the latest PSU was automatically applied. It doesn't matter if we created the database from a template or used a custom install. Regardless of which DBCA method was used, after DB creation we'd see something similar to: [sql] SQL> select comments, action_time from dba_registry_history 2 where bundle_series like '%PSU' order by 2; COMMENTS ACTION_TIME ------------------------------ ------------------------------ PSU 11.2.0.4.160119 04-MAR-16 02.43.52.292530 PM SQL> [/sql] Clearly the latest PSU (JAN2016 in this case) installed in the Oracle Home was applied automatically by the DBCA. And of course this is reflected in the official README documentation (in this example for DB PSU patch 21948347 [JAN2016] - requires a My Oracle Support login to view) which states:There are no actions required for databases that have been upgraded or created after installation of PSU 11.2.0.4.160119.However this functionality has completely changed with Oracle Database 12c! The change in behaviour is documented in My Oracle Support (MOS) Note: " 12.1:DBCA (Database Creation) does not execute "datapatch" (Doc ID 2084676.1)" which states:
DBCA does not execute datapatch in Oracle 12.1.0.X. The solution is to apply the SQL changes manually after creating a new DatabaseSimilarly the 12c JAN2016 DB PSU (patch 21948354) README documentation states:
You must execute the steps in Section 3.3.2, "Loading Modified SQL Files into the Database" for any new or upgraded database.This is a significant change in behaviour and is the root cause of the PDB creation error!
Resolving the "ORA-00600 [kql_tab_diana:new dep]" error
Back to the CDB creation error, the first logical place to check whenever experiencing plug-in or PDB creation errors is the PDB_PLUG_IN_VIOLATIONS view: [sql] SQL> CREATE PLUGGABLE DATABASE MY_PROD FROM NON$CDB@clone_link FILE_NAME_CONVERT=('/u01/app/oracle/oradata/MY_PROD','/u01/app/oracle/oradata/CPRD1/MY_PROD'); Pluggable database created. SQL> SELECT name, type, status, message, action FROM pdb_plug_in_violations ORDER BY 1,2; NAME TYPE STATUS MESSAGE ACTION -------- --------- --------- ---------------------------------------- ---------------------------------------- MY_PROD ERROR PENDING PDB plugged in is a non-CDB, requires no Run noncdb_to_pdb.sql. ncdb_to_pdb.sql be run. MY_PROD WARNING PENDING CDB parameter compatible mismatch: Previ Please check the parameter in the curren ous '11.2.0.4.0' Current '12.1.0.2.0' t CDB MY_PROD WARNING PENDING Service name or network name of service Drop the service and recreate it with an MY_PROD in the PDB is invalid or conflic appropriate name. ts with an existing service name or netw ork name in the CDB. SQL> [/sql] Nothing there is really concerning yet. It's pretty much what we'd expect to see at this point. However, taking the next step in the PDB clone process encounters the error: [sql] SQL> ALTER SESSION SET CONTAINER=MY_PROD; Session altered. SQL> @$ORACLE_HOME/rdbms/admin/noncdb_to_pdb.sql ... ... SQL> DECLARE 2 threads pls_integer := &&1; 3 BEGIN 4 utl_recomp.recomp_parallel(threads); 5 END; 6 / DECLARE * ERROR at line 1: ORA-04045: errors during recompilation/revalidation of SYS.DBMS_SQLPATCH ORA-00600: internal error code, arguments: [kql_tab_diana:new dep], [0x062623070], [0x7FB582065DE0], [1], [2], [], [], [], [], [], [], [] ORA-06512: at "SYS.DBMS_UTILITY", line 1294 ORA-06512: at line 1 Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options [/sql] Now looking in the PDB_PLUG_IN_VIOLATIONS view the error is evident: [sql] SQL> SELECT name, type, status, message, action FROM pdb_plug_in_violations ORDER BY 1,2; NAME TYPE STATUS MESSAGE ACTION -------- --------- --------- -------------------------------------------------- -------------------------------------------------- MY_PROD ERROR PENDING SQL patch ID/UID 22139226/19729684 (Database PSU 1 Call datapatch to install in the PDB or the CDB 2.1.0.2.160119, Oracle JavaVM Component (Jan2016)) : Installed in the PDB but not in the CDB. MY_PROD ERROR PENDING PDB plugged in is a non-CDB, requires noncdb_to_pd Run noncdb_to_pdb.sql. b.sql be run. MY_PROD WARNING RESOLVED Service name or network name of service MY_PROD in Drop the service and recreate it with an appropria the PDB is invalid or conflicts with an existing te name. service name or network name in the CDB. MY_PROD WARNING RESOLVED CDB parameter compatible mismatch: Previous '11.2. Please check the parameter in the current CDB 0.4.0' Current '12.1.0.2.0' MY_PROD WARNING PENDING Database option OLS mismatch: PDB installed versio Fix the database option in the PDB or the CDB n NULL. CDB installed version 12.1.0.2.0. MY_PROD WARNING PENDING Database option DV mismatch: PDB installed version Fix the database option in the PDB or the CDB NULL. CDB installed version 12.1.0.2.0. 6 rows selected. SQL> [/sql] At this point since the CDB clone has failed and since the noncdb_to_pdb.sql script cannot be run twice, the new PDB should be dropped. Resolving the root cause of the error by patching and then repeating the clone is necessary.Applying the PSU
Fortunately the fix is conceptually simple: apply the PSU patch into the database. Though the catch is that I actually had installed the "Combo of 12.1.0.2.160119 OJVM PSU and 12.1.0.2.160119 DB PSU (Jan 2016)" bundle patch (22191659) into the Oracle Home. This combo includes the DB PSU (patch 21948354) plus the OJVM PSU (patch 22139226). And while the DB PSU can be applied without outage, the OJVM patch cannot. Instead for the OJVM patch or the combo, the CDB and the PDBs must all be restarted in UPGRADE mode. Restarting in UPGRADE mode is fine in this case study where the CDB was just recently created to house the newly upgraded PDB. But if trying to plug the new database into an existing CDB with other applications running in production, shutting down the entire CDB to run datapatch may cause a problem. Following the README documentation for the just the JAN2016 DB PSU (patch 21948354) doesn't help. It states that the patch can be applied the database and pluggable databases open (section " 3.3.2 Loading Modified SQL Files into the Database"). However because I've installed the combo patch into the Oracle Home, trying to patch with the database open will cause the patching to fail: [sql] $ ./datapatch -verbose SQL Patching tool version 12.1.0.2.0 on Fri Mar 4 15:45:27 2016 Copyright (c) 2015, Oracle. All rights reserved. Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_3260_2016_03_04_15_45_27/sqlpatch_invocation.log Connecting to database...OK Note: Datapatch will only apply or rollback SQL fixes for PDBs that are in an open state, no patches will be applied to closed PDBs. Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1) Bootstrapping registry and package to current versions...done Determining current state...done Current state of SQL patches: Patch 22139226 (Database PSU 12.1.0.2.160119, Oracle JavaVM Component (Jan2016)): Installed in the binary registry only Bundle series PSU: ID 160119 in the binary registry and not installed in any PDB Adding patches to installation queue and performing prereq checks... Installation queue: For the following PDBs: CDB$ROOT PDB$SEED Nothing to roll back The following patches will be applied: 22139226 (Database PSU 12.1.0.2.160119, Oracle JavaVM Component (Jan2016)) 21948354 (Database Patch Set Update : 12.1.0.2.160119 (21948354)) Error: prereq checks failed! patch 22139226: The pluggable databases that need to be patched must be in upgrade mode Prereq check failed, exiting without installing any patches. Please refer to MOS Note 1609718.1 and/or the invocation log /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_3260_2016_03_04_15_45_27/sqlpatch_invocation.log for information on how to resolve the above errors. SQL Patching tool complete on Fri Mar 4 15:45:52 2016 $ [/sql] The solution to this error is to start the CDB and PDBs in UPGRADE mode (as per the OJVM patch documentation) and then re-run datapatch: [sql] SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. SQL> startup upgrade; ORACLE instance started. Total System Global Area 2097152000 bytes Fixed Size 2926320 bytes Variable Size 603982096 bytes Database Buffers 1476395008 bytes Redo Buffers 13848576 bytes Database mounted. Database opened. SQL> alter pluggable database all open upgrade; Pluggable database altered. SQL> exit Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options $ ./datapatch -verbose SQL Patching tool version 12.1.0.2.0 on Fri Mar 4 15:50:59 2016 Copyright (c) 2015, Oracle. All rights reserved. Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_5137_2016_03_04_15_50_59/sqlpatch_invocation.log Connecting to database...OK Note: Datapatch will only apply or rollback SQL fixes for PDBs that are in an open state, no patches will be applied to closed PDBs. Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1) Bootstrapping registry and package to current versions...done Determining current state...done Current state of SQL patches: Patch 22139226 (Database PSU 12.1.0.2.160119, Oracle JavaVM Component (Jan2016)): Installed in the binary registry only Bundle series PSU: ID 160119 in the binary registry and not installed in any PDB Adding patches to installation queue and performing prereq checks... Installation queue: For the following PDBs: CDB$ROOT PDB$SEED Nothing to roll back The following patches will be applied: 22139226 (Database PSU 12.1.0.2.160119, Oracle JavaVM Component (Jan2016)) 21948354 (Database Patch Set Update : 12.1.0.2.160119 (21948354)) Installing patches... Patch installation complete. Total patches installed: 8 Validating logfiles... Patch 22139226 apply (pdb CDB$ROOT): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/22139226/19729684/22139226_apply_CPRD1_CDBROOT_2016Mar04_15_51_23.log (no errors) Patch 21948354 apply (pdb CDB$ROOT): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/21948354/19553095/21948354_apply_CPRD1_CDBROOT_2016Mar04_15_51_24.log (no errors) Patch 22139226 apply (pdb PDB$SEED): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/22139226/19729684/22139226_apply_CPRD1_PDBSEED_2016Mar04_15_51_28.log (no errors) Patch 21948354 apply (pdb PDB$SEED): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/21948354/19553095/21948354_apply_CPRD1_PDBSEED_2016Mar04_15_51_29.log (no errors) SQL Patching tool complete on Fri Mar 4 15:51:31 2016 $ [/sql] Now retrying the CDB cloning process: [sql] SQL> ALTER SESSION SET CONTAINER=MY_PROD; Session altered. SQL> @$ORACLE_HOME/rdbms/admin/noncdb_to_pdb.sql ... ... SQL> alter session set container = "&pdbname"; Session altered. SQL> SQL> -- leave the PDB in the same state it was when we started SQL> BEGIN 2 execute immediate '&open_sql &restricted_state'; 3 EXCEPTION 4 WHEN OTHERS THEN 5 BEGIN 6 IF (sqlcode -900) THEN 7 RAISE; 8 END IF; 9 END; 10 END; 11 / PL/SQL procedure successfully completed. SQL> SQL> SQL> WHENEVER SQLERROR CONTINUE; SQL> ALTER PLUGGABLE DATABASE MY_PROD OPEN; Warning: PDB altered with errors. SQL> connect / as sysdba Connected. SQL> SELECT name, type, status, message, action FROM pdb_plug_in_violations ORDER BY 1,2; NAME TYPE STATUS MESSAGE ACTION -------- --------- --------- -------------------------------------------------- -------------------------------------------------- MY_PROD ERROR RESOLVED PDB plugged in is a non-CDB, requires noncdb_to_pd Run noncdb_to_pdb.sql. b.sql be run. MY_PROD ERROR PENDING PSU bundle patch 160119 (Database Patch Set Update Call datapatch to install in the PDB or the CDB : 12.1.0.2.160119 (21948354)): Installed in the C DB but not in the PDB. MY_PROD WARNING RESOLVED Service name or network name of service MY_PROD in Drop the service and recreate it with an appropria the PDB is invalid or conflicts with an existing te name. service name or network name in the CDB. MY_PROD WARNING PENDING Database option OLS mismatch: PDB installed versio Fix the database option in the PDB or the CDB n NULL. CDB installed version 12.1.0.2.0. MY_PROD WARNING PENDING Database option DV mismatch: PDB installed version Fix the database option in the PDB or the CDB NULL. CDB installed version 12.1.0.2.0. MY_PROD WARNING RESOLVED CDB parameter compatible mismatch: Previous '11.2. Please check the parameter in the current CDB 0.4.0' Current '12.1.0.2.0' 6 rows selected. SQL> [/sql] Note that first time the error was related to the OJVM PSU patch and stated that the PDB was patched but the CDB was not. Now after patching the CDB the error message states that the DB PSU patch is installed in the CDB but not the PDB. Again the solution is to run datapatch one more time. Fortunately since we're only patching a PDB, we no longer need to worry about starting the CDB and PDBs in UPGRADE mode to apply the OJVM patch. The OJVM patch does not apply to the PDBs. Hence we can patch successfully with both the CDB and PDBs open: [sql] $ ./datapatch -verbose SQL Patching tool version 12.1.0.2.0 on Fri Mar 4 16:19:06 2016 Copyright (c) 2015, Oracle. All rights reserved. Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_9245_2016_03_04_16_19_06/sqlpatch_invocation.log Connecting to database...OK Note: Datapatch will only apply or rollback SQL fixes for PDBs that are in an open state, no patches will be applied to closed PDBs. Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation (Doc ID 1585822.1) Bootstrapping registry and package to current versions...done Determining current state...done Current state of SQL patches: Patch 22139226 (Database PSU 12.1.0.2.160119, Oracle JavaVM Component (Jan2016)): Installed in binary and CDB$ROOT PDB$SEED MY_PROD Bundle series PSU: ID 160119 in the binary registry and ID 160119 in PDB CDB$ROOT, ID 160119 in PDB PDB$SEED Adding patches to installation queue and performing prereq checks... Installation queue: For the following PDBs: CDB$ROOT PDB$SEED Nothing to roll back Nothing to apply For the following PDBs: MY_PROD Nothing to roll back The following patches will be applied: 21948354 (Database Patch Set Update : 12.1.0.2.160119 (21948354)) Installing patches... Patch installation complete. Total patches installed: 1 Validating logfiles... Patch 21948354 apply (pdb MY_PROD): SUCCESS logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/21948354/19553095/21948354_apply_CPRD1_MY_PROD_2016Mar04_16_19_31.log (no errors) SQL Patching tool complete on Fri Mar 4 16:19:32 2016 $ [/sql] And finally the cloned PDB can be opened successfully: [sql] SQL> ALTER PLUGGABLE DATABASE MY_PROD CLOSE; Pluggable database altered. SQL> ALTER PLUGGABLE DATABASE MY_PROD OPEN; Pluggable database altered. SQL> show pdbs CON_ID CON_NAME OPEN MODE RESTRICTED ---------- ------------------------------ ---------- ---------- 2 PDB$SEED READ ONLY NO 3 MY_PROD READ WRITE NO SQL> SELECT name, type, status, message, action FROM pdb_plug_in_violations ORDER BY 1,2; NAME TYPE STATUS MESSAGE ACTION -------- --------- --------- -------------------------------------------------- -------------------------------------------------- MY_PROD ERROR RESOLVED PDB plugged in is a non-CDB, requires noncdb_to_pd Run noncdb_to_pdb.sql. b.sql be run. MY_PROD ERROR RESOLVED PSU bundle patch 160119 (Database Patch Set Update Call datapatch to install in the PDB or the CDB : 12.1.0.2.160119 (21948354)): Installed in the C DB but not in the PDB. MY_PROD WARNING RESOLVED Service name or network name of service MY_PROD in Drop the service and recreate it with an appropria the PDB is invalid or conflicts with an existing te name. service name or network name in the CDB. MY_PROD WARNING PENDING Database option OLS mismatch: PDB installed versio Fix the database option in the PDB or the CDB n NULL. CDB installed version 12.1.0.2.0. MY_PROD WARNING PENDING Database option DV mismatch: PDB installed version Fix the database option in the PDB or the CDB NULL. CDB installed version 12.1.0.2.0. MY_PROD WARNING RESOLVED CDB parameter compatible mismatch: Previous '11.2. Please check the parameter in the current CDB 0.4.0' Current '12.1.0.2.0' 6 rows selected. SQL> [/sql] The warnings marked as "PENDING" can be safely ignored.Conclusion
What started out as an issue when cloning a non-CDB into a PDB led to some learning about patching with Oracle Database 12c. The most important take-away is that Oracle Database 12c introduces a change in behaviour when it comes to patch applications through the DBCA. This change is well documented in both the patch and MOS documents. So if a DBA reads through the documentation thoroughly, they won't have a problem. However if the DBA is used to doing things the "old way" and only skims through the documentation they may unexpectedly get caught with errors such as the ORA-00600 encountered when creating a PDB through cloning.References
My Oracle Support (MOS) Documents:- 12.1:DBCA(Database Creation) does not execute "datapatch" (Doc ID 2084676.1)
- How to Convert Non PDB to PDB Database in 12c - Testcase (Doc ID 2012448.1)
- How to migrate an existing pre12c database(nonCDB) to 12c CDB database ? (Doc ID 1564657.1)
- Complete Checklist for Upgrading to Oracle Database 12c Release 1 using DBUA (Doc ID 1516557.1)
