Truncates and sequences replication in Oracle GoldenGate

Posted in: Oracle, Technical Track

We use the terms DDL and DML in our DBA life with Oracle all the time. The first stands for Data Definition Language and it is about Data Definition changes. The latter is about manipulation with your data itself, and stands for Data Manipulation Language. When we speak about replication of our data by replication tools between Oracle databases, we generally either enable DDL, work only replicating DML, or do it for both together. In general, I would recommend replicating both DML and DDL just in case, to prevent the replication to be broken in case of unexpected structural changes in the replicated schemas. But in some cases you do not want to replicate all DDL or any DDL at all for certain reasons. I will discuss a couple of operations which are handled slightly different from pure DDL/DML changes in GoldenGate.

The first of them is truncate operation. In Oracle it is definitely DDL and you can see that.

orcl> select object_id,data_object_id,last_ddl_time from dba_objects where object_name='EMP_TEST';
       OBJECT_ID   DATA_OBJECT_ID LAST_DDL_TIME
---------------- ---------------- -----------------
	  148769	   148769 06/24/16 16:07:04
orcl> truncate table emp_test;
Table truncated.
orcl> select object_id,data_object_id,last_ddl_time from dba_objects where object_name='EMP_TEST';
       OBJECT_ID   DATA_OBJECT_ID LAST_DDL_TIME
---------------- ---------------- -----------------
	  148769	   148770 06/24/16 16:15:52
orcl>

It is clear that the object gets the new data_object_id and last_ddl_time shows new time.

There is a general assumption that you need to enable a DDL support to replicate truncates. But is this true? In reality you can replicate truncates (with some limitations) without full DDL support, and I want to show you how it can be done. What you need to do is setup a parameter GETTRUNCATES.
Let’s setup it on extract and see how it works.

Here is my extract parameter file:

[oracle@bigdatalite oggora]$ cat dirprm/trext.prm
extract trext
userid ogg, password AACAAAAAAAAAAAIARIXFKCQBMFIGFARA, BLOWFISH, ENCRYPTKEY DEFAULT
--RMTHOSTOPTIONS
RMTHOST bigdatalite, MGRPORT 7849
RMTTRAIL ./dirdat/tr, format release 11.2
GETTRUNCATES
TABLE trsource.*;

We don’t have DDL support and if we try to add a column on the source and put a value to that column our replicat on other side will be abended.

orcl> alter table trsource.emp add col1 varchar2(10) ;
Table altered.
orcl> update trsource.emp set col1='Test1' where empno=7499;
1 row updated.
orcl> commit;

And on the target side:

2016-06-27 13:51:47  INFO    OGG-01021  Oracle GoldenGate Delivery for Oracle, trrep.prm:  Command received from GGSCI: STATS.
2016-06-27 13:57:37  ERROR   OGG-01161  Oracle GoldenGate Delivery for Oracle, trrep.prm:  Bad column index (8) specified for table TRSOURCE.EMP, max columns = 8.
2016-06-27 13:57:37  ERROR   OGG-01668  Oracle GoldenGate Delivery for Oracle, trrep.prm:  PROCESS ABENDING.

You are going to get similar errors for other DDL operations but not for truncates.

orcl> truncate table trsource.emp;
Table truncated.
orcl>
GGSCI (bigdatalite.localdomain) 1> send trext, stats
Sending STATS request to EXTRACT TREXT ...
Start of Statistics at 2016-06-27 14:05:24.
Output to ./dirdat/tr:
Extracting from TRSOURCE.EMP to TRSOURCE.EMP:
*** Total statistics since 2016-06-27 14:05:07 ***
	Total inserts                   	           0.00
	Total updates                   	           0.00
	Total deletes                   	           0.00
	Total truncates                 	           1.00
	Total discards                  	           0.00
	Total operations                	           1.00

You can see that we have captured the truncate by our extract. Even our DDL support is disabled. What we need is to set up the same parameter GETTRUNCATES on replicat side. Why do we need to set it up explicitly? Because the default behaviour and parameter for GoldenGate is “IGNORETRUNCATES” for all processes. As result, the truncates will be applied to the target system.

We are setting our parameter on replicat side and see the result:

[oracle@bigdatalite ogg11ora]$ cat dirprm/trrep.prm
replicat trrep
--trace
DISCARDFILE ./dirdsc/discard.out, append
userid ogg@test, password AACAAAAAAAAAAAIARIXFKCQBMFIGFARA, BLOWFISH, ENCRYPTKEY DEFAULT
assumetargetdefs
--DDL include all
GETTRUNCATES
map trsource.emp, target trdest.emp;
[oracle@bigdatalite ogg11ora]$
GGSCI (bigdatalite.localdomain) 4> send trrep, stats
Sending STATS request to REPLICAT TRREP ...
Start of Statistics at 2016-06-27 14:08:40.
Replicating from TRSOURCE.EMP to TRDEST.EMP:
*** Total statistics since 2016-06-27 14:08:25 ***
	Total inserts                   	           0.00
	Total updates                   	           0.00
	Total deletes                   	           0.00
	Total truncates                 	           1.00
	Total discards                  	           0.00
	Total operations                	           1.00
test> select count(*) from trdest.emp;
	COUNT(*)
----------------
	       0
test>

We don’t need full DDL support if we want to replicate truncates only. Sometimes it may help us when we have workflow including truncates, but we don’t want to replicate all DDL commands for some reasons. Just keep in mind that it works with some limitations. You cannot replicate by using “truncate partition” for Oracle. It will require full DDL support.

The second thing I want to discuss in this topic is support for sequences values replication. Sometimes people assume that it requires DDL support, but this is not true. As matter of fact replicating of sequences values doesn’t require you to enable DDL support for your replication. Of course, you need full DDL replication support to replicate CREATE, ALTER, DROP, RENAME for sequences, but the values are replicated as DML.

To enable the replication of sequences you need to create a special user on source and target databases, add the user to the GGSCHEMA parameter to your .GLOBALS file, and run one script to create all necessary procedures in the newly created schema.
Let’s have a closer look. I have a user OGG I am using for connection and I plan to use the same user for sequence support.

Here is my .GLOBALS file:

[oracle@bigdatalite oggora]$ cat GLOBALS
GGSCHEMA OGG

The same I have on the target side:

[oracle@bigdatalite ogg11ora]$ cat GLOBALS
GGSCHEMA OGG

I ran the script sequence.sql on both sides.

orcl> @sequence.sql
Please enter the name of a schema for the GoldenGate database objects:
ogg
Setting schema name to OGG
UPDATE_SEQUENCE STATUS:
Line/pos				 Error
---------------------------------------- -----------------------------------------------------------------
No errors				 No errors
GETSEQFLUSH
Line/pos				 Error
---------------------------------------- -----------------------------------------------------------------
No errors				 No errors
SEQTRACE
Line/pos				 Error
---------------------------------------- -----------------------------------------------------------------
No errors				 No errors
REPLICATE_SEQUENCE STATUS:
Line/pos				 Error
---------------------------------------- -----------------------------------------------------------------
No errors				 No errors
STATUS OF SEQUENCE SUPPORT
--------------------------------------------------------------
SUCCESSFUL installation of Oracle Sequence Replication support
orcl>

And on the source side add primary key supplemental logging to the sys.seq$ table:

orcl> ALTER TABLE sys.seq$ ADD SUPPLEMENTAL LOG DATA (PRIMARY KEY) COLUMNS;
Table altered.
orcl>

You may have a look to the procedures created by the scripts:
SEQTRACE
GETSEQFLUSH
REPLICATESEQUENCE
UPDATESEQUENCE

These procedures enable interface to flush, update and replicate the sequences.

Now we are creating a sequence on the source and target with the same parameters.

orcl> create sequence trsource.empno_seq start with 8100;
Sequence created.
orcl>

Adding parameter SEQUENCE to our parameter file for extract:

[oracle@bigdatalite oggora]$ cat dirprm/trext.prm
extract trext
userid ogg, password AACAAAAAAAAAAAIARIXFKCQBMFIGFARA, BLOWFISH, ENCRYPTKEY DEFAULT
--RMTHOSTOPTIONS
RMTHOST bigdatalite, MGRPORT 7849
RMTTRAIL ./dirdat/tr, format release 11.2
--DDL include objname trsource.*
GETTRUNCATES
SEQUENCE tsource.*;
TABLE trsource.*;
[oracle@bigdatalite oggora]$

On the target we are creating the same sequence:

test> create sequence trdest.empno_seq start with 8100;
Sequence created.
test>
[oracle@bigdatalite ogg11ora]$ cat dirprm/trrep.prm
replicat trrep
--trace
DISCARDFILE ./dirdsc/discard.out, append
userid ogg@test, password AACAAAAAAAAAAAIARIXFKCQBMFIGFARA, BLOWFISH, ENCRYPTKEY DEFAULT
assumetargetdefs
--DDL include all
GETTRUNCATES
map trsource., target trdest.;

I made our sequences a bit different on purpose. Our source sequence had a slightly bigger current value than target:

orcl> select trsource.empno_seq.currval from dual;
	 CURRVAL
----------------
	    8102
orcl>
test> select trdest.empno_seq.currval from dual;
	 CURRVAL
----------------
	    8100
test>

What we need is to run command FLUSH SEQUENCE on our extract side:

GGSCI (bigdatalite.localdomain) 9> dblogin userid ogg@orcl,password AACAAAAAAAAAAAIARIXFKCQBMFIGFARA, BLOWFISH, ENCRYPTKEY DEFAULT
Successfully logged into database.
GGSCI (bigdatalite.localdomain as ogg@orcl) 3> flush sequence trsource.empno_seq
Successfully flushed 1 sequence(s) trsource.empno_seq.
GGSCI (bigdatalite.localdomain as ogg@orcl) 4>

And on target we can see:

test> select * from dba_sequences where SEQUENCE_NAME='EMPNO_SEQ';
SEQUENCE_OWNER		       SEQUENCE_NAME			     MIN_VALUE	      MAX_VALUE     INCREMENT_BY C O	   CACHE_SIZE	   LAST_NUMBER
------------------------------ ------------------------------ ---------------- ---------------- ---------------- - - ---------------- ----------------
TRDEST			       EMPNO_SEQ				     1 9999999999999999 	       1 N N		   20		  8143
test> select last_number from dba_sequences where SEQUENCE_NAME='EMPNO_SEQ';
     LAST_NUMBER
----------------
	    8143
test>

The last number for the sequence on the target has been increased to 8143 when on the source we have only cache was flushed and we got 8123 as a last number for the sequence:

orcl> select last_number from dba_sequences where SEQUENCE_NAME='EMPNO_SEQ';
     LAST_NUMBER
----------------
	    8123
orcl>

Let’s try to get new values for the sequence.

orcl> select trsource.empno_seq.nextval from dual;
	 NEXTVAL
----------------
	    8104
orcl> select trsource.empno_seq.nextval from dual;
	 NEXTVAL
----------------
	    8105

We continue to increase values on the source, and as soon as we crossed max number for the source (8123) we got new value on the target:

orcl> select trsource.empno_seq.nextval from dual;
	 NEXTVAL
----------------
	    8119
........
	 NEXTVAL
----------------
	    8124
orcl>
test> select last_number from dba_sequences where SEQUENCE_NAME='EMPNO_SEQ';
     LAST_NUMBER
----------------
	    8144
test>

And the statistics on the target will be shown as updates:

GGSCI (bigdatalite.localdomain) 1> send trrep, stats
Sending STATS request to REPLICAT TRREP ...
Start of Statistics at 2016-06-29 13:20:36.
Replicating from TRSOURCE.EMPNO_SEQ to TRDEST.EMPNO_SEQ:
*** Total statistics since 2016-06-29 13:10:52 ***
	Total updates                   	           4.00
	Total discards                  	           0.00
	Total operations                	           4.00

We can see that the two operations are a bit different from all other standard DDL and DML in Oracle GoldenGate. I hope this small piece of information may help you in your implementation, or help to support your GoldenGate environment.

Stay tuned and keep your eyes on Pythian blog.

email

Author

Interested in working with Gleb? Schedule a tech call.

About the Author

Regarded by his peers as an Oracle guru, Gleb is known for being able to resolve any problem related to Oracle. He loves the satisfaction of troubleshooting, and his colleagues even say that seeking Gleb’s advice regarding an issue is more efficient than looking it up. Gleb enjoys the variety of challenges he faces while working at Pythian, rather than working on the same thing every day. His areas of speciality include Oracle RAC, Exadata, RMAN, SQL tuning, high availability, storage, performance tuning, and many more. When he’s not working, running, or cycling, Gleb can be found reading.

9 Comments. Leave new

Hi Gleb,

Thank you for this post.

Did you try replicating Create User, Procedure and Tablespace using GoldenGate for Oracle? If yes then let me know too. Thanks!

Reply

Hi Gunny,
I did not test it in the current environment but ddl for users, procedures and tablespaces are supported. Just to keep in mind that you have a mapping condition to filter only certain operations. So, you need to consider it if you want to capture it.

Have a good holidays,
Gleb

Reply

Any idea how can we achieve it. I have posted my question here:
https://dba.stackexchange.com/questions/158503/create-user-tablespace-replication-in-oracle-goldengate

If you think you can be of any help I will really appreciate it.

Thanks in advance.

Reply

Hi Gleb,
In Oracle DB version 11.2.0.4. We want to bypass all DDLs so I think we need to do classic extract and non-integrated apply. The DB is heavy on sequence usages. In this case, is it good to also replicate truncate and sequence as you show here?
I read that by default, 11.2.0.4 onward, Goldengate replication is by default integrated (which included DDL). If there ways to force non-integrated replication?

Reply
Gleb Otochkin
January 16, 2017 9:46 am

Hi,
Difference between classic and integrated extract and replicat mostly is how they work with messages and only partially how they track DDL (for OGG version 12.2). The decision about using classic vs integrated depends more about supported datatypes and database configuration and versions. As example classic extract may not support your datatype and in an integrated extract may have bit better integration with RAC. But decision about replicated objects and operations lays in your configuration for extract and replicat but not in a mode for your processes. So, I would check exactly what you want to replicat and verify you can do it with classic extract.
Regards,
Gleb

Reply

Hi Gleb,

We had enabled full DDL replication by executing the below scripts,

• @marker_setup.sql
• @ddl_setup.sql
• @role_setup.sql
• GRANT GGS_GGSUSER_ROLE TO ;
• @ddl_enable.sql

Now we created a sequence on source and was expecting the same to be created on target database but it’s not happening. The DDL replication is working perfect for all other DB objects but not for sequences. I had gone through your article and i see you had mentioned to execute @sequence.sql script and explicitly used SEQUENCE parameter in Extract, Pump & Replicat processes. But my understanding from your article is, it tells only if we have to replicate sequence values, is my understanding correct?

Now my question here is what should we do to replicate CREATE/ALTER/DROP of sequences + Sequence values. It would be of great help for a seeker who wants to excel in Golden Gate and I really appreciate your knowledge sharing thought.

Reply

Hi Gleb,

We had configured full DDL replication by executing below scripts,

• @marker_setup.sql
• @ddl_setup.sql
• @role_setup.sql
• GRANT GGS_GGSUSER_ROLE TO USER;
• @ddl_enable.sql

Now when we tried creating sequence on source DB we were expecting the same to be created automatically on Target DB but it’s not happening. DDL replication of different objects is happening perfectly but for sequences it’s not working.

I read this article of yours which is fantastic and i have one query regarding sequences. Now we have enabled Full DDL replication, do we need to run explicitly @sequence.sql script and use SEQUENCE parameter in GG Extract & Pump parameter files to capture the CREATE/DROP/ALTER/RENAME of sequences + Sequence values?

(or) Am i missing something here? Please can you shed some light on this and it will be definitely helpful.

Reply
Gleb Otochkin
May 22, 2018 11:49 am

Hi Kumar,
The DDL operation for a sequence require just a usual DDL support to be enabled. I don’t know what mapping parameters you use but for operations like create, drop sequence it should work altogether with other DDL.
The changes in values for the sequences are considered to be DML and require to execute sequence.sql which will create necessary procedures in the database and a “flush” procedure before a replication.

Thanks,
Gleb

Reply

Hi Gleb,
We do replication from Oracle on premises to Oracle AWS RDS with OGG 12.3.0 where OGG installed on EC2 hub and connecting to the source and target db. We having an issue on sequence value replication. AWS RDS doesn’t allow to run sequence.sql as it needs sysdba privilege. Any suggestion how to resolve sequence replication? Kindly appreciated.
Irina

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *