Troubleshooting RAC GoldenGate relocation

Posted in: Oracle, Technical Track

During RAC online patching, GoldenGate resource is required to be relocated to surviving node.

When relocation fails, there are a few logs which can be useful for troubleshooting.

Oracle Grid Infrastructure Standalone Agents for Oracle Clusterware (XAG) was not installed and using existing version from $GRID_HOME.

Unfortunately, I don’t have the exact version and would suspect the version is probably same as:

[oracle@racnode-dc1-1 ~]$ $GRID_HOME/bin/agctl query releaseversion
The Oracle Grid Infrastructure Agents release version is 3.1.0

[oracle@racnode-dc1-1 ~]$ $GRID_HOME/bin/srvctl -V
srvctl version: 12.1.0.2.0
[oracle@racnode-dc1-1 ~]$

The name of the GoldenGate instance/resource (instance_name): gg_xx

GoldenGate resources can be determined using the following command:

$GRID_HOME/bin/crsctl stat res -t|grep -A2 xag
xag.gg_xx-vip.vip
      1        ONLINE  ONLINE       racnode-dc1-2              STABLE
xag.gg_xx.goldengate
      1        ONLINE  ONLINE       racnode-dc1-2              STABLE

Here is the output for GoldenGate relocation and can be performed from any node.

$GRID_HOME/bin/agctl relocate goldengate gg_xx --node racnode-dc1-1

CRS-2673: Attempting to stop 'xag.gg_xx.goldengate' on 'racnode-dc1-2'
CRS-2677: Stop of 'xag.gg_xx.goldengate' on 'racnode-dc1-2' succeeded
CRS-2673: Attempting to stop 'xag.gg_xx-vip.vip' on 'racnode-dc1-2'
CRS-2677: Stop of 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' succeeded
CRS-2672: Attempting to start 'xag.gg_xx-vip.vip' on 'racnode-dc1-1'
CRS-2676: Start of 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' succeeded
CRS-2672: Attempting to start 'xag.gg_xx.goldengate' on 'racnode-dc1-1'
CRS-2674: Start of 'xag.gg_xx.goldengate' on 'racnode-dc1-1' failed
CRS-2679: Attempting to clean 'xag.gg_xx.goldengate' on 'racnode-dc1-1'
CRS-2681: Clean of 'xag.gg_xx.goldengate' on 'racnode-dc1-1' succeeded

CRS-2564: Failed to relocate resource 'xag.gg_xx.goldengate'. Will attempt to restore it on 'racnode-dc1-2' now.

CRS-2673: Attempting to stop 'xag.gg_xx-vip.vip' on 'racnode-dc1-1'
CRS-2677: Stop of 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' succeeded
CRS-2672: Attempting to start 'xag.gg_xx-vip.vip' on 'racnode-dc1-2'
CRS-2676: Start of 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' succeeded
CRS-2672: Attempting to start 'xag.gg_xx.goldengate' on 'racnode-dc1-2'
CRS-2676: Start of 'xag.gg_xx.goldengate' on 'racnode-dc1-2' succeeded
CRS-4000: Command Relocate failed, or completed with errors.

$GRID_HOME/bin/agctl status goldengate gg_xx
Goldengate instance 'gg_xx' is running on racnode-dc1-2

Here are the logs to check in order of personal preference.

Check GoldenGate Error Log:
$GG_HOME/ggserr.log

Check XAG Agent trace file: GoldenGate user is ggsuser
$ORACLE_BASE/diag/crs/`hostname -s`/crs/trace/crsd_scriptagent_ggsuser.trc

Check alert.log for crs:
$ORACLE_BASE/diag/crs/`hostname -s`/crs/trace/alert.log

In summary, there are multiple logs to check and when checking the right one first will make troubleshooting more efficient.

 

email

Author

Want to talk with an expert? Schedule a call with our team to get the conversation started.

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *