Troubleshooting RAC GoldenGate relocation
During RAC online patching, GoldenGate resource is required to be relocated to surviving node. When relocation fails, there are a few logs which can be useful for troubleshooting. Oracle Grid Infrastructure Standalone Agents for Oracle Clusterware (XAG) was not installed and using existing version from $GRID_HOME. Unfortunately, I don't have the exact version and would suspect the version is probably same as:
[oracle@racnode-dc1-1 ~]$ $GRID_HOME/bin/agctl query releaseversion The Oracle Grid Infrastructure Agents release version is 3.1.0 [oracle@racnode-dc1-1 ~]$ $GRID_HOME/bin/srvctl -V srvctl version: 12.1.0.2.0 [oracle@racnode-dc1-1 ~]$The name of the GoldenGate instance/resource (instance_name): gg_xx GoldenGate resources can be determined using the following command:
$GRID_HOME/bin/crsctl stat res -t|grep -A2 xag xag.gg_xx-vip.vip 1 ONLINE ONLINE racnode-dc1-2 STABLE xag.gg_xx.goldengate 1 ONLINE ONLINE racnode-dc1-2 STABLEHere is the output for GoldenGate relocation and can be performed from any node.
$GRID_HOME/bin/agctl relocate goldengate gg_xx --node racnode-dc1-1 CRS-2673: Attempting to stop 'xag.gg_xx.goldengate' on 'racnode-dc1-2' CRS-2677: Stop of 'xag.gg_xx.goldengate' on 'racnode-dc1-2' succeeded CRS-2673: Attempting to stop 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' CRS-2677: Stop of 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' succeeded CRS-2672: Attempting to start 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' CRS-2676: Start of 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' succeeded CRS-2672: Attempting to start 'xag.gg_xx.goldengate' on 'racnode-dc1-1' CRS-2674: Start of 'xag.gg_xx.goldengate' on 'racnode-dc1-1' failed CRS-2679: Attempting to clean 'xag.gg_xx.goldengate' on 'racnode-dc1-1' CRS-2681: Clean of 'xag.gg_xx.goldengate' on 'racnode-dc1-1' succeeded CRS-2564: Failed to relocate resource 'xag.gg_xx.goldengate'. Will attempt to restore it on 'racnode-dc1-2' now. CRS-2673: Attempting to stop 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' CRS-2677: Stop of 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' succeeded CRS-2672: Attempting to start 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' CRS-2676: Start of 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' succeeded CRS-2672: Attempting to start 'xag.gg_xx.goldengate' on 'racnode-dc1-2' CRS-2676: Start of 'xag.gg_xx.goldengate' on 'racnode-dc1-2' succeeded CRS-4000: Command Relocate failed, or completed with errors. $GRID_HOME/bin/agctl status goldengate gg_xx Goldengate instance 'gg_xx' is running on racnode-dc1-2Here are the logs to check in order of personal preference. Check GoldenGate Error Log: $GG_HOME/ggserr.log Check XAG Agent trace file: GoldenGate user is ggsuser $ORACLE_BASE/diag/crs/`hostname -s`/crs/trace/crsd_scriptagent_ggsuser.trc Check alert.log for crs: $ORACLE_BASE/diag/crs/`hostname -s`/crs/trace/alert.log In summary, there are multiple logs to check and when checking the right one first will make troubleshooting more efficient.
