During RAC online patching, GoldenGate resource is required to be relocated to surviving node.
When relocation fails, there are a few logs which can be useful for troubleshooting.
Oracle Grid Infrastructure Standalone Agents for Oracle Clusterware (XAG) was not installed and using existing version from $GRID_HOME.
Unfortunately, I don’t have the exact version and would suspect the version is probably same as:
[[email protected] ~]$ $GRID_HOME/bin/agctl query releaseversion The Oracle Grid Infrastructure Agents release version is 3.1.0 [[email protected] ~]$ $GRID_HOME/bin/srvctl -V srvctl version: 12.1.0.2.0 [[email protected] ~]$
The name of the GoldenGate instance/resource (instance_name): gg_xx
GoldenGate resources can be determined using the following command:
$GRID_HOME/bin/crsctl stat res -t|grep -A2 xag
xag.gg_xx-vip.vip
1 ONLINE ONLINE racnode-dc1-2 STABLE
xag.gg_xx.goldengate
1 ONLINE ONLINE racnode-dc1-2 STABLE
Here is the output for GoldenGate relocation and can be performed from any node.
$GRID_HOME/bin/agctl relocate goldengate gg_xx --node racnode-dc1-1 CRS-2673: Attempting to stop 'xag.gg_xx.goldengate' on 'racnode-dc1-2' CRS-2677: Stop of 'xag.gg_xx.goldengate' on 'racnode-dc1-2' succeeded CRS-2673: Attempting to stop 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' CRS-2677: Stop of 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' succeeded CRS-2672: Attempting to start 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' CRS-2676: Start of 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' succeeded CRS-2672: Attempting to start 'xag.gg_xx.goldengate' on 'racnode-dc1-1' CRS-2674: Start of 'xag.gg_xx.goldengate' on 'racnode-dc1-1' failed CRS-2679: Attempting to clean 'xag.gg_xx.goldengate' on 'racnode-dc1-1' CRS-2681: Clean of 'xag.gg_xx.goldengate' on 'racnode-dc1-1' succeeded CRS-2564: Failed to relocate resource 'xag.gg_xx.goldengate'. Will attempt to restore it on 'racnode-dc1-2' now. CRS-2673: Attempting to stop 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' CRS-2677: Stop of 'xag.gg_xx-vip.vip' on 'racnode-dc1-1' succeeded CRS-2672: Attempting to start 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' CRS-2676: Start of 'xag.gg_xx-vip.vip' on 'racnode-dc1-2' succeeded CRS-2672: Attempting to start 'xag.gg_xx.goldengate' on 'racnode-dc1-2' CRS-2676: Start of 'xag.gg_xx.goldengate' on 'racnode-dc1-2' succeeded CRS-4000: Command Relocate failed, or completed with errors. $GRID_HOME/bin/agctl status goldengate gg_xx Goldengate instance 'gg_xx' is running on racnode-dc1-2
Here are the logs to check in order of personal preference.
Check GoldenGate Error Log:
$GG_HOME/ggserr.log
Check XAG Agent trace file: GoldenGate user is ggsuser
$ORACLE_BASE/diag/crs/`hostname -s`/crs/trace/crsd_scriptagent_ggsuser.trc
Check alert.log for crs:
$ORACLE_BASE/diag/crs/`hostname -s`/crs/trace/alert.log
In summary, there are multiple logs to check and when checking the right one first will make troubleshooting more efficient.
No comments