So there I was, reviewing the environment to create an action plan for patching. To my surprise, I noticed the following when checking DBFS:
Oracle Instance not alive for sid "DBFS2"
I will demonstrate the simple command I used to ensure that all services that are supposed to be running are running.
What happened? It turns out there was a network outage and when starting resources, DBFS was missing.
It’s much harder to know what are all the resources that need to be started for a complex environment with RAC, GoldenGate, DataGuard, ACFS, DBFS.
crsctl stat res -t -w ‘((TARGET != ONLINE) or (STATE != ONLINE)’
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.xxxxxxxx.yy_users_dr.svc
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
--------------------------------------------------------------------------------
From the command above, Target=State=OFFLINE which is good.
crsctl stat res -t -w ‘((TARGET != ONLINE) or (STATE != ONLINE)’
crsctl stat res -t -w '((TARGET != ONLINE) or (STATE != ONLINE)'
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
dbfs_mount
OFFLINE ONLINE hostxxxx01 STABLE
OFFLINE OFFLINE hostxxxx02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.dbfs.db
2 OFFLINE OFFLINE Instance Shutdown,STABLE
ora.xxxxxxxx.yy_users_dr.svc
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE STABLE
xag.ggue_src.goldengate
1 OFFLINE ONLINE hostxxxx01 STABLE
--------------------------------------------------------------------------------
The above is not good because State=ONLINE but Target=OFFLINE for dbfs_mount at hostxxxx02.
It seems Oracle is seldom intuitive as checking the condition (TARGET != STATE) does not return any results.
crsctl stat res -t -w '((TARGET != STATE)'
In conclusion, always check TARGET=STATE for CRS to avoid having services down that are supposed to be running.
Alternatively, I suppose just restarting CRS versus starting each service individually might be better, provided resources are integrated with CRS.
No comments