A simple command that saved me
So there I was, reviewing the environment to create an action plan for patching. To my surprise, I noticed the following when checking DBFS: [code] Oracle Instance not alive for sid "DBFS2" [/code] I will demonstrate the simple command I used to ensure that all services that are supposed to be running are running. What happened? It turns out there was a network outage and when starting resources, DBFS was missing.
It's much harder to know what are all the resources that need to be started for a complex environment with RAC, GoldenGate, DataGuard, ACFS, DBFS. crsctl stat res -t -w '((TARGET != ONLINE) or (STATE != ONLINE)'
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.xxxxxxxx.yy_users_dr.svc 1 OFFLINE OFFLINE STABLE 2 OFFLINE OFFLINE STABLE
--------------------------------------------------------------------------------
crsctl stat res -t -w '((TARGET != ONLINE) or (STATE != ONLINE))'
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
dbfs_mount OFFLINE ONLINE hostxxxx01 STABLE OFFLINE OFFLINE hostxxxx02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.dbfs.db 2 OFFLINE OFFLINE Instance Shutdown,STABLE ora.xxxxxxxx.yy_users_dr.svc 1 OFFLINE OFFLINE STABLE 2 OFFLINE OFFLINE STABLE xag.ggue_src.goldengate 1 OFFLINE ONLINE hostxxxx01 STABLE
--------------------------------------------------------------------------------
The above is not good because State=ONLINE but Target=OFFLINE for dbfs_mount at hostxxxx02.
It seems Oracle is seldom intuitive, as checking the condition (TARGET != STATE) does not return any results.
crsctl stat res -t -w '((TARGET != STATE))'
In conclusion, always check TARGET=STATE for CRS to avoid having services down that are supposed to be running. Alternatively, I suppose just restarting CRS versus starting each service individually might be better, provided resources are integrated with CRS.
