Stop a CRS error with [Stop of ‘ora.crsd’ on ‘host02’ failed.]. I will demonstrate my attempts to troubleshoot and identify the cause for failure.
# crsctl stop crs
CRS-2675: Stop of 'dbfs_mount' on 'host02' failed CRS-2675: Stop of 'dbfs_mount' on 'host02' failed CRS-2673: Attempting to stop 'dbfs_mount' on 'host02' CRS-2675: Stop of 'dbfs_mount' on 'host02' failed CRS-2673: Attempting to stop 'dbfs_mount' on 'host02' CRS-2675: Stop of 'dbfs_mount' on 'host02' failed CRS-2799: Failed to shut down resource 'dbfs_mount' on 'host02' CRS-2799: Failed to shut down resource 'ora.GG_PROD.dg' on 'host02' CRS-2799: Failed to shut down resource 'ora.asm' on 'host02' CRS-2799: Failed to shut down resource 'ora.dbfs.db' on 'host02' CRS-2799: Failed to shut down resource 'ora.host02.ASM2.asm' on 'host02' CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'host02' has failed CRS-2675: Stop of 'ora.crsd' on 'host02' failed CRS-2799: Failed to shut down resource 'ora.crsd' on 'host02' CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'host02' has failed CRS-4687: Shutdown command has completed with errors. CRS-4000: Command Stop failed, or completed with errors.
DBFS is registered and managed by CRS:
$ $GRID_HOME/bin/crsctl stat res -w “TYPE = local_resource” -p
NAME=dbfs_mount TYPE=local_resource ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r-- ACTIONS= ACTION_SCRIPT=/u02/app/12.1.0/grid/crs/script/mount-dbfs.sh ACTION_TIMEOUT=60 AGENT_FILENAME=%CRS_HOME%/bin/scriptagent AUTO_START=restore CHECK_INTERVAL=30 CHECK_TIMEOUT=0 CLEAN_TIMEOUT=60 DEGREE=1 DELETE_TIMEOUT=60 DESCRIPTION= ENABLED=1 INSTANCE_FAILOVER=1 INTERMEDIATE_TIMEOUT=0 LOAD=1 LOGGING_LEVEL=1 MODIFY_TIMEOUT=60 OFFLINE_CHECK_INTERVAL=0 RESTART_ATTEMPTS=10 SCRIPT_TIMEOUT=300 SERVER_CATEGORY= START_CONCURRENCY=0 START_DEPENDENCIES=hard(ora.dbfs.db)pullup(ora.dbfs.db) START_TIMEOUT=0 STOP_CONCURRENCY=0 STOP_DEPENDENCIES=hard(ora.dbfs.db) STOP_TIMEOUT=0 UPTIME_THRESHOLD=1h USER_WORKLOAD=no NAME=dbfs_mount TYPE=local_resource ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r-- ACTIONS= ACTION_SCRIPT=/u02/app/12.1.0/grid/crs/script/mount-dbfs.sh ACTION_TIMEOUT=60 AGENT_FILENAME=%CRS_HOME%/bin/scriptagent AUTO_START=restore CHECK_INTERVAL=30 CHECK_TIMEOUT=0 CLEAN_TIMEOUT=60 DEGREE=1 DELETE_TIMEOUT=60 DESCRIPTION= ENABLED=1 INSTANCE_FAILOVER=1 INTERMEDIATE_TIMEOUT=0 LOAD=1 LOGGING_LEVEL=1 MODIFY_TIMEOUT=60 OFFLINE_CHECK_INTERVAL=0 RESTART_ATTEMPTS=10 SCRIPT_TIMEOUT=300 SERVER_CATEGORY= START_CONCURRENCY=0 START_DEPENDENCIES=hard(ora.dbfs.db)pullup(ora.dbfs.db) START_TIMEOUT=0 STOP_CONCURRENCY=0 STOP_DEPENDENCIES=hard(ora.dbfs.db) STOP_TIMEOUT=0 UPTIME_THRESHOLD=1h USER_WORKLOAD=no
Check /var/log/messagees for dbfs shows:
# grep -i dbfs /var/log/messages
Apr 17 19:42:26 host02 DBFS_/ggdata: unmounting DBFS from /ggdata Apr 17 19:42:26 host02 DBFS_/ggdata: umounting the filesystem using '/bin/fusermount -u /ggdata' Apr 17 19:42:26 host02 DBFS_/ggdata: Stop - stopped, but still mounted, error Apr 17 20:45:59 host02 DBFS_/ggdata: mount-dbfs.sh mounting DBFS at /ggdata from database DBFS Apr 17 20:45:59 host02 DBFS_/ggdata: /ggdata already mounted, use mount-dbfs.sh stop before attempting to start Apr 17 21:01:29 host02 DBFS_/ggdata: unmounting DBFS from /ggdata Apr 17 21:01:29 host02 DBFS_/ggdata: umounting the filesystem using '/bin/fusermount -u /ggdata' Apr 17 21:01:29 host02 DBFS_/ggdata: Stop - stopped, but still mounted, error Apr 17 21:01:36 host02 dbfs_client[71957]: OCI_ERROR 3114 - ORA-03114: not connected to ORACLE Apr 17 21:01:41 host02 dbfs_client[71957]: /FS1/dirdat/ih000247982 Block error RC:-5 Apr 17 21:03:06 host02 DBFS_/ggdata: unmounting DBFS from /ggdata Apr 17 21:03:06 host02 DBFS_/ggdata: umounting the filesystem using '/bin/fusermount -u /ggdata' Apr 17 21:03:06 host02 DBFS_/ggdata: Stop - stopped, now not mounted Apr 17 21:09:19 host02 DBFS_/ggdata: filesystem /ggdata not currently mounted, no need to stop Apr 17 22:06:16 host02 DBFS_/ggdata: mount-dbfs.sh mounting DBFS at /ggdata from database DBFS Apr 17 22:06:17 host02 DBFS_/ggdata: ORACLE_SID is DBFS2 Apr 17 22:06:17 host02 DBFS_/ggdata: doing mount /ggdata using SID DBFS2 with wallet now Apr 17 22:06:18 host02 DBFS_/ggdata: Start -- ONLINE
Check crsd_scriptagent_oracle.trc for all mount-dbfs.sh operations shows:
$ grep mount-dbfs.sh $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/crsd_scriptagent_oracle.trc|grep “2019-04-17 20:5”
2019-04-17 20:50:19.129630 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:50:49.129881 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:51:19.130437 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:51:49.130808 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:52:19.131105 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:52:49.131583 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:53:19.131895 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:53:49.132610 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:54:19.133075 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:54:49.133235 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:55:19.134053 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:55:49.127582 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:19.124667 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:43.313088 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop]
2019-04-17 20:56:43.415970 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:43.480595 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop]
2019-04-17 20:56:43.583726 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:43.644162 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop]
2019-04-17 20:56:43.746863 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:43.807101 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop]
2019-04-17 20:56:43.909821 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:49.133409 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:57:19.126737 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:57:49.127546 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:58:19.127902 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:58:49.128440 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:59:19.124916 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:59:49.128398 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
Check crsd_scriptagent_oracle.trc for specific time shows:
$ grep “2019-04-17 20:56” $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/crsd_scriptagent_oracle.trc
2019-04-17 20:56:02.793903 : AGFW:3274315520: {1:53477:37077} Agent received the message: AGENT_HB[Engine] ID 12293:16017523
2019-04-17 20:56:19.124667 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:19.176927 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Checking status now
2019-04-17 20:56:19.176973 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Check -- ONLINE
2019-04-17 20:56:32.794287 : AGFW:3274315520: {1:53477:37077} Agent received the message: AGENT_HB[Engine] ID 12293:16017529
2019-04-17 20:56:43.312534 : AGFW:3274315520: {2:37893:29307} Agent received the message: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017535
2019-04-17 20:56:43.312574 : AGFW:3274315520: {2:37893:29307} Preparing STOP command for: dbfs_mount host02 1
2019-04-17 20:56:43.312584 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: ONLINE to: STOPPING
2019-04-17 20:56:43.313088 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop]
2019-04-17 20:56:43.365201 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] unmounting DBFS from /ggdata
2019-04-17 20:56:43.415516 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] umounting the filesystem using '/bin/fusermount -u /ggdata'
2019-04-17 20:56:43.415541 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] /bin/fusermount: failed to unmount /ggdata: Device or resource busy
2019-04-17 20:56:43.415552 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Stop - stopped, but still mounted, error
2019-04-17 20:56:43.415611 : AGFW:3276416768: {2:37893:29307} Command: stop for resource: dbfs_mount host02 1 completed with status: FAIL
2019-04-17 20:56:43.415929 :CLSFRAME:3449863744: TM [MultiThread] is changing desired thread # to 3. Current # is 2
2019-04-17 20:56:43.415970 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:43.416033 : AGFW:3274315520: {2:37893:29307} Agent sending reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017535
2019-04-17 20:56:43.467939 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Checking status now
2019-04-17 20:56:43.467964 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Check -- ONLINE
2019-04-17 20:56:43.468295 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: STOPPING to: ONLINE
2019-04-17 20:56:43.468453 : AGFW:3274315520: {2:37893:29307} Agent sending last reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017535
2019-04-17 20:56:43.480145 : AGFW:3274315520: {2:37893:29307} Agent received the message: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017541
2019-04-17 20:56:43.480158 : AGFW:3274315520: {2:37893:29307} Preparing STOP command for: dbfs_mount host02 1
2019-04-17 20:56:43.480167 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: ONLINE to: STOPPING
2019-04-17 20:56:43.480595 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop]
2019-04-17 20:56:43.532854 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] unmounting DBFS from /ggdata
2019-04-17 20:56:43.583146 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] umounting the filesystem using '/bin/fusermount -u /ggdata'
2019-04-17 20:56:43.583173 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] /bin/fusermount: failed to unmount /ggdata: Device or resource busy
2019-04-17 20:56:43.583184 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Stop - stopped, but still mounted, error
2019-04-17 20:56:43.583240 : AGFW:3276416768: {2:37893:29307} Command: stop for resource: dbfs_mount host02 1 completed with status: FAIL
2019-04-17 20:56:43.583710 : AGFW:3274315520: {2:37893:29307} Agent sending reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017541
2019-04-17 20:56:43.583726 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:43.583819 :CLSFRAME:3449863744: TM [MultiThread] is changing desired thread # to 4. Current # is 3
2019-04-17 20:56:43.636166 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Checking status now
2019-04-17 20:56:43.636208 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Check -- ONLINE
2019-04-17 20:56:43.636531 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: STOPPING to: ONLINE
2019-04-17 20:56:43.636586 : AGFW:3274315520: {2:37893:29307} Agent sending last reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017541
2019-04-17 20:56:43.643596 : AGFW:3274315520: {2:37893:29307} Agent received the message: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017547
2019-04-17 20:56:43.643608 : AGFW:3274315520: {2:37893:29307} Preparing STOP command for: dbfs_mount host02 1
2019-04-17 20:56:43.643616 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: ONLINE to: STOPPING
2019-04-17 20:56:43.644162 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop]
2019-04-17 20:56:43.696015 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] unmounting DBFS from /ggdata
2019-04-17 20:56:43.746299 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] umounting the filesystem using '/bin/fusermount -u /ggdata'
2019-04-17 20:56:43.746319 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] /bin/fusermount: failed to unmount /ggdata: Device or resource busy
2019-04-17 20:56:43.746330 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Stop - stopped, but still mounted, error
2019-04-17 20:56:43.746394 : AGFW:3276416768: {2:37893:29307} Command: stop for resource: dbfs_mount host02 1 completed with status: FAIL
2019-04-17 20:56:43.746863 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:43.746936 : AGFW:3274315520: {2:37893:29307} Agent sending reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017547
2019-04-17 20:56:43.799233 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Checking status now
2019-04-17 20:56:43.799268 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Check -- ONLINE
2019-04-17 20:56:43.799596 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: STOPPING to: ONLINE
2019-04-17 20:56:43.799658 : AGFW:3274315520: {2:37893:29307} Agent sending last reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017547
2019-04-17 20:56:43.806684 : AGFW:3274315520: {2:37893:29307} Agent received the message: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017553
2019-04-17 20:56:43.806695 : AGFW:3274315520: {2:37893:29307} Preparing STOP command for: dbfs_mount host02 1
2019-04-17 20:56:43.806704 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: ONLINE to: STOPPING
2019-04-17 20:56:43.807101 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop]
2019-04-17 20:56:43.859029 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] unmounting DBFS from /ggdata
2019-04-17 20:56:43.909318 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] umounting the filesystem using '/bin/fusermount -u /ggdata'
2019-04-17 20:56:43.909344 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] /bin/fusermount: failed to unmount /ggdata: Device or resource busy
2019-04-17 20:56:43.909355 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Stop - stopped, but still mounted, error
2019-04-17 20:56:43.909410 : AGFW:3276416768: {2:37893:29307} Command: stop for resource: dbfs_mount host02 1 completed with status: FAIL
2019-04-17 20:56:43.909821 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:43.909871 : AGFW:3274315520: {2:37893:29307} Agent sending reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017553
2019-04-17 20:56:43.961670 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Checking status now
2019-04-17 20:56:43.961715 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Check -- ONLINE
2019-04-17 20:56:43.962013 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: STOPPING to: ONLINE
2019-04-17 20:56:43.962064 : AGFW:3274315520: {2:37893:29307} Agent sending last reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017553
2019-04-17 20:56:49.133409 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check]
2019-04-17 20:56:49.185415 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Checking status now
2019-04-17 20:56:49.185465 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Check -- ONLINE
While the actual process that was locking /ggdata was not determined at the time, fuser can be used to determine the process.
# fuser -mv /ggdata/
USER PID ACCESS COMMAND
/ggdata: root kernel mount /ggdata
ggsuser 64776 F.... extract
oracle 65049 f.... oracle_65049_ih
oracle 65084 F.... oracle_65084_ih
oracle 65094 F.... oracle_65094_ih
oracle 65139 F.... oracle_65139_ih
oracle 65740 F.... oracle_65740_ih
ggsuser 84987 F.... extract
ggsuser 84990 F.... extract
ggsuser 84994 F.... extract
ggsuser 84997 F.... extract
ggsuser 85000 F.... extract
ggsuser 85003 F.... extract
In conclusion, the following files are useful for investigation: crsd_scriptagent_oracle.trc /var/log/messages.
Since the errors were rather vague, fuser will identify the process causing the failure to unmount.
Stay tuned as I will share other options to resolve ‘failed to unmount” in a future post.
No comments