How to troubleshoot a failure to mount DBFS
Stop a CRS error with [Stop of 'ora.crsd' on 'host02' failed.]. I will demonstrate my attempts to troubleshoot and identify the cause for failure. # crsctl stop crs [code] CRS-2675: Stop of 'dbfs_mount' on 'host02' failed CRS-2675: Stop of 'dbfs_mount' on 'host02' failed CRS-2673: Attempting to stop 'dbfs_mount' on 'host02' CRS-2675: Stop of 'dbfs_mount' on 'host02' failed CRS-2673: Attempting to stop 'dbfs_mount' on 'host02' CRS-2675: Stop of 'dbfs_mount' on 'host02' failed CRS-2799: Failed to shut down resource 'dbfs_mount' on 'host02' CRS-2799: Failed to shut down resource 'ora.GG_PROD.dg' on 'host02' CRS-2799: Failed to shut down resource 'ora.asm' on 'host02' CRS-2799: Failed to shut down resource 'ora.dbfs.db' on 'host02' CRS-2799: Failed to shut down resource 'ora.host02.ASM2.asm' on 'host02' CRS-2794: Shutdown of Cluster Ready Services-managed resources on 'host02' has failed CRS-2675: Stop of 'ora.crsd' on 'host02' failed CRS-2799: Failed to shut down resource 'ora.crsd' on 'host02' CRS-2795: Shutdown of Oracle High Availability Services-managed resources on 'host02' has failed CRS-4687: Shutdown command has completed with errors. CRS-4000: Command Stop failed, or completed with errors. [/code] DBFS is registered and managed by CRS: $ $GRID_HOME/bin/crsctl stat res -w "TYPE = local_resource" -p [code] NAME=dbfs_mount TYPE=local_resource ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r-- ACTIONS= ACTION_SCRIPT=/u02/app/12.1.0/grid/crs/script/mount-dbfs.sh ACTION_TIMEOUT=60 AGENT_FILENAME=%CRS_HOME%/bin/scriptagent AUTO_START=restore CHECK_INTERVAL=30 CHECK_TIMEOUT=0 CLEAN_TIMEOUT=60 DEGREE=1 DELETE_TIMEOUT=60 DESCRIPTION= ENABLED=1 INSTANCE_FAILOVER=1 INTERMEDIATE_TIMEOUT=0 LOAD=1 LOGGING_LEVEL=1 MODIFY_TIMEOUT=60 OFFLINE_CHECK_INTERVAL=0 RESTART_ATTEMPTS=10 SCRIPT_TIMEOUT=300 SERVER_CATEGORY= START_CONCURRENCY=0 START_DEPENDENCIES=hard(ora.dbfs.db)pullup(ora.dbfs.db) START_TIMEOUT=0 STOP_CONCURRENCY=0 STOP_DEPENDENCIES=hard(ora.dbfs.db) STOP_TIMEOUT=0 UPTIME_THRESHOLD=1h USER_WORKLOAD=no NAME=dbfs_mount TYPE=local_resource ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r-- ACTIONS= ACTION_SCRIPT=/u02/app/12.1.0/grid/crs/script/mount-dbfs.sh ACTION_TIMEOUT=60 AGENT_FILENAME=%CRS_HOME%/bin/scriptagent AUTO_START=restore CHECK_INTERVAL=30 CHECK_TIMEOUT=0 CLEAN_TIMEOUT=60 DEGREE=1 DELETE_TIMEOUT=60 DESCRIPTION= ENABLED=1 INSTANCE_FAILOVER=1 INTERMEDIATE_TIMEOUT=0 LOAD=1 LOGGING_LEVEL=1 MODIFY_TIMEOUT=60 OFFLINE_CHECK_INTERVAL=0 RESTART_ATTEMPTS=10 SCRIPT_TIMEOUT=300 SERVER_CATEGORY= START_CONCURRENCY=0 START_DEPENDENCIES=hard(ora.dbfs.db)pullup(ora.dbfs.db) START_TIMEOUT=0 STOP_CONCURRENCY=0 STOP_DEPENDENCIES=hard(ora.dbfs.db) STOP_TIMEOUT=0 UPTIME_THRESHOLD=1h USER_WORKLOAD=no [/code] Check /var/log/messagees for dbfs shows: # grep -i dbfs /var/log/messages [code] Apr 17 19:42:26 host02 DBFS_/ggdata: unmounting DBFS from /ggdata Apr 17 19:42:26 host02 DBFS_/ggdata: umounting the filesystem using '/bin/fusermount -u /ggdata' Apr 17 19:42:26 host02 DBFS_/ggdata: Stop - stopped, but still mounted, error Apr 17 20:45:59 host02 DBFS_/ggdata: mount-dbfs.sh mounting DBFS at /ggdata from database DBFS Apr 17 20:45:59 host02 DBFS_/ggdata: /ggdata already mounted, use mount-dbfs.sh stop before attempting to start Apr 17 21:01:29 host02 DBFS_/ggdata: unmounting DBFS from /ggdata Apr 17 21:01:29 host02 DBFS_/ggdata: umounting the filesystem using '/bin/fusermount -u /ggdata' Apr 17 21:01:29 host02 DBFS_/ggdata: Stop - stopped, but still mounted, error Apr 17 21:01:36 host02 dbfs_client[71957]: OCI_ERROR 3114 - ORA-03114: not connected to ORACLE Apr 17 21:01:41 host02 dbfs_client[71957]: /FS1/dirdat/ih000247982 Block error RC:-5 Apr 17 21:03:06 host02 DBFS_/ggdata: unmounting DBFS from /ggdata Apr 17 21:03:06 host02 DBFS_/ggdata: umounting the filesystem using '/bin/fusermount -u /ggdata' Apr 17 21:03:06 host02 DBFS_/ggdata: Stop - stopped, now not mounted Apr 17 21:09:19 host02 DBFS_/ggdata: filesystem /ggdata not currently mounted, no need to stop Apr 17 22:06:16 host02 DBFS_/ggdata: mount-dbfs.sh mounting DBFS at /ggdata from database DBFS Apr 17 22:06:17 host02 DBFS_/ggdata: ORACLE_SID is DBFS2 Apr 17 22:06:17 host02 DBFS_/ggdata: doing mount /ggdata using SID DBFS2 with wallet now Apr 17 22:06:18 host02 DBFS_/ggdata: Start -- ONLINE [/code] Check crsd_scriptagent_oracle.trc for all mount-dbfs.sh operations shows: $ grep mount-dbfs.sh $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/crsd_scriptagent_oracle.trc|grep "2019-04-17 20:5" [code] 2019-04-17 20:50:19.129630 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:50:49.129881 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:51:19.130437 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:51:49.130808 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:52:19.131105 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:52:49.131583 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:53:19.131895 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:53:49.132610 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:54:19.133075 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:54:49.133235 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:55:19.134053 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:55:49.127582 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:19.124667 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:43.313088 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop] 2019-04-17 20:56:43.415970 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:43.480595 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop] 2019-04-17 20:56:43.583726 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:43.644162 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop] 2019-04-17 20:56:43.746863 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:43.807101 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop] 2019-04-17 20:56:43.909821 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:49.133409 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:57:19.126737 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:57:49.127546 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:58:19.127902 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:58:49.128440 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:59:19.124916 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:59:49.128398 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] [/code] Check crsd_scriptagent_oracle.trc for specific time shows: $ grep "2019-04-17 20:56" $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace/crsd_scriptagent_oracle.trc [code] 2019-04-17 20:56:02.793903 : AGFW:3274315520: {1:53477:37077} Agent received the message: AGENT_HB[Engine] ID 12293:16017523 2019-04-17 20:56:19.124667 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:19.176927 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Checking status now 2019-04-17 20:56:19.176973 :CLSDYNAM:3276416768: [dbfs_mount]{1:53477:37077} [check] Check -- ONLINE 2019-04-17 20:56:32.794287 : AGFW:3274315520: {1:53477:37077} Agent received the message: AGENT_HB[Engine] ID 12293:16017529 2019-04-17 20:56:43.312534 : AGFW:3274315520: {2:37893:29307} Agent received the message: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017535 2019-04-17 20:56:43.312574 : AGFW:3274315520: {2:37893:29307} Preparing STOP command for: dbfs_mount host02 1 2019-04-17 20:56:43.312584 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: ONLINE to: STOPPING 2019-04-17 20:56:43.313088 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop] 2019-04-17 20:56:43.365201 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] unmounting DBFS from /ggdata 2019-04-17 20:56:43.415516 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] umounting the filesystem using '/bin/fusermount -u /ggdata' 2019-04-17 20:56:43.415541 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] /bin/fusermount: failed to unmount /ggdata: Device or resource busy 2019-04-17 20:56:43.415552 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Stop - stopped, but still mounted, error 2019-04-17 20:56:43.415611 : AGFW:3276416768: {2:37893:29307} Command: stop for resource: dbfs_mount host02 1 completed with status: FAIL 2019-04-17 20:56:43.415929 :CLSFRAME:3449863744: TM [MultiThread] is changing desired thread # to 3. Current # is 2 2019-04-17 20:56:43.415970 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:43.416033 : AGFW:3274315520: {2:37893:29307} Agent sending reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017535 2019-04-17 20:56:43.467939 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Checking status now 2019-04-17 20:56:43.467964 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Check -- ONLINE 2019-04-17 20:56:43.468295 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: STOPPING to: ONLINE 2019-04-17 20:56:43.468453 : AGFW:3274315520: {2:37893:29307} Agent sending last reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017535 2019-04-17 20:56:43.480145 : AGFW:3274315520: {2:37893:29307} Agent received the message: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017541 2019-04-17 20:56:43.480158 : AGFW:3274315520: {2:37893:29307} Preparing STOP command for: dbfs_mount host02 1 2019-04-17 20:56:43.480167 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: ONLINE to: STOPPING 2019-04-17 20:56:43.480595 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop] 2019-04-17 20:56:43.532854 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] unmounting DBFS from /ggdata 2019-04-17 20:56:43.583146 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] umounting the filesystem using '/bin/fusermount -u /ggdata' 2019-04-17 20:56:43.583173 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] /bin/fusermount: failed to unmount /ggdata: Device or resource busy 2019-04-17 20:56:43.583184 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Stop - stopped, but still mounted, error 2019-04-17 20:56:43.583240 : AGFW:3276416768: {2:37893:29307} Command: stop for resource: dbfs_mount host02 1 completed with status: FAIL 2019-04-17 20:56:43.583710 : AGFW:3274315520: {2:37893:29307} Agent sending reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017541 2019-04-17 20:56:43.583726 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:43.583819 :CLSFRAME:3449863744: TM [MultiThread] is changing desired thread # to 4. Current # is 3 2019-04-17 20:56:43.636166 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Checking status now 2019-04-17 20:56:43.636208 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Check -- ONLINE 2019-04-17 20:56:43.636531 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: STOPPING to: ONLINE 2019-04-17 20:56:43.636586 : AGFW:3274315520: {2:37893:29307} Agent sending last reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017541 2019-04-17 20:56:43.643596 : AGFW:3274315520: {2:37893:29307} Agent received the message: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017547 2019-04-17 20:56:43.643608 : AGFW:3274315520: {2:37893:29307} Preparing STOP command for: dbfs_mount host02 1 2019-04-17 20:56:43.643616 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: ONLINE to: STOPPING 2019-04-17 20:56:43.644162 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop] 2019-04-17 20:56:43.696015 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] unmounting DBFS from /ggdata 2019-04-17 20:56:43.746299 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] umounting the filesystem using '/bin/fusermount -u /ggdata' 2019-04-17 20:56:43.746319 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] /bin/fusermount: failed to unmount /ggdata: Device or resource busy 2019-04-17 20:56:43.746330 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Stop - stopped, but still mounted, error 2019-04-17 20:56:43.746394 : AGFW:3276416768: {2:37893:29307} Command: stop for resource: dbfs_mount host02 1 completed with status: FAIL 2019-04-17 20:56:43.746863 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:43.746936 : AGFW:3274315520: {2:37893:29307} Agent sending reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017547 2019-04-17 20:56:43.799233 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Checking status now 2019-04-17 20:56:43.799268 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Check -- ONLINE 2019-04-17 20:56:43.799596 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: STOPPING to: ONLINE 2019-04-17 20:56:43.799658 : AGFW:3274315520: {2:37893:29307} Agent sending last reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017547 2019-04-17 20:56:43.806684 : AGFW:3274315520: {2:37893:29307} Agent received the message: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017553 2019-04-17 20:56:43.806695 : AGFW:3274315520: {2:37893:29307} Preparing STOP command for: dbfs_mount host02 1 2019-04-17 20:56:43.806704 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: ONLINE to: STOPPING 2019-04-17 20:56:43.807101 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[stop] 2019-04-17 20:56:43.859029 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] unmounting DBFS from /ggdata 2019-04-17 20:56:43.909318 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] umounting the filesystem using '/bin/fusermount -u /ggdata' 2019-04-17 20:56:43.909344 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] /bin/fusermount: failed to unmount /ggdata: Device or resource busy 2019-04-17 20:56:43.909355 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [stop] Stop - stopped, but still mounted, error 2019-04-17 20:56:43.909410 : AGFW:3276416768: {2:37893:29307} Command: stop for resource: dbfs_mount host02 1 completed with status: FAIL 2019-04-17 20:56:43.909821 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:43.909871 : AGFW:3274315520: {2:37893:29307} Agent sending reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017553 2019-04-17 20:56:43.961670 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Checking status now 2019-04-17 20:56:43.961715 :CLSDYNAM:3276416768: [dbfs_mount]{2:37893:29307} [check] Check -- ONLINE 2019-04-17 20:56:43.962013 : AGFW:3274315520: {2:37893:29307} dbfs_mount host02 1 state changed from: STOPPING to: ONLINE 2019-04-17 20:56:43.962064 : AGFW:3274315520: {2:37893:29307} Agent sending last reply for: RESOURCE_STOP[dbfs_mount host02 1] ID 4099:16017553 2019-04-17 20:56:49.133409 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Executing action script: /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh[check] 2019-04-17 20:56:49.185415 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Checking status now 2019-04-17 20:56:49.185465 :CLSDYNAM:3282720512: [dbfs_mount]{1:53477:37077} [check] Check -- ONLINE [/code] While the actual process that was locking /ggdata was not determined at the time, fuser can be used to determine the process. [code] # fuser -mv /ggdata/ USER PID ACCESS COMMAND /ggdata: root kernel mount /ggdata ggsuser 64776 F.... extract oracle 65049 f.... oracle_65049_ih oracle 65084 F.... oracle_65084_ih oracle 65094 F.... oracle_65094_ih oracle 65139 F.... oracle_65139_ih oracle 65740 F.... oracle_65740_ih ggsuser 84987 F.... extract ggsuser 84990 F.... extract ggsuser 84994 F.... extract ggsuser 84997 F.... extract ggsuser 85000 F.... extract ggsuser 85003 F.... extract [/code] In conclusion, the following files are useful for investigation: crsd_scriptagent_oracle.trc /var/log/messages. Since the errors were rather vague, fuser will identify the process causing the failure to unmount. Stay tuned as I will share other options to resolve 'failed to unmount" in a future post.
