How to use mount-dbfs.sh to resolve "Device or resource busy"
Tags:
Oracle,
Technical Track
I was not satisfied with using fuser -mv to identify processes causing DBFS failure to unmount and killing the processes haphazardly. Upon further investigation, mount-dbfs.sh can be used to kill processes causing "Device or resource busy" and unmount DBFS. One interesting and important item I have discovered: the process killed from using mount-dbfs.sh is ***not**** the same as shown from fuser. Stopping DBFS failed as shown: [code] oracle@host $ /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh stop unmounting DBFS from /ggdata umounting the filesystem using '/bin/fusermount -u /ggdata' /bin/fusermount: failed to unmount /ggdata: Device or resource busy Stop - stopped, but still mounted, error oracle@host $ [/code] From another session, vi was used to open file from DBFS mount (ggdata). Notice the PID for vim is 38331. [code] root@host # fuser -mv /ggdata USER PID ACCESS COMMAND /ggdata: root kernel mount /ggdata ggsuser 2343 ..c.. bash ggsuser 38331 F.c.. vim root@host # exit [/code] Run mount-dbfs.sh clean in debug mode and notice: + /bin/kill -9 27532 [code] oracle@host $ /bin/bash -x /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh clean + CONFIG=/etc/oracle/mount-dbfs.conf + VERSION=20160215 + '[' -r /etc/oracle/mount-dbfs.conf ']' + . /etc/oracle/mount-dbfs.conf ++ DBNAME=DBFS ++ MOUNT_POINT=/ggdata ++ DBFS_USER=dbfs_user ++ ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_1 ++ GRID_HOME=/u01/app/12.1.0/grid ++ LOGGER_FACILITY=user ++ MOUNT_OPTIONS=allow_other,direct_io ++ PERL_ALARM_TIMEOUT=14 ++ DBFS_PASSWD=*** ++ DBFS_PWDFILE_BASE=/tmp/.dbfs-passwd.txt ++ WALLET=true ++ TNS_ADMIN=/u01/app/oracle/product/12.1.0/db_1/oracle/tnsadmin ++ DBFS_LOCAL_TNSALIAS=DBFS ++ IS_PDB=false ++ PDB=pdbXX ++ PDB_SERVICE=dbfspdb + CDB=DBFS ++ uname -s + UNAME_S=Linux + '[' Linux = Linux ']' + LINUX=1 + SOLARIS=0 + GREP=/bin/grep + AWK=/bin/awk + SED=/bin/sed + ECHO=/bin/echo + LOGGER='/bin/logger -t DBFS_/ggdata' + RMF='/bin/rm -f' + TOUCH=/bin/touch + CHMOD=/bin/chmod + PS=/bin/ps + SLEEP=/bin/sleep + KILL=/bin/kill + BASENAME=/bin/basename + STAT=/usr/bin/stat + ID=/usr/bin/id + WC=/usr/bin/wc + SRVCTL=/u01/app/oracle/product/12.1.0/db_1/bin/srvctl + DBFS_CLIENT=/u01/app/oracle/product/12.1.0/db_1/bin/dbfs_client + HN=/bin/hostname + PERL=/usr/bin/perl + MOUNT=/bin/mount + LANG=en_US.UTF-8 + NLS_LANG=American_America.AL32UTF8 + NUMACTL=/usr/bin/numactl + RPMCTL=/bin/rpm + '[' -z '' ']' + STATUS_TIMEOUT=0 + '[' 1 -eq 1 ']' + MOUNT=/bin/mount + XARGS='/usr/bin/xargs -r' + FUSERMOUNT=/bin/fusermount + LD_LIBRARY_PATH=/u01/app/oracle/product/12.1.0/db_1/lib:/lib64 + DBFS_PWDFILE=/tmp/.dbfs-passwd.txt.39786 + export ORACLE_HOME LD_LIBRARY_PATH TNS_ADMIN + export STAT MOUNT_POINT PERL_ALARM_TIMEOUT SOLARIS LINUX + export PATH=/u01/app/oracle/product/12.1.0/db_1/bin:/u01/app/oracle/product/12.1.0/db_1/bin:/usr/java/jdk1.8.0_65/bin:/usr/java/jdk1.8.0_65/jre/bin:/usr/java/jdk1.8.0_65/jre/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/oracle/.local/bin:/home/oracle/bin + PATH=/u01/app/oracle/product/12.1.0/db_1/bin:/u01/app/oracle/product/12.1.0/db_1/bin:/usr/java/jdk1.8.0_65/bin:/usr/java/jdk1.8.0_65/jre/bin:/usr/java/jdk1.8.0_65/jre/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/oracle/.local/bin:/home/oracle/bin + export STATUS_TIMEOUT + export LANG NLS_LANG ++ /usr/bin/id -u + '[' 1500 -eq 0 ']' + SCRIPTPATH=/u02/app/12.1.0/grid/crs/script/mount-dbfs.sh ++ /bin/basename /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh + SCRIPTNAME=mount-dbfs.sh + /bin/echo /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh + /bin/grep '^/' + '[' 0 -ne 0 ']' + cd /tmp + case "$1" in + logit info 'cleaning up DBFS nicely using (fusermount -u|umount)' + type=info + msg='cleaning up DBFS nicely using (fusermount -u|umount)' + '[' info = info ']' + /bin/echo cleaning up DBFS nicely using '(fusermount' '-u|umount)' cleaning up DBFS nicely using (fusermount -u|umount) + /bin/logger -t DBFS_/ggdata -p user.info 'cleaning up DBFS nicely using (fusermount -u|umount)' + '[' 1 -eq 1 ']' + /bin/fusermount -u /ggdata /bin/fusermount: failed to unmount /ggdata: Device or resource busy + /bin/sleep 1 + FORCE_CLEANUP=0 + '[' 0 -gt 1 ']' + /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh status + '[' 0 -eq 0 ']' + FORCE_CLEANUP=1 + '[' 1 -eq 1 ']' + logit error 'tried (fusermount -u|umount), still mounted, now cleaning with (fusermount -u -z|umount -f) and kill' + type=error + msg='tried (fusermount -u|umount), still mounted, now cleaning with (fusermount -u -z|umount -f) and kill' + '[' error = info ']' + '[' error = error ']' + /bin/echo tried '(fusermount' '-u|umount),' still mounted, now cleaning with '(fusermount' -u '-z|umount' '-f)' and kill tried (fusermount -u|umount), still mounted, now cleaning with (fusermount -u -z|umount -f) and kill + /bin/logger -t DBFS_/ggdata -p user.error 'tried (fusermount -u|umount), still mounted, now cleaning with (fusermount -u -z|umount -f) and kill' + '[' 1 -eq 1 ']' + /bin/fusermount -u -z /ggdata + '[' 1 -eq 1 ']' ++ /bin/ps -ef ++ /bin/grep -w /ggdata ++ /bin/grep dbfs_client ++ /bin/grep -v grep ++ /bin/awk '{print $2}' + PIDS=27532 + '[' -n 27532 ']' -------------------------------------------------------------------------------- + /bin/kill -9 27532 -------------------------------------------------------------------------------- ++ /bin/ps -ef ++ /bin/grep -w /ggdata ++ /bin/grep mount.dbfs ++ /bin/grep -v grep ++ /bin/awk '{print $2}' + PIDS= + '[' -n '' ']' + exit 1 [/code] Here are the parameters for mount-dbfs.sh: [code] oracle@host $ /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh -h Usage: mount-dbfs.sh { start | stop | check | status | restart | clean | abort | version } oracle@host $ [/code] In conclusion, it's much better to use the script for what it's designed for versus manually having to identify and kill processes.
