How to use mount-dbfs.sh to resolve “Device or resource busy”

Posted in: Technical Track

I was not satisfied with using fuser -mv to identify processes causing DBFS failure to unmount and killing the processes haphazardly. Upon further investigation, mount-dbfs.sh can be used to kill processes causing “Device or resource busy” and unmount DBFS.

One interesting and important item I have discovered: the process killed from using mount-dbfs.sh is ***not**** the same as shown from fuser.

Stopping DBFS failed as shown:

oracle@host $ /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh stop
unmounting DBFS from /ggdata
umounting the filesystem using '/bin/fusermount -u /ggdata'
/bin/fusermount: failed to unmount /ggdata: Device or resource busy
Stop - stopped, but still mounted, error
oracle@host $

From another session, vi was used to open file from DBFS mount (ggdata).
Notice the PID for vim is 38331.

root@host # fuser -mv /ggdata
                     USER        PID ACCESS COMMAND
/ggdata:             root     kernel mount /ggdata
                     ggsuser    2343 ..c.. bash
                     ggsuser   38331 F.c.. vim
root@host # exit

Run mount-dbfs.sh clean in debug mode and notice: + /bin/kill -9 27532

oracle@host $ /bin/bash -x /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh clean
+ CONFIG=/etc/oracle/mount-dbfs.conf
+ VERSION=20160215
+ '[' -r /etc/oracle/mount-dbfs.conf ']'
+ . /etc/oracle/mount-dbfs.conf
++ DBNAME=DBFS
++ MOUNT_POINT=/ggdata
++ DBFS_USER=dbfs_user
++ ORACLE_HOME=/u01/app/oracle/product/12.1.0/db_1
++ GRID_HOME=/u01/app/12.1.0/grid
++ LOGGER_FACILITY=user
++ MOUNT_OPTIONS=allow_other,direct_io
++ PERL_ALARM_TIMEOUT=14
++ DBFS_PASSWD=***
++ DBFS_PWDFILE_BASE=/tmp/.dbfs-passwd.txt
++ WALLET=true
++ TNS_ADMIN=/u01/app/oracle/product/12.1.0/db_1/oracle/tnsadmin
++ DBFS_LOCAL_TNSALIAS=DBFS
++ IS_PDB=false
++ PDB=pdbXX
++ PDB_SERVICE=dbfspdb
+ CDB=DBFS
++ uname -s
+ UNAME_S=Linux
+ '[' Linux = Linux ']'
+ LINUX=1
+ SOLARIS=0
+ GREP=/bin/grep
+ AWK=/bin/awk
+ SED=/bin/sed
+ ECHO=/bin/echo
+ LOGGER='/bin/logger -t DBFS_/ggdata'
+ RMF='/bin/rm -f'
+ TOUCH=/bin/touch
+ CHMOD=/bin/chmod
+ PS=/bin/ps
+ SLEEP=/bin/sleep
+ KILL=/bin/kill
+ BASENAME=/bin/basename
+ STAT=/usr/bin/stat
+ ID=/usr/bin/id
+ WC=/usr/bin/wc
+ SRVCTL=/u01/app/oracle/product/12.1.0/db_1/bin/srvctl
+ DBFS_CLIENT=/u01/app/oracle/product/12.1.0/db_1/bin/dbfs_client
+ HN=/bin/hostname
+ PERL=/usr/bin/perl
+ MOUNT=/bin/mount
+ LANG=en_US.UTF-8
+ NLS_LANG=American_America.AL32UTF8
+ NUMACTL=/usr/bin/numactl
+ RPMCTL=/bin/rpm
+ '[' -z '' ']'
+ STATUS_TIMEOUT=0
+ '[' 1 -eq 1 ']'
+ MOUNT=/bin/mount
+ XARGS='/usr/bin/xargs -r'
+ FUSERMOUNT=/bin/fusermount
+ LD_LIBRARY_PATH=/u01/app/oracle/product/12.1.0/db_1/lib:/lib64
+ DBFS_PWDFILE=/tmp/.dbfs-passwd.txt.39786
+ export ORACLE_HOME LD_LIBRARY_PATH TNS_ADMIN
+ export STAT MOUNT_POINT PERL_ALARM_TIMEOUT SOLARIS LINUX
+ export PATH=/u01/app/oracle/product/12.1.0/db_1/bin:/u01/app/oracle/product/12.1.0/db_1/bin:/usr/java/jdk1.8.0_65/bin:/usr/java/jdk1.8.0_65/jre/bin:/usr/java/jdk1.8.0_65/jre/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/oracle/.local/bin:/home/oracle/bin
+ PATH=/u01/app/oracle/product/12.1.0/db_1/bin:/u01/app/oracle/product/12.1.0/db_1/bin:/usr/java/jdk1.8.0_65/bin:/usr/java/jdk1.8.0_65/jre/bin:/usr/java/jdk1.8.0_65/jre/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/oracle/.local/bin:/home/oracle/bin
+ export STATUS_TIMEOUT
+ export LANG NLS_LANG
++ /usr/bin/id -u
+ '[' 1500 -eq 0 ']'
+ SCRIPTPATH=/u02/app/12.1.0/grid/crs/script/mount-dbfs.sh
++ /bin/basename /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh
+ SCRIPTNAME=mount-dbfs.sh
+ /bin/echo /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh
+ /bin/grep '^/'
+ '[' 0 -ne 0 ']'
+ cd /tmp
+ case "$1" in
+ logit info 'cleaning up DBFS nicely using (fusermount -u|umount)'
+ type=info
+ msg='cleaning up DBFS nicely using (fusermount -u|umount)'
+ '[' info = info ']'
+ /bin/echo cleaning up DBFS nicely using '(fusermount' '-u|umount)'
cleaning up DBFS nicely using (fusermount -u|umount)
+ /bin/logger -t DBFS_/ggdata -p user.info 'cleaning up DBFS nicely using (fusermount -u|umount)'
+ '[' 1 -eq 1 ']'
+ /bin/fusermount -u /ggdata
/bin/fusermount: failed to unmount /ggdata: Device or resource busy
+ /bin/sleep 1
+ FORCE_CLEANUP=0
+ '[' 0 -gt 1 ']'
+ /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh status
+ '[' 0 -eq 0 ']'
+ FORCE_CLEANUP=1
+ '[' 1 -eq 1 ']'
+ logit error 'tried (fusermount -u|umount), still mounted, now cleaning with (fusermount -u -z|umount -f) and kill'
+ type=error
+ msg='tried (fusermount -u|umount), still mounted, now cleaning with (fusermount -u -z|umount -f) and kill'
+ '[' error = info ']'
+ '[' error = error ']'
+ /bin/echo tried '(fusermount' '-u|umount),' still mounted, now cleaning with '(fusermount' -u '-z|umount' '-f)' and kill
tried (fusermount -u|umount), still mounted, now cleaning with (fusermount -u -z|umount -f) and kill
+ /bin/logger -t DBFS_/ggdata -p user.error 'tried (fusermount -u|umount), still mounted, now cleaning with (fusermount -u -z|umount -f) and kill'
+ '[' 1 -eq 1 ']'
+ /bin/fusermount -u -z /ggdata
+ '[' 1 -eq 1 ']'
++ /bin/ps -ef
++ /bin/grep -w /ggdata
++ /bin/grep dbfs_client
++ /bin/grep -v grep
++ /bin/awk '{print $2}'
+ PIDS=27532
+ '[' -n 27532 ']'

--------------------------------------------------------------------------------
+ /bin/kill -9 27532
--------------------------------------------------------------------------------

++ /bin/ps -ef
++ /bin/grep -w /ggdata
++ /bin/grep mount.dbfs
++ /bin/grep -v grep
++ /bin/awk '{print $2}'
+ PIDS=
+ '[' -n '' ']'
+ exit 1

Here are the parameters for mount-dbfs.sh:

oracle@host $ /u02/app/12.1.0/grid/crs/script/mount-dbfs.sh -h
Usage: mount-dbfs.sh { start | stop | check | status | restart | clean | abort | version }
oracle@host $

In conclusion, it’s much better to use the script for what it’s designed for versus manually having to identify and kill processes.

email

Interested in working with Michael? Schedule a tech call.

No comments

Leave a Reply

Your email address will not be published. Required fields are marked *