First of all, if you are using NFS to store Oracle database data files, I strongly advise you to enable Oracle Direct NFS (DNFS) to access these files. I will leave the long explanation on why you should do it out of this blog post. However, I’ll mention that the main reason is performance. (Google it or wait for my following blog posts for more information.) DNFS removes a serialization point over the traditional “kernelized” NFS. However, if you are not using NFS, DNFS is probably not of interest to you, and you should stop reading here. But, if you want to set it up, let’s go! :)
The only legitimate reason I can see for not using DNFS is a bug that doesn’t allow you to use it. If this is the case, I would advise you to resolve the problem (probably by applying a patch from Oracle Support) and enable DNFS.
How to set up
Reference: Step by Step – Configure Direct NFS Client (DNFS) on Linux (11g) [ID 762374.1]
One of DNFS’s greatest features is an easy way to enable/disable it and a relatively easy way to check if it’s enabled or not. It is worth mentioning that DNFS is available from Oracle 11G only. If your system is configured using NFS (see Oracle Support notes on how to configure it), use the following two commands to enable/disable DNFS.
- Enable
ln -s $ORACLE_HOME/lib/libnfsodm11.so $ORACLE_HOME/lib/libodm11.so
- Disable
ln -s $ORACLE_HOME/lib/libodmd11.so $ORACLE_HOME/lib/libodm11.so
Of course, you should stop all DBs that use the ORACLE_HOME, make a backup of the initial symbol link, etc. I’ll leave the precaution measures to you. The simple fact is that the two commands above are the only commands you should know to enable/disable Oracle Direct NFS.
You would ask: How does Oracle know what NFS server to connect to, what IP to be used, etc? In the simplest configuration, Oracle retrieves all the information it needs from the /etc/mtab Linux configuration file. This file is populated by Linux each time you mount an NFS volume. The static volumes’ mount configuration is stored in an /etc/fstab file.
How to check
This is where the story gets a bit tricky. The Oracle Support note mentioned above [ID 762374.1] states the following:
alert log shows Direct NFS client is enabled by checking the below message Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0
However, I came across several cases where the message was in an alert.log file, but DNFS still wasn’t enabled.
I found that the easiest and most reliable way to verify if DNFS is enabled and working as expected is to check Oracle Instances process’ opened files list. Find a process that has opened at least one data file and run lsof Linux command against the process. I usually check the DBWR foreground process as it has some data files opened right from the instance start up time.
The following is an example of how an output should look like if the DNFS is in use:
-bash-3.2$ ps -ef | grep dbw oracle 23126 1 0 06:53 ? 00:00:00 ora_dbw0_MYINST1 oracle 23128 1 0 06:53 ? 00:00:00 ora_dbw1_MYINST1 oracle 23246 22377 0 07:04 pts/0 00:00:00 grep dbw -bash-3.2$ /usr/sbin/lsof -p 23126 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME oracle 23126 oracle cwd DIR 104,3 4096 14811513 /u02/app/oracle/product/11.2.0.1/dbs oracle 23126 oracle rtd DIR 104,3 4096 2 / oracle 23126 oracle txt REG 104,3 210824714 15892486 /u02/app/oracle/product/11.2.0.1/bin/oracle ... oracle 23126 oracle 32u IPv4 322388 TCP mydbhost:64553->mynfshost:nfs (ESTABLISHED) oracle 23126 oracle 33u IPv4 322389 TCP mydbhost:21549->mynfshost:nfs (ESTABLISHED) ...
Pay attention to the last two rows I highlighted. You should see just TCP/IP sockets opened by the process towards NFS server. There should not be any data files listed. If you see any of the data files (located on NFS) in the output, then the instance doesn’t use DNFS. BTW: I would appreciate if you provided an output with data files listed in the comments section below this blog post as I would like readers to see an example. Unfortunately, I don’t have a good (bad) example handy as all instances I have access to use DNFS today :).
At this point, I would like to end my very first DNFS-related blog post. I have a lot more to say about DNFS, but I’ll leave it for future blog posts. The following is just comments and questions I have about DNFS:
- NFS server must be configured to accept NFS clients’ requests coming from a non-standard port range.
- How to load balance DNFS traffic if you have several NICs (oranfstab file)?
- Use init.ora parameters (events 19392, 19394, 19396) to troubleshoot DNFS configuration.
- Don’t use the same NFS volumes names on different NFS servers (DNFS can’t work it out).
- Performance difference between kNFS & DNFS (testing results). Is there any real benefits using DNFS over kNFS?
- Good references list for DNFS information (MOS notes, Blog posts, etc.)
- Some other details you want here :) …
Let’s hope that, with all your support and a bit of free time, I will blog about the topics mentioned :)
Stay tuned folks,
Yury
17 Comments. Leave new
Finding this post svoles a problem for me. Thanks!
From reading this post Direct NFS seems to be an alternative to kernel based NFS. What I wonder:
Direct NFS reads NFS shares to mount from /etc/mtab. That means that I still need to mount these shares using kernel based NFS, else the shares wouldn’t appear there, right? So I need to setup kernel based shares anyway.
You have right understanding. In order to use DNFS on Linux/Unix platform kNFS should be configured.
Since I wrote this blog post I came with a DNFS presentation. You may be interested to have a look on it from the following link.
https://www.slideshare.net/yvelikanov/sharing-experience-implementing-direct-nfs
I am switching over to Oracle dNFS for my Linux RHEL6 11.2.0.3 using NetApp Storage. So far so good but I have run into a problem where the orapw file is not read/processed when the Oracle Binaries are also on a remote source. Oracle nfs ODM seems to not read the orapw file in the dbs directory.
The datafiles and the binaries are on separate mounts and from separate servers.
I’ve got 4 NIC’s for the storage connections confirured and they are all working appropriately.
If I relocate the pwfile into a put a symbolic link from teh dbs firectory to it.. the pwfile is recognized. And it worked fine without the Oracle dNFS enabled.
The alert log shows the following:
Oracle image stored on remote server servernas1
…..
Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 3.0
….
Completed: ALTER DATABASE open
Errors in file /oracle/BDM/saptrace/diag/rdbms/bdm/BDM/trace/BDM_ora_10246.trc:
ORA-17503: ksfdopn:3 Failed to open file /oracle/v11203a/dbs/orapwBDM
ORA-17500: ODM err:No such file or directory
This error is raised when performing a simple ‘select * from v$pwfile_users;’ or a grand sysdba to …
Any thoughts as to what may be causing that? Is there any option to restrict Oracle dNFS from processing the binaries and the dbs directory ?
Thanks
Rich
Hey Rich,
Just a quick educated guess from my side quickly scanning morning emails.
Can you check if NFS volumes from different Fillers have the same names: e.g. nfsserver01:/oravol1 and nfsserver02:/oravol1
If this is the case then there is a known DNFS issue. I have faced it myself. Just rename volume to be unique across all NFS mounts on the DB server.
Yury
oracle 25696 oracle 16u REG 0,20 1544 51957 /u01/app/oracle/product/11.2.0/dbhome_2/dbs/hc_OEMREP.dat (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 17uR REG 0,20 24 51955 /u01/app/oracle/product/11.2.0/dbhome_2/dbs/lkOEMREP (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 18r REG 0,20 1101312 59735 /u01/app/oracle/product/11.2.0/dbhome_2/rdbms/mesg/oraus.msb (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 256u REG 0,20 10928128 781 /u01/oradata/OEMREP/control01.ctl (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 257u REG 0,20 10928128 782 /u01/FRA/OEMREP/control02.ctl (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 258uW REG 0,20 734011392 789 /u01/oradata/OEMREP/system01.dbf (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 259uW REG 0,20 629153792 790 /u01/oradata/OEMREP/sysaux01.dbf (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 260uW REG 0,20 613425152 791 /u01/oradata/OEMREP/undotbs01.dbf (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 261uW REG 0,20 5251072 793 /u01/oradata/OEMREP/users01.dbf (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 262uW REG 0,20 104865792 143808 /u01/oradata/OEMREP/mgmt_depot.dbf (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 263uW REG 0,20 2621448192 143809 /u01/oradata/OEMREP/mgmt.dbf (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 264uW REG 0,20 209723392 143810 /u01/oradata/OEMREP/mgmt_ad4j.dbf (ca-filer2:/vol/oraemsvrmnt/)
oracle 25696 oracle 265uW REG 0,20 46145536 792 /u01/oradata/OEMREP/temp01.dbf (ca-filer2:/vol/oraemsvrmnt/)
Thanks for the output. For others. This is how the list of opened files looks like if DNFS ISN’T enabled.
how should the mounts be defined —
I have several linux oracle servers. We are moving from Netapp to EMC. Only one of linux oracle servers is going to mount to our new EMC vnx5300.
I am getting the error
@rebuild_rlispub_dec13.sql
ORACLE instance started.
Total System Global Area 2488709120 bytes
Fixed Size 2162200 bytes
Variable Size 469762536 bytes
Database Buffers 2013265920 bytes
Redo Buffers 3518464 bytes
CREATE CONTROLFILE SET DATABASE “RLISPUB” RESETLOGS NOARCHIVELOG
*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-00200: control file could not be created
ORA-00202: control file: ‘/oracle11/oradata/rlispub/control01.ctl’
ORA-17502: ksfdcre:/oracle11/oradata/rlispub/control01.ctl Failed to create file
An easier way to verify and view the DNFS stats would be to query the dnfs v$ views:
SQL> select * from v$dnfs_servers;
SQL> select * from v$dnfs_files;
SQL> select * from v$dnfs_files;
SQL> select * from v$dnfs_channels;
My point and warning to outside world V$ views sometimes aren’t reliable source of information. I was there myself.
On the side note for some geeky folks “lsof” option may be much easier than SQL :)
PS Thanks for the comment Jesse
???????, ???! ???? ?????? ?????? ? 1-? ??????? ?? ???? DirectNFS ? Google, ????? ??? ??? ????????????. ???? ????????? ????? ?? ?????? ?????? ?????? ? MOS.
??? ????? ?????? ? ???, ??? ??? ?????????? (??? ????? ??? ?, ??? ????, ?? ??????? ????? ???????????) ??? ??? ??????? ??????.
? MOS ?????? ?? ??????? ????? ????? ?? ???????????, ??? ??????? ?????????, ? ????? ????? ????? ? 11.2.0.2 CloneDB. ?????? ?????? ? ??????????? – ?????? ? ??? ???? – ??? ????? ????????? ??? ????????? NetApp.
ops. :(
1) Pythian web site does not support Unicode despite sourse shows: meta charset=”UTF-8″ Yura, please help your web-admins to setup NLS settings.
2) Build using Ajax approach
about Direct NFS:
1. Saved me some minuted looking for right MOS article when ai needed it. Thank You! It is in top 10 in Google search for the subject.
2. Another benefit – article reminds (to some like me, that was learning DNFS feature, but had forgotten because not using)
3. MOS article have links to others, one of them have presentation that explain internals as well as new DNFS based feature CloneDB.
If will investigate the DNFS again – take a look. This may be very attractive for big DB like in banks and for NetApp lovers.
MOS notes recommended:
* Step by Step – Configure Direct NFS Client (DNFS) on Linux (11g) [ID 762374.1] (yours)
* DNFS: Example About How To Setup DNFS (Direct NFS) On Oracle Release 11.2 [ID 1452614.1]
* Recommended Patches for Direct NFS Client [ID 1495104.1]
2. Another benefit – article reminds that it is really easy to setup (to some like me, that was learning DNFS feature, but had forgotten because not using)
Hi Yury,
we get the following trying to start the db after
Direct NFS: please check that oradism is setuid
details:
make -f ins_rdbms.mk dnfs_on
>startup
hangs – alert log error:
ALTER DATABASE MOUNT
Direct NFS: please check that oradism is setuid
[email protected] ~]# cd /u01/app/oracle/OURS/11.2.0/bin
[[email protected] bin]# ls oradism
oradism
[[email protected] bin]# ls -l oradism
-rwxr-x—. 1 oraours dba 71758 Sep 17 2011 oradism
[[email protected] bin]# chown root:root oradism
[[email protected] bin]# chmod 4755 oradism
[[email protected] bin]# ls -l oradism
-rwxr-xr-x. 1 root root 71758 Sep 17 2011 oradism
[[email protected] lib]$ make -f ins_rdbms.mk dnfs_on
[[email protected] lib]$ ls -l $ORACLE_HOME/lib/lib*od*11.so
-rw-r–r–. 1 oraours dba 60431 Sep 17 2011 /u01/app/oracle/OURS/11.2.0/lib/libnfsodm11.so
-rw-r–r–. 1 oraours oraours 60431 Jun 2 2015 /u01/app/oracle/OURS/11.2.0/lib/libodm11.so
-rw-r–r–. 1 oraours dba 12259 Sep 17 2011 /u01/app/oracle/OURS/11.2.0/lib/libodmd11.so
[[email protected] lib]$ make -f ins_rdbms.mk dnfs_off
rm -f /u01/app/oracle/OURS/11.2.0/lib/libodm11.so; cp /u01/app/oracle/OURS/11.2.0/rdbms/lib/libodm11.so.dummy /u01/app/oracle/OURS/11.2.0/lib/libodm11.so
[[email protected] lib]$ ls -l $ORACLE_HOME/lib/lib*od*11.so
-rw-r–r–. 1 oraours dba 60431 Sep 17 2011 /u01/app/oracle/OURS/11.2.0/lib/libnfsodm11.so
-rw-r–r–. 1 oraours oraours 12259 Jun 2 2015 /u01/app/oracle/OURS/11.2.0/lib/libodm11.so
-rw-r–r–. 1 oraours dba 12259 Sep 17 2011 /u01/app/oracle/OURS/11.2.0/lib/libodmd11.so
Sys admin got that problem solved. the oradism was not truly changing properties properly for the setuid.
Thanks for your article!
Hi, Yuri. Thanks for the posts, it helped a lot.
One small question – is there any logic behind the number of established NFS connections once dNFS is enabled and is there any way to control/limit this number?
Thanks a lot
Hi Yuri,
I am using zero padded files for nfs as mentioned in Oracle Doc https://docs.oracle.com/en/database/oracle/oracle-database/21/cwlin/creating-files-on-a-nas-device-for-use-with-oracle-automatic-storage-management.html#GUID-D1F279EF-2D7D-4F70-9AC3-3078FF6E5649.
On running lsof command I get this
ora_dbw0_ 42330 oracle 256u BLK 7,1 0t0 18446621119394645416 /dev/oracleasm/disks/DATA1
ora_dbw0_ 42330 oracle 257u BLK 7,3 0t0 18446621119394649640 /dev/oracleasm/disks/DATA3
ora_dbw0_ 42330 oracle 258u BLK 7,2 0t0 18446621119394672360 /dev/oracleasm/disks/DATA2