Oracle DBA: Huge Pages and Oracle Cluster 11GR2

In looking at this subject, most of you will say: Hey Yury! What’s new that you can tell us about? The following note describes it all:
11gR2 Grid Infrastructure Does not Use ULIMIT Setting Appropriately [ID 983715.1]

These were exactly my thoughts when I was called to troubleshoot a case where a DB instance, starting with the “srvctl start instance” command failed to use Huge Pages in Linux.

It appears that it isn’t that obvious what exactly you should configure to make sure that your Oracle Instances uses Huge Pages memory regions for shared memory. The tricky area is the /etc/security/limits.conf file.

Assumptions:
– grid – 11G Grid infrastructure Linux owner
– oracle – DB Instances owner
Huge pages are configured in Linux already as shown in the following example:

>grep -i huge /proc/meminfo
HugePages_Total: 28416
HugePages_Free:  19678
HugePages_Rsvd:    992
Hugepagesize:     2048 kB
>cat /proc/sys/vm/nr_hugepages
28416

Q:=A= What exactly do I need to configure to make sure that Oracle Cluster uses Huge Pages?
A: You must configure the following:
-1- pam_limits.so

grep pam_limits.so /etc/pam.d/login
session    required     pam_limits.so

-2- Adjust /etc/init.d/ohasd
REF: ID 983715.1

cat -n /etc/init.d/ohasd  | grep ulimit
    71    ulimit -n 65536
    72    ulimit -l 58395968

!!!IMPORTANT!!!
-3- /etc/security/limits.conf for grid user
NOTE: This is what has been missed in the case I was called to troubleshoot
Add the following lines to the /etc/security/limits.conf file

grid soft memlock 15728640
grid hard memlock 15728640
grid soft nproc 2047
grid hard nproc 16384

The important bit to mention is that you must configure limits for “grid” user rather than for “oracle” user as this seems to be logical from the first look.

After all 3 components are set I am sure you will find that Huge Pages are in use if you start your instances using “srvctl” utility.

Q:=B= What exactly do I need to configure to make sure that Huge Pages get used if I use sqlplus to start an Oracle Instance?
A: Add the following lines to the /etc/security/limits.conf file

oracle soft memlock 15728640
oracle hard memlock 15728640
oracle soft nproc 2047
oracle hard nproc 16384

In that case you must configure limits for the user you use to start an Instance.

Q:=C= What else I should be aware of in that context?

If you need to restart Oracle Cluster 11GR2 without restarting the whole server (on OS level) you must do one of the following:
-1- I would suggest to use an adjusted /etc/init.d/ohasd script to restart the Cluster (see question =A=).
or
-2- If you want to go the other way to restart Cluster processes (like “crsctl start has”) set the following limits before you run the command in your shell environment.

ulimit -n 65536
ulimit -l 50000000

PS I am sure that the bug 9084067 is going to be fixed in the future (if not fixed already).

All the best,
Yury