Enabling large pages on Oracle Database 11g running on IBM AIX

Posted in: Technical Track

For implementing Large Pages on AIX first you will need to choose large page size at OS level.
On AIX you can have multiple large page sizes of 4KB, 64KB, 16MB, and 16GB.

In this example we will be using a large page size of 16MB.

Steps for implemenatation:

1- Based on MOS Doc ID 372157.1 first you need to enable Large Pages at OS level

# vmo -r  -o lgpg_size=16777216 -o lgpg_regions=<Total number of pages>
# vmo -o lru_file_repage=0
# vmo -p -o v_pinshm=1
# lsuser -a capabilities oracle
# chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle
# bosboot -a

This needs a server reboot

For complete instruction please review note ID 372157.1

2- Setting parameters at instance level

On AIX databases you only need to set LOCK_SGA to TRUE:

alter system set lock_sga=TRUE scope=spfile;

Note: On AIX databases, USE_LARGE_PAGES parameter has NO impact.
These parameters are only valid for databases running on Linux, the value of this parameter even if set to FALSE will be ignored on AIX.

By default when Large Pages is available on AIX it will be used by database instances regardless of USE_LARGE_PAGES parameter value. You only need to set LOCK_SGA.

3- Restart the instance and confirm Large Pages is in use:

After setting lock_sga instance must be restarted.
As I explained above, when Large Pages is available at OS level it will be used by instance, but the key point in here is how to confirm whether Large Pages is in use or not.

How to check if Huge Pages is used by Oracle instance.

For Oracle 11g running on AIX, no informational message is written to the alert log as what we see in the alert log of databases running on Linux.

So for your database instance running on AIX do NOT expect following lines in the alert log:

****************** Large Pages Information *****************
Total Shared Global Region in Large Pages = xx MB (100%)
Large Pages used by this instance: xxx (xxx MB)
Large Pages unused system wide = x (xxx MB) (alloc incr 4096 KB)
Large Pages configured system wide = xxx (xxx MB)
Large Page size = 16 MB
***********************************************************

The only way you can make sure large pages is being used by instance is checking memory usage at OS level:

Consider SGA_TARGET in your instance is 8G
Total number of Large Pages (with size of 16M) will be 8G/16M + 1 which is : 8589934592 / 16777216 + 1 = 513

Check the number of large 16M pages in use at OS level before starting your instance:

$ vmstat -P all
System configuration: mem=98304MB
pgsz            memory                           page
----- -------------------------- ------------------------------------
           siz      avm      fre    re    pi    po    fr     sr    cy
   4K  4420992  2926616   487687     0     0     5  1280   2756     0
  64K   582056   581916      754     0     0     0     0      0     0
  16M     2791       87     2704     0     0     0     0      0     0

In this example number of 16M pages in use before starting instance is 87 pages from total available of 2791 pages.

We start the instance with SGA size of 8G:

SQL> startup
ORACLE instance started.
Total System Global Area 8551575552 bytes
Fixed Size                  2238616 bytes
Variable Size            2348812136 bytes
Database Buffers         6190792704 bytes
Redo Buffers                9732096 bytes
Database mounted.
Database opened.
SQL> show parameter sga_target
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
sga_target                           big integer 8G
SQL> show parameter lock_sga
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
lock_sga                             boolean     TRUE

Then we check Large pages in use again :

$ vmstat -P all
System configuration: mem=98304MB
pgsz            memory                           page
----- -------------------------- ------------------------------------
           siz      avm      fre    re    pi    po    fr     sr    cy
   4K  4428160  2877041   420004     0     0     5  1279   2754     0
  64K   581608   530522    51695     0     0     0     0      0     0
  16M     2791      600     2191     0     0     0     0      0     0

As you can see the total number of 16M pages in use is now 600 pages, which is exactly 513 pages more than what it was before instance startup.
This proves that 16M pages have been used by our instance.

You can also check memory usage of your instance by checking one of the instance processes like pmon:

$ ps -ef|grep pmon
  oracle 14024886 31392176   0 14:05:34  pts/0  0:00 grep pmon
  oracle 41681022        1   0   Mar 11      -  3:12 ora_pmon_KBS

Then check memory used by this process is from 16M Pages:

$ svmon -P 41681022
-------------------------------------------------------------------------------
     Pid Command          Inuse      Pin     Pgsp  Virtual 64-bit Mthrd  16MB
41681022 oracle         2180412  2109504     1778  2158599      Y     N     Y
     PageSize                Inuse        Pin       Pgsp    Virtual
     s    4 KB               31820          0       1650       9975
     m   64 KB                2959        516          8       2961
     L   16 MB                 513        513          0        513

I hope this will be useful for you, and good luck.

email

Interested in working with Kamran? Schedule a tech call.

9 Comments. Leave new

I find
vmstat -l
a simpler way of showing if largepages are used before/after db starts:

server:posdb$vmstat -l

System configuration: lcpu=8 mem=16384MB ent=1.00

kthr memory page faults cpu large-page
—– ———– ———————— ———— ———————– ———–
r b avm fre re pi po fr sr cy in sy cs us sy id wa pc ec alp flp
2 1 2676239 1223215 0 0 0 0 2 0 724 5770 1779 26 2 71 1 0.44 44.1 321 9

In this case, 321 in use and 9 left over.

Handy if only one size of largepages are in use at any given time.
I tend to stick to 16M although I’ve read a few docs that seem to imply 64K largepages are somehow better.
Never confirmed that.

Reply
Kamran Bakhshandeh
April 5, 2016 6:20 pm

That’s correct, this is another way showing large pages is used, vmstat is also an OS command.

The purpose of this article to show that there on IBM AIX there won’t be any alert log messages indicating that Large Pages is used and we have to stick to checking memory usage at OS level.

Thanks

Kamran

Reply

Hi,

What’s your experience with large pages on AIX? What performance gain did you archive?

Cheers, Markus

Reply
Kamran Bakhshandeh
April 15, 2016 1:35 am

Large Pages is useful when memory on server is large and your SGA is going to be large too.
For example when you have 100G of memory on your AIX server and you are planning SGA with size of 64G, if you go with default page size which is 4K on AIX, then you will have more than 16 millions of pages for your SGA at OS level, but if you go with 16M page size. total pages at OS level will be 4096 pages. For OS finding and accessing a page from thousands of pages will be much faster than finding a page from millions of them.

Reply

Thanks. But what’s your experience? Is it 0, 5, 10, 15% faster? Does it cost 0, 5, 10, 20% less CPU?

Reply
Kamran Bakhshandeh
April 17, 2016 11:47 pm

Actually following best practices, for large memory sizes we never implemented regular page and then apply large pages, we used large pages for large memory sizes from beginning , so I don’t have any practical benchmark to compare improvements before and after implementing large pages.

Regards

Kamran

Reply

I saw a marked decrease in CPU time used by “kernel”.
This is consistent with paging mechanism spending less time finding one page out of millions.
That is accounted for in topas for example as “kernel” CPU use.

Reply

Besides small page swapping, what reasons would implementing large pages slow things down? I have a client on AIX 7.1/Power8 and Oracle 11g that implemented large pages as you described above and performance got worse. I haven’t seen any swapping going on either. What else could be happening?

At another client, IBM recommended setting AIX to use 64k pages and not pin SGA. They insisted that had just as good of performance. I am going to try that with my current client and see what happens.

Reply

Hi,

Do you think the AIX ASO/DSO which configure large pages automatically, have the effect on Oracle?

Regards

Mehdi

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *