Recently, working with some performance problems on AWS RDS, I noticed that the customers had Oracle RDS instances that didn’t use the “Hugepages” memory. They had several Oracle RDS instances of different types and all of them had more than 30GiB of memory. Even it was not the primary performance problem, it definitely introduced some overhead. All of those machines were using default Automatic Memory Management (AMM) and 4K memory pages for SGA. I am not planning to discuss in detail the benefits of using large pages memory in this post, as it has already been discussed in a number of blogs, white papers, Oracle documentation and support articles. For example, you can have a look it in the Oracle documentation or in the Oracle white paper. Here I would like to discuss how AWS RDS for Oracle databases supports large pages and how they are enabled.
When you plan a database landscape on AWS RDS, you need to pay attention to several factors such as memory and storage. As you know, instance type and size can directly affect your database performance. But when you make your final selection in choosing between one or another instance type, I recommend double-checking whether it is eligible to use Hugepages. AWS officially supports the large pages in memory, at least from July last year, and provides documentation and a list of instances which are supposed to use Hugepages here.
Not too long time ago, I was able to enable the large pages for some types of instances if the memory was greater than 7GiB. But, as you can see now, the threshold had been raised and requires at least 14GiB memory. I am going to test it using the t2 type instance for AWS RDS. For the start, I am creating a db.t2.medium instance with just 4GiB memory. Let’s have a look at the memory parameters and how instance is using it:
Here is an excerpt from the parameter group:
memory_max_target = IF({DBInstanceClassHugePagesDefault}, 0, {DBInstanceClassMemory*3/4})
memory_target = IF({DBInstanceClassHugePagesDefault}, 0, {DBInstanceClassMemory*3/4})
use_large_pages = {DBInstanceClassHugePagesDefault}
And when we start the database, we see that in the alert log:
…
Mon Jun 25 20:15:37 2018
Reason for not supporting certain system pagesizes:
Mon Jun 25 20:15:37 2018
2048K - Dynamic allocate and free memory regions
Mon Jun 25 20:15:37 2018
…
use_large_pages = "FALSE"
spfile = "/rdsdbbin/oracle/dbs/spfileORCL.ora"
filesystemio_options = "setall"
sga_target = 0
memory_target = 2816M
memory_max_target = 2816M
It is clear the instance is using the Automatic Memory Management (AMM), the parameter use_large_pages switched to “FALSE” and the reason for not using the large pages is “Dynamic allocate and free memory regions”.
What if we try to set up the memory and large pages parameters explicitly in the custom parameter group for RDS and apply the changes?
Here are some settings from my custom parameter group:
use_large_pages = ONLY
memory_max_target = 0
memory_target = 0
We can see that changes were applied to the parameters and the instance was rebooted.
…
Mon Jun 25 20:22:50 2018
ALTER SYSTEM SET memory_target=0 SCOPE=SPFILE;
Mon Jun 25 20:22:50 2018
ALTER SYSTEM SET pga_aggregate_target=2147483648 SCOPE=SPFILE;
Mon Jun 25 20:22:51 2018
ALTER SYSTEM RESET DBFIPS_140 SCOPE=SPFILE SID='*';
Mon Jun 25 20:22:51 2018
ALTER SYSTEM RESET memory_max_target SCOPE=SPFILE SID='*';
…
But after the reboot I see that message in the “Recent events” for the RDS instance:
“HugePages is not supported for this instance. You cannot enable HugePages for paravirtual DB instances, or DB instances with less than 14 GiB of memory. The use_large_pages parameter will be overwritten to FALSE. Consult the documentation for instructions on how to enable HugePages.”
So, it is not allowing me to enable the Hugepages manually for the instance with less than 14GiB memory. And if we connect to the database and check the parameters, we can see that even if the AMM was disabled, the “Hugepages” were not used because the “use_large_pages” parameter was explicitly set to “FALSE”.
orcl> show parameter large_pages
NAME TYPE VALUE
--------------- ------ -----
use_large_pages string FALSE
orcl> show parameter memory_target
NAME TYPE VALUE
------------- ----------- -----
memory_target big integer 0
orcl>
One more thing is worth mentioning here. When we look at the alert log for the starting instance, we see the message “Per process system memlock (soft) limit = 64K” which tells us that user limits for the Oracle software owner were not adjusted to use the large pages for SGA.
I tried to increase the instance type to db.t2.large with 8GiB and repeated the approach for the m3 type instance with the same results. Only when I used db2.xlarge with 16GiB memory onboard could I clearly see that even with the default parameter group, the system started to use large pages. We could see it in the alert log for the database. We have “memlock” limit setup to 12 G and 2M pages allocated for SGA.
Mon Jun 25 21:14:12 2018
Expected per process system memlock (soft) limit to lock
SHARED GLOBAL AREA (SGA) into memory: 12G
Mon Jun 25 21:14:12 2018
Available system pagesizes:
4K, 2048K
Mon Jun 25 21:14:12 2018
Supported system pagesize(s):
Mon Jun 25 21:14:12 2018
PAGESIZE AVAILABLE_PAGES EXPECTED_PAGES ALLOCATED_PAGES ERROR(s)
Mon Jun 25 21:14:12 2018
2048K 5954 5954 5954 NONE
Mon Jun 25 21:14:12 2018
Reason for not supporting certain system pagesizes:
Mon Jun 25 21:14:12 2018
4K - Large pagesizes only
Mon Jun 25 21:14:12 2018
And checking using sqlplus shows that the large pages are in use. The system is switched from AMM to Automatic Shared Memory Management (ASMM) and the parameter “use_large_pages” has been switched to “ONLY”
orcl> show parameter large
NAME TYPE VALUE
--------------- ----------- -----
large_pool_size big integer 0
use_large_pages string ONLY
orcl> show parameter memory_max
NAME TYPE VALUE
----------------------------- ----------- -----
inmemory_max_populate_servers integer 0
memory_max_target big integer 0
orcl> show parameter sga
NAME TYPE VALUE
---------------------------- ----------- -------
sga_max_size big integer 11904M
sga_target big integer 11904M
So, in summary, I can say that if you create or migrate to Oracle RDS on AWS you should look at the instance type and make sure it supports Hugepages. You will also need to keep in mind that the large pages will be available only for instances with bigger than 14GiB memory. For any instances with less memory, the database parameters will be ignored, user limits will not be set and default small pages will be used. Also, the AWS documentation says that for some instances which are not on the list, the Hugepages will be automatically enabled only after 100GiB of memory.
No comments