The short question
While preparing a presentation I am giving this year about HugePages, a colleague of mine at Pythian posed an interesting question: Does Oracle use HugePages for PGA?
The short answer
No, it doesn’t.
The long question
We DBAs, not only Oracle but any DBA managing a database that may use large amounts of RAM, should be familiar with the HugePages feature. If you are not, don’t miss my presentation at Technical SPOUG day 2018.
The basic idea is that memory is allocated in 4KB chunks by default and these chunks are accessed by means of a memory structure, the PageTable, that keeps track of them. For large amounts of RAM, this memory structure can grow very quickly and become slow and cumbersome. If the memory chunks are set to a larger size, say 2MB, the PageTable size is proportionally reduced and the performance is good again.
When we think of using this highly recommended feature for Oracle databases, we consider only the SGA to fit into HugePages but not the PGA. Why is that? Because the documentation, MOS notes and blog posts one finds everywhere mention only the SGA. It is hard to find a mention of the PGA, except for a reminder here and there that the total amount RAM allocated for HugePages should leave enough free memory on the system for the OS and the PGA.
This alone may serve as a weak argument to say that Oracle does not allocate PGA in HugePages, but the question still stands: Is this true?
It occurs to me, and probably other DBAs too, that when I think about PGA I think of it as a single entity, similar to the SGA. While this is helpful as a simplification, the truth is in the definition: The PGA is “the single process private memory structure that contains the data and control information of a server process“. So, when I (we) think about PGA, I am actually thinking about the PGA_AGGREGATE_TARGET parameter of the database. While this simplification makes it easier to work with initial memory allocation, it also introduces a bias in reasoning when it comes to PGA and, related to the subject of this blog post, it triggers the question: Does Oracle allocate PGA in HugePages?
The long answer (including a small question)
How could the PGA get into HugePages?
As explained in this dated but accurate blog post, the HugePages implementation includes interfaces that allow private memory segments to be allocated to HugePages so, theoretically, PGA memory may be allocated here. Then it depends entirely on how Oracle has implemented the memory allocation for PGA.
In MOS note “Bigpages vs. Hugetlb on RedHat Linux (Doc ID 261889.1),” the following is written:
A program that wants to allocate shared memory has to add a flag, SHM_HUGETLB, to the shmget() flags. This approach ensures that the Oracle shared memory segments will be allocated out of this pool.
Oracle Database 10g will do this by default;
While this mentions quite old Oracle versions, I daresay that this still holds true for the newest ones, meaning that the interface used to allocate shared memory (SGA), and hence HugePages, is the shmget() call.
Now, according to Fritz Hoogland’s findings in his post “Oracle database operating system memory allocation management for PGA – part 4: Oracle 22.214.171.124 and AMM,” Oracle allocates private memory using mmap() calls as follows:
mmap(0x7f0194f7a000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x7f0194f7a000
The above call does not include the MAP_HUGETLB flag, which is required for the mmap() call to request memory in the HugePages area, so the PGA is not allocating any HugePages and this is the Oracle developer’s chosen way.
Even though I’ve been unable to find hard evidence in the Oracle documentation or any MOS note, we can say that HugePages are not used for PGA.
I’ve shown evidence that this could be possible, the instrumentation is there, but Oracle does not attempt to allocate PGA memory in the HugePages area according to the mmap() calls they use.
This makes sense because most of the existing server processes will have relatively small memory requirements and allocating private memory in 2MB chunks will lead to memory exhaustion pretty quickly. As a basic example, consider an OLTP system with 2000 idle sessions. Using HugePages, the total allocated private memory would be close to 4GBs (2K*2MB), while with 4KB chunks we are talking roughly 8MBs (2K*4KB).
On the other side, given that the PGA of a process may grow quite big depending on the amount of data, think memory sorts in a DWH or DSS, while it is clear at this point that Oracle is not allocating PGA memory in HugePages, this may change in the future.
One thing I would keep in mind discussing about memory allocation for PGA. Oracle server process allocates more than 4k even if it is in idle state. I would say that it allocated from 800k to 3Mb in idle state. So, for 2k session it will be way more than 8M.
Thank you Gleb.
You are right, the use of 4K allocation is not accurate compared to reality. Even though, the idea of memory fragmentation behind that paragraph still stands true although not very well transmitted.
For small page allocation a copy of the page table is attached to each process. PGA allocations need this as each memory process allocation is independent. Each process has its private chunk of the PGA pool (not exactly a pool like SGA). So this system page table copy is necessary. It’s a overhead, but necessary.
Unlike this approach, for big page allocation there is no system page table for each process. There is only one copy, the original one. This is a not problem to access memory from one memory global pool, which is exactly the description of SGA. But for PGA this is impossible.
For these reason PGA memory is always a small page type allocation.
So, when designing your system memory parameters on Linux, keep an amount of memory for your PGAs (typically a little bit more than the sum of your instances PGAs) is the small table Linux pool.
As an update: this blog is no longer fully accurate. From 18c Oracle accounts MGA – Managed Global Area – as PGA, and can allocate it in the HugePages. More: MGA (Managed Global Area) Reference Note (Doc ID 2638904.1)