Does Oracle allocate PGA in HugePages?

The short question

While preparing a presentation I am giving this year about HugePages, a colleague of mine at Pythian posed an interesting question: Does Oracle use HugePages for PGA?

The short answer

No, it doesn’t.

The long question

We DBAs, not only Oracle but any DBA managing a database that may use large amounts of RAM, should be familiar with the HugePages feature. If you are not, don’t miss my presentation at Technical SPOUG day 2018.

The basic idea is that memory is allocated in 4KB chunks by default and these chunks are accessed by means of a memory structure, the PageTable, that keeps track of them. For large amounts of RAM, this memory structure can grow very quickly and become slow and cumbersome. If the memory chunks are set to a larger size, say 2MB, the PageTable size is proportionally reduced and the performance is good again.

When we think of using this highly recommended feature for Oracle databases, we consider only the SGA to fit into HugePages but not the PGA. Why is that? Because the documentation, MOS notes and blog posts one finds everywhere mention only the SGA. It is hard to find a mention of the PGA, except for a reminder here and there that the total amount RAM allocated for HugePages should leave enough free memory on the system for the OS and the PGA.

This alone may serve as a weak argument to say that Oracle does not allocate PGA in HugePages, but the question still stands: Is this true?

It occurs to me, and probably other DBAs too, that when I think about PGA I think of it as a single entity, similar to the SGA. While this is helpful as a simplification, the truth is in the definition: The PGA is “the single process private memory structure that contains the data and control information of a server process“. So, when I (we) think about PGA, I am actually thinking about the PGA_AGGREGATE_TARGET parameter of the database. While this simplification makes it easier to work with initial memory allocation, it also introduces a bias in reasoning when it comes to PGA and, related to the subject of this blog post, it triggers the question: Does Oracle allocate PGA in HugePages?

The long answer (including a small question)

How could the PGA get into HugePages?

As explained in this dated but accurate blog post, the HugePages implementation includes interfaces that allow private memory segments to be allocated to HugePages so, theoretically, PGA memory may be allocated here. Then it depends entirely on how Oracle has implemented the memory allocation for PGA.

In MOS note “Bigpages vs. Hugetlb on RedHat Linux (Doc ID 261889.1),” the following is written:

A program that wants to allocate shared memory has to add a flag, SHM_HUGETLB, to the shmget() flags. This approach ensures that the Oracle shared memory segments will be allocated out of this pool.
[…]
Oracle Database 10g will do this by default;

While this mentions quite old Oracle versions, I daresay that this still holds true for the newest ones, meaning that the interface used to allocate shared memory (SGA), and hence HugePages, is the shmget() call.

Now, according to Fritz Hoogland’s findings in his post “Oracle database operating system memory allocation management for PGA – part 4: Oracle 11.2.0.4 and AMM,” Oracle allocates private memory using mmap() calls as follows:

mmap(0x7f0194f7a000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 6, 0) = 0x7f0194f7a000

The above call does not include the MAP_HUGETLB flag, which is required for the mmap() call to request memory in the HugePages area, so the PGA is not allocating any HugePages and this is the Oracle developer’s chosen way.

Conclusion

Even though I’ve been unable to find hard evidence in the Oracle documentation or any MOS note, we can say that HugePages are not used for PGA.

I’ve shown evidence that this could be possible, the instrumentation is there, but Oracle does not attempt to allocate PGA memory in the HugePages area according to the mmap() calls they use.

This makes sense because most of the existing server processes will have relatively small memory requirements and allocating private memory in 2MB chunks will lead to memory exhaustion pretty quickly. As a basic example, consider an OLTP system with 2000 idle sessions. Using HugePages, the total allocated private memory would be close to 4GBs (2K*2MB), while with 4KB chunks we are talking roughly 8MBs (2K*4KB).

On the other side, given that the PGA of a process may grow quite big depending on the amount of data, think memory sorts in a DWH or DSS, while it is clear at this point that Oracle is not allocating PGA memory in HugePages, this may change in the future.