How to Prevent Linux 7 Housekeeping from Breaking Oracle

Linux 7 introduced multiple changes and enhancements to the kernel. Among them was automated housekeeping for temporary files.

In version 6, tmpwatch is available to manage unused files in temp directories. It can be scheduled via cron to automatically keep temp files in check. As of version 7, this functionality moves to systemd-tmpfiles and became a background service largely invisible to users and admins. Like tmpwatch, systemd-tmpfiles removes files and directories based on their access, modified, or created times. The default is to delete files and directories in /tmp that haven’t been accessed for 10 days, and the ones in /var/tmp that haven’t been accessed for 30 days.

That probably seems like a good idea. In many cases, it is. However, it turns out to be a very bad thing for an Oracle server.

Oracle uses the hidden /var/tmp/.oracle directory to store socket files. These are a special type of file that enable interprocess connections (IPC) on a system. They’re created by Oracle when a connection is established via the Listener and used by ASM in addition to CRS, CSS, and EVM daemons.

There are multiple Oracle Support documents that describe the consequences of deleting these files, including 370605.1, 391790.1, 1322234.1, 2099377.1, and 2492508.1. The symptoms are all… well, bad! These include the inability to connect to or start clusterware or a database, ORA-600/ORA-7445 errors, GI crashes, and SCAN failures. Resolutions nearly always requires downtime for the database or cluster.

I just encountered a situation where temp file deletion by the systemd-tmpfiles resulted in excessive virtual memory consumption and CPU load as Oracle ASM processes attempted to read deleted socket files. In this example, I was able to solve things without a restart, but the host did come close to crashing from resource starvation.

We were first alerted to growing virtual memory use on the system. By looking at the current and previous day’s swapping with sar -S, I saw the system normally used 10-12% of its swap space. Over the course of a few hours, that jumped to over 50%. I reviewed the processes using the most swap:

# List info for processes using more than 10k swap space:
for p in $(grep "VmSwap" /proc/*/status | grep -vi " 0 kb" | cut -d: -f1,3 | awk '$2 >= 10000' | sort -n -k 2 | cut -d\/ -f3)
do
egrep "^Name:|^State:|^Pid:|^Vm|switches:" /proc/$p/status
done

This showed the four biggest consumers were ASMCMD daemon processes. Each was consuming over 20G of swap:

Name:   asmcmd daemon
State:  S (sleeping)
Pid:    7533
VmPeak: 23526424 kB
VmSize: 23526424 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:  21691104 kB
VmRSS:  20972500 kB
VmData: 23254580 kB
VmStk:       140 kB
VmExe:      1912 kB
VmLib:     78352 kB
VmPTE:     45168 kB
VmSwap:  2014504 kB
voluntary_ctxt_switches:        6557964
nonvoluntary_ctxt_switches:     63522372

Running strace on these processes showed Failed to connect to ASM instance as well as a number of invalid calls to socket files that didn’t exist. The associated processes were also showing up in the ASMCMD log files. Messages for Invalid file handle for pipe /tmp/asmcmd_fg_7533 were appearing several times a second. Further analysis showed these processes were systematically grabbing additional swap from the system, with accompanying CPU increases. The database eventually began reporting ORA-700 for “[kskvmstatact: excessive swapping observed]”. In this situation, these were foreground sessions and were killed without any impact to the stability or availability of any Oracle components. The memory captured by the processes was immediately returned and load dropped to normal levels.

There was no indication any user had removed files and the directory itself was still there, but all atime/ctime/mtimes on the files were under the 10-day threshold. While not an indictment of systemd-tmpfiles directly, the system is running Linux 7 and the service is active, which are good indicators that it “helpfully” removed the socket files. Here’s another example of what can happen when files in /var/tmp/.oracle are deleted.

Fortunately, there’s an easy fix as described in Document 2498572.1. Simply tell systemd-tmpfiles to ignore the hidden .oracle directories by adding the following to the systemd-tmpfiles configuration file located at /usr/lib/tmpfiles.d/tmp.conf:

x /tmp/.oracle*
x /var/tmp/.oracle*
x /usr/tmp/.oracle*

Then, restart the service with systemctl restart systemd-tmpfiles-clean.timer.

Note that on Exadata, /var/tmp is linked to /tmp which means the 10-day access threshold on /tmp is enforced.

This is a good thing to remember when installing or working with Oracle on RHEL/CentOS/OEL 7 systems.