So, after an abnormally quiet weekend, I was being paged: Oracle Management Services (OMS) was down!
Well, this requires monitoring the monitoring, right? So, how to monitor the monitoring?
Oracle Enterprise Manager (OEM) 13c has a feature called Out of Band Notification, which allows configuring an agent with email credentials to send notifications when it is not able to communicate with OMS and Repo DB because they are down.
Details of that configuration are on this MOS note: EM 13c, 12c: How to Set Up Out Of Band Email Notification in Enterprise Manager Cloud Control (Doc ID 1472854.1)
How does it work?
The agent on the OMS host checks the status of the ‘OMS and repository’ target (oracle_emrep) by running the metric ‘Response’ which runs the Perl script:
If the oracle_emrep target is detected as down, then emrepdown.pl will call on the same directory.
The emrepdown.pl uses the Perl “Net::SMTP” method to send an email using the Out Of Band email information (To Email ID, Email Gateway, From Email ID) defined in the agent’s /sysman/config/emd.properties configuration file.
Note: This method does not currently support SSL email authentication. An internal ER has been raised for this.
How to set this up?
1) Run the following commands which will set the email parameters in the emd.properties file.
Do this on the chained agent (ie. the agent on the same machine as the OMS which monitors the oracle_emrep target)
a) Set the agent ORACLE_HOME
$ export ORACLE_HOME= $ export PATH= ORACLE_HOME/bin:$PATH
$ export ORACLE_HOME=/oracle/13c/13cagent/core/188.8.131.52.0 $ export PATH=$ORACLE_HOME/bin:$PATH
b) Check if any values are currently set for the Out of Band parameters:
$ emctl getproperty agent -name emd_email_address $ emctl getproperty agent -name emd_from_email_address $ emctl getproperty agent -name emd_email_gateway
If the message is returned…
emd_email_address is not a valid configuration property
… it means that this is not yet set up. Continue to the next section.
c) Set the Out of Band parameters
emctl setproperty agent -allow_new -name emd_email_address -value [youremailaddress] emctl setproperty agent -allow_new -name emd_from_email_address -value [senderAddress] emctl setproperty agent -allow_new -name emd_email_gateway -value [outgoingsmtpserver]
$ emctl setproperty agent -allow_new -name emd_email_gateway -value smtp.server.hostname $ emctl setproperty agent -allow_new -name emd_email_address -value [email protected] $ emctl setproperty agent -allow_new -name emd_from_email_address -value [email protected]
Tip: The value for the emd_email_gateway can be the same as is used for ‘normal’ email notifications via the OMS. This can be accessed via setup/notifications/notification methods.
If you need to use “Use Secure Connection:SSL” normally, then this means that your mail server requires SSL authentication, meaning that the OOB method will not be suitable. Remember: the OOB method does not support SSL email authentication at this time.
2) Stop and start the agent for these parameters to take effect.
More information like to test this configuration can be found on MOS note: EM 13c, 12c: How to Set Up Out Of Band Email Notification in Enterprise Manager Cloud Control (Doc ID 1472854.1)
Hope that helps, cheers!