Overview
This post describes a recent outage during a routine Exadata (X7-2) patching to the 19.2.12.0.0.200317 release using “patchmgr.” An incorrect configuration of NTP (Network Time Protocol) on the InfiniBand (IB) switches placed all switches in the “discover” state after the patching. Even though we were using the “-rolling” flag, it continued to patch all three IB switches present in the cluster, which led to the outage. Please read on for best practices and recommendations to avoid outages while patching InfiniBand switches.
This NTP issue and some of the steps are also listed in a document recently published on My Oracle Support (MOS): Infiniband Switch rebooted as a part of Patching can caused multiple node evictions in the Cluster (Doc ID 2703311.1)
First, let’s look at the whole Exadata. Below are all the nodes with its respective versions:
Note: The details in this post are from a live production system, with the names, IPs, domains, etc., changed.
[[email protected] pythian]# ./exa-versions.sh Cluster is a X7-2 Elastic Rack HC 10TB -- Database Servers exa01db01 exa01db02 exa01db03 ------------------------------------------------------------ 19.2.12.0.0.200317 19.2.12.0.0.200317 19.2.12.0.0.200317 ------------------------------------------------------------ -- Cells exa01celadm01 exa01celadm02 exa01celadm03 exa01celadm04 exa01celadm05 exa01celadm06 ------------------------------------------------------------------------------------------------------------------------ 19.2.12.0.0.200317 19.2.12.0.0.200317 19.2.12.0.0.200317 19.2.12.0.0.200317 19.2.12.0.0.200317 19.2.12.0.0.200317 ------------------------------------------------------------------------------------------------------------------------ -- Infiniband Switches exa01sw-iba01 exa01sw-ibb01 exa01sw-ibs01 ------------------------------------------------------------ 2.2.12-2 2.2.12-2 2.2.12-2 ------------------------------------------------------------
Patching prerequisites
As usual, before patching any Exadata component, it’s important to review the health of the entire cluster, thus we’ve executed the “exachk” utility which reported no issues on the IB switches:
The “exachk” utility checks for the NTP configurations, as highlighted above, to ensure the NTP configuration is not the default. It doesn’t check either current timestamps, or if NTP is running properly:
Another mandatory step before patching any Exadata component is to execute patchmgr’s pre-upgrade checks (“precheck”). This precheck runs a series of tests to make sure the device is ready for the upgrade. Below are the details of these checks which reported no issues on the IB switches:
[[email protected] patch_19.2.12.0.0.200317]# ./patchmgr -ibswitches ~/ib_group -ibswitch_precheck -upgrade 2020-08-15 15:45:31 -0500 1 of 1 :Working: DO: Initiate pre-upgrade validation check on InfiniBand switch(es). ----- InfiniBand switch update process started 2020-08-15 15:45:32 -0500 ----- [NOTE ] Log file at /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/upgradeIBSwitch.log [INFO ] List of InfiniBand switches for upgrade: ( exa01sw-iba01 exa01sw-ibb01 exa01sw-ibs01 ) [SUCCESS ] Verifying Network connectivity to exa01sw-iba01 [SUCCESS ] Verifying Network connectivity to exa01sw-ibb01 [SUCCESS ] Verifying Network connectivity to exa01sw-ibs01 [SUCCESS ] Validating verify-topology output [INFO ] Master Subnet Manager is set to "exa01sw-ibs01" in all Switches [INFO ] ---------- Starting with InfiniBand Switch exa01sw-ibs01 [WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package. To downgrade to other versions: - Manually download the InfiniBand switch firmware package to the patch directory - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version - Run patchmgr command to initiate downgrade. [SUCCESS ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibs01. [INFO ] Starting pre-update validation on exa01sw-ibs01 [SUCCESS ] Verifying that /tmp has 150M in exa01sw-ibs01, found 492M [SUCCESS ] Verifying that / has 20M in exa01sw-ibs01, found 26M [SUCCESS ] NTP daemon is running on exa01sw-ibs01. [INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 15:55:04 [INFO ] Validating the current firmware on the InfiniBand Switch [SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ibs01 [SUCCESS ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibs01 through getHostByName [SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ibs01 [INFO ] Finished pre-update validation on exa01sw-ibs01 [SUCCESS ] Pre-update validation on exa01sw-ibs01 [SUCCESS ] Prereq check on exa01sw-ibs01 [INFO ] ---------- Starting with InfiniBand Switch exa01sw-iba01 [WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package. To downgrade to other versions: - Manually download the InfiniBand switch firmware package to the patch directory - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version - Run patchmgr command to initiate downgrade. [SUCCESS ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-iba01. [INFO ] Starting pre-update validation on exa01sw-iba01 [SUCCESS ] Verifying that /tmp has 150M in exa01sw-iba01, found 492M [SUCCESS ] Verifying that / has 20M in exa01sw-iba01, found 26M [SUCCESS ] NTP daemon is running on exa01sw-iba01. [INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 17:34:39 [INFO ] Validating the current firmware on the InfiniBand Switch [SUCCESS ] Firmware verification on InfiniBand switch exa01sw-iba01 [SUCCESS ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-iba01 through getHostByName [SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-iba01 [INFO ] Finished pre-update validation on exa01sw-iba01 [SUCCESS ] Pre-update validation on exa01sw-iba01 [SUCCESS ] Prereq check on exa01sw-iba01 [INFO ] ---------- Starting with InfiniBand Switch exa01sw-ibb01 [WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package. To downgrade to other versions: - Manually download the InfiniBand switch firmware package to the patch directory - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version - Run patchmgr command to initiate downgrade. [SUCCESS ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibb01. [INFO ] Starting pre-update validation on exa01sw-ibb01 [SUCCESS ] Verifying that /tmp has 150M in exa01sw-ibb01, found 492M [SUCCESS ] Verifying that / has 20M in exa01sw-ibb01, found 26M [SUCCESS ] NTP daemon is running on exa01sw-ibb01. [INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:24:46 [INFO ] Validating the current firmware on the InfiniBand Switch [SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ibb01 [SUCCESS ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibb01 through getHostByName [SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ibb01 [INFO ] Finished pre-update validation on exa01sw-ibb01 [SUCCESS ] Pre-update validation on exa01sw-ibb01 [SUCCESS ] Prereq check on exa01sw-ibb01 [SUCCESS ] Overall status ----- InfiniBand switch update process ended 2020-08-15 15:47:41 -0500 ----- 2020-08-15 15:47:41 -0500 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es). 2020-08-15 15:47:41 -0500 :SUCCESS: Completed run of command: ./patchmgr -ibswitches /root/ib_group -ibswitch_precheck -upgrade 2020-08-15 15:47:41 -0500 :INFO : upgrade attempted on nodes in file /root/ib_group: [exa01sw-iba01 exa01sw-ibb01 exa01sw-ibs01] 2020-08-15 15:47:41 -0500 :INFO : For details, check the following files in /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317: 2020-08-15 15:47:41 -0500 :INFO : - upgradeIBSwitch.log 2020-08-15 15:47:41 -0500 :INFO : - upgradeIBSwitch.trc 2020-08-15 15:47:41 -0500 :INFO : - patchmgr.stdout 2020-08-15 15:47:41 -0500 :INFO : - patchmgr.stderr 2020-08-15 15:47:41 -0500 :INFO : - patchmgr.log 2020-08-15 15:47:41 -0500 :INFO : - patchmgr.trc 2020-08-15 15:47:41 -0500 :INFO : Exit status:0 2020-08-15 15:47:41 -0500 :INFO : Exiting.
Patching the switches / full outage
After validating all prerequisites, “patchmgr” successfully applied the patch to the switches in “standby” state:
[[email protected] patch_19.2.12.0.0.200317]# ./patchmgr -ibswitches ~/ib_group -upgrade ----- InfiniBand switch update process started 2020-08-15 15:49:39 -0500 ----- [NOTE ] Log file at /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/upgradeIBSwitch.log [INFO ] List of InfiniBand switches for upgrade: ( exa01sw-iba01 exa01sw-ibb01 exa01sw-ibs01 ) [SUCCESS ] Verifying Network connectivity to exa01sw-iba01 [SUCCESS ] Verifying Network connectivity to exa01sw-ibb01 [SUCCESS ] Verifying Network connectivity to exa01sw-ibs01 [SUCCESS ] Validating verify-topology output [INFO ] Proceeding with upgrade of InfiniBand switches to version 2.2.14_1 [INFO ] Master Subnet Manager is set to "exa01sw-ibs01" in all Switches [INFO ] ---------- Starting with InfiniBand Switch exa01sw-ibs01 [WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package. To downgrade to other versions: - Manually download the InfiniBand switch firmware package to the patch directory - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version - Run patchmgr command to initiate downgrade. [SUCCESS ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibs01. [INFO ] Starting pre-update validation on exa01sw-ibs01 [SUCCESS ] Verifying that /tmp has 150M in exa01sw-ibs01, found 492M [SUCCESS ] Verifying that / has 20M in exa01sw-ibs01, found 26M [SUCCESS ] Service opensmd is running on InfiniBand Switch exa01sw-ibs01 [SUCCESS ] NTP daemon is running on exa01sw-ibs01. [INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 15:59:12 [INFO ] Validating the current firmware on the InfiniBand Switch [SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ibs01 [SUCCESS ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibs01 through getHostByName [SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ibs01 [INFO ] Finished pre-update validation on exa01sw-ibs01 [SUCCESS ] Pre-update validation on exa01sw-ibs01 [INFO ] Package will be downloaded at firmware update time via scp [SUCCESS ] Execute plugin check for Patching on exa01sw-ibs01 [INFO ] Starting upgrade on exa01sw-ibs01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade [INFO ] Rebooting exa01sw-ibs01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH [SUCCESS ] Load firmware 2.2.14_1 onto exa01sw-ibs01 [SUCCESS ] Verify that /conf/configvalid is set to 1 on exa01sw-ibs01 [INFO ] Set SMPriority to 8 on exa01sw-ibs01 [INFO ] Starting post-update validation on exa01sw-ibs01 [SUCCESS ] Service opensmd is running on InfiniBand Switch exa01sw-ibs01 [SUCCESS ] NTP daemon is running on exa01sw-ibs01. [INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:07:02 [INFO ] /conf/configvalid is 1 [INFO ] Validating the current firmware on the InfiniBand Switch [SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ibs01 [SUCCESS ] Execute plugin check for Post Patch on exa01sw-ibs01 [INFO ] Finished post-update validation on exa01sw-ibs01 [SUCCESS ] Post-update validation on exa01sw-ibs01 [SUCCESS ] Update InfiniBand switch exa01sw-ibs01 to 2.2.14_1 [INFO ] ---------- Starting with InfiniBand Switch exa01sw-ibb01 [WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package. To downgrade to other versions: - Manually download the InfiniBand switch firmware package to the patch directory - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version - Run patchmgr command to initiate downgrade. [SUCCESS ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibb01. [INFO ] Starting pre-update validation on exa01sw-ibb01 [SUCCESS ] Verifying that /tmp has 150M in exa01sw-ibb01, found 492M [SUCCESS ] Verifying that / has 20M in exa01sw-ibb01, found 26M [SUCCESS ] Service opensmd is running on InfiniBand Switch exa01sw-ibb01 [SUCCESS ] NTP daemon is running on exa01sw-ibb01. [INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:45:01 [INFO ] Validating the current firmware on the InfiniBand Switch [SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ibb01 [SUCCESS ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibb01 through getHostByName [SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ibb01 [INFO ] Finished pre-update validation on exa01sw-ibb01 [SUCCESS ] Pre-update validation on exa01sw-ibb01 [INFO ] Package will be downloaded at firmware update time via scp [SUCCESS ] Execute plugin check for Patching on exa01sw-ibb01 [INFO ] Starting upgrade on exa01sw-ibb01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade [INFO ] Rebooting exa01sw-ibb01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH [SUCCESS ] Load firmware 2.2.14_1 onto exa01sw-ibb01 [SUCCESS ] Verify that /conf/configvalid is set to 1 on exa01sw-ibb01 [INFO ] Set SMPriority to 5 on exa01sw-ibb01 [INFO ] Starting post-update validation on exa01sw-ibb01 [SUCCESS ] Service opensmd is running on InfiniBand Switch exa01sw-ibb01 [SUCCESS ] NTP daemon is running on exa01sw-ibb01. [INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:24:14 [INFO ] /conf/configvalid is 1 [INFO ] Validating the current firmware on the InfiniBand Switch [SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ibb01 [SUCCESS ] Execute plugin check for Post Patch on exa01sw-ibb01 [INFO ] Finished post-update validation on exa01sw-ibb01 [SUCCESS ] Post-update validation on exa01sw-ibb01 [SUCCESS ] Update InfiniBand switch exa01sw-ibb01 to 2.2.14_1
After successfully patching the two “standby” switches, “patchmgr” proceeded to patch the “master” switch (note that “patchmgr” doesn’t check for the actual statuses of the IB switches after the patching). When it rebooted, all nodes lost connectivity, as there were no surviving switches in the “standby” state to manage the connections. This abruptly interrupted “patchmgr”:
[INFO ] ---------- Starting with InfiniBand Switch exa01sw-iba01
[WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
To downgrade to other versions:
- Manually download the InfiniBand switch firmware package to the patch directory
- Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
- Run patchmgr command to initiate downgrade.
[SUCCESS ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-iba01.
[INFO ] Starting pre-update validation on exa01sw-iba01
[SUCCESS ] Verifying that /tmp has 150M in exa01sw-iba01, found 492M
[SUCCESS ] Verifying that / has 20M in exa01sw-iba01, found 26M
[SUCCESS ] Service opensmd is running on InfiniBand Switch exa01sw-iba01
[SUCCESS ] NTP daemon is running on exa01sw-iba01.
[INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 18:12:35
[INFO ] Validating the current firmware on the InfiniBand Switch
[SUCCESS ] Firmware verification on InfiniBand switch exa01sw-iba01
[SUCCESS ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-iba01 through getHostByName
[SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-iba01
[INFO ] Finished pre-update validation on exa01sw-iba01
[SUCCESS ] Pre-update validation on exa01sw-iba01
[INFO ] Package will be downloaded at firmware update time via scp
[SUCCESS ] Execute plugin check for Patching on exa01sw-iba01
[INFO ] Starting upgrade on exa01sw-iba01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO ] Rebooting exa01sw-iba01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
As seen above, the last operation executed by “patchmgr” was the reboot of exa01sw-iba01. As previously explained, since there were no nodes in the “standby” state, we lost connectivity to all the nodes and had to log into the DB nodes using the ILOM:
[[email protected] ~]$ ssh [email protected] Password: Oracle(R) Integrated Lights Out Manager Version 4.0.4.52 r133103 Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. Warning: HTTPS certificate is set to factory default. Hostname: exa01dbadm01-ilom -> start /SP/Console Are you sure you want to start /SP/console (y/n)? y Serial console started. To stop, type ESC ( exa01db01 login: root Password: Last login: Sat Aug 15 16:26:07 CDT 2020 from exa01db01.example.com on ssh Last login: Sat Aug 15 16:31:59 on ttyS0 [[email protected] ~]#
Troubleshooting
All IB switches returned to “standby” state when the clocks reached the pre-reboot timepoint. Given the complexity of this issue, we engaged Oracle to assist in the analysis.
We lost all connectivity, as the three switches were in “discover” state. By the time we logged back into the system, ibs01 had already become the “master.” The other two switches remained in the “boot in progress” state for some time:
[[email protected] ~]# ibswitches Switch : 0x0010e0dc11fba0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.ZZ" enhanced port 0 lid 3 lmc 0 Switch : 0x0010e0dc1ceda0a0 ports 36 "BOOT IN PROGRESS" enhanced port 0 lid 0 lmc 0 Switch : 0x0010e0dc0f6ea0a0 ports 36 "BOOT IN PROGRESS" enhanced port 0 lid 0 lmc 0 [[email protected] ~]# getmaster Local SM enabled and running, state MASTER Last change in Master SubnetManager status detected at: Sat Aug 15 16:25:50 CDT 2020 Master SubnetManager on sm lid 3 sm guid 0x10e0dc11fba0a0 : SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.ZZ Master SubnetManager Activity Count: 11637 Priority: 14
The Oracle engineer identified the root cause of this issue as the incorrect configuration of NTP. This configuration error led to a clock difference on the switches, fixed by Linux after the reboot, thus moving the clock backward. This caused the switch to enter “discover” state and become unusable. He pinpointed the issue by detecting a time drift on the /var/log/messages file of the IB switch:
Aug 15 18:16:50 exa01sw-iba01 envd[2532]: Connector 13B present Aug 15 16:29:05 exa01sw-iba01 envd[2532]: Connector 14B present
The switches reverted to “standby” state when the system clock reached the same time it had before the reboot.
Solution
To avoid running into an outage, it’s preferable to patch switches one at a time and execute a series of checks on each switch after patching, before proceeding.
NOTE: Only perform changes using the “Ilom-admin” (ILOM shell) account. Do not perform any changes on the IB switches with the “root” (regular shell) account.
Configure and enable NTP
- Carefully review the NTP settings on each switch:
Hostname: exa01sw-iba01.example.com -> show /SP/clock /SP/clock Targets: Properties: datetime = Mon Aug 17 08:58:58 2020 timezone = CDT (America/Chicago) uptime = 0 days, 23:06:02 usentpserver = disabled Commands: cd set show -> show /SP/clients/ntp/server/1 /SP/clients/ntp/server/1 Targets: Properties: address = 0.0.0.0 Commands: cd set show -> show /SP/clients/ntp/server/2 /SP/clients/ntp/server/2 Targets: Properties: address = 0.0.0.0 Commands: cd set show -> exit [[email protected] ~]# ls -l /etc/ntp.conf lrwxrwxrwx 1 root root 21 2015-09-10 11:37 /etc/ntp.conf -> /config/conf/ntp.conf [[email protected] ~]# ls -l /etc/default/ntpdate lrwxrwxrwx 1 root root 20 2016-03-31 07:39 /etc/default/ntpdate -> /config/conf/ntpdate [[email protected] ~]# cat /config/conf/ntp.conf ## DO NOT EDIT THIS FILE ## #server none #server none driftfile /conf/ntp-drift pidfile /var/run/ntpd.pid [[email protected] ~]# cat /config/conf/ntpdate ## DO NOT EDIT THIS FILE ## # servers to check. (Separate multiple servers with spaces.) NTPSERVERS="XXX.YYY.ZZZ.AAA XXX.BBB.CCC.AAA" # # additional options for ntpdate NTPOPTIONS="-u"The “root” account (regular shell) changed the NTP’s settings only at the OS level. During the reboot Linux invoked “ntpdate” and adjusted the system’s clock (backward) using the OS configurations, causing a significant drift that placed the switches in “discover” state.
- Change to the “ilom-admin” (ILOM shell) account and adjust the configurations, before enabling NTP.
Hostname: exa01sw-ibb01.example.com -> show /SP/clients/ntp/server/1 /SP/clients/ntp/server/1 Targets: Properties: address = 0.0.0.0 Commands: cd set show -> show /SP/clients/ntp/server/2 /SP/clients/ntp/server/2 Targets: Properties: address = 0.0.0.0 Commands: cd set show -> show /SP/clock /SP/clock Targets: Properties: datetime = Sat Aug 22 08:25:29 2020 timezone = CDT (US/Central) uptime = 89 days, 13:42:39 usentpserver = disabled Commands: cd set show -> set /sp/clock/ usentpserver=disabled Set 'usentpserver' to 'disabled' -> set /sp/clients/ntp/server/1/ address=XXX.YYY.ZZZ.AAA Set 'address' to 'XXX.YYY.ZZZ.AAA' -> set /sp/clients/ntp/server/2/ address=XXX.BBB.CCC.AAA Set 'address' to 'XXX.BBB.CCC.AAA' -> set /sp/clock/ usentpserver=enabled Set 'usentpserver' to 'enabled' -> show /SP/clock /SP/clock Targets: Properties: datetime = Sat Aug 22 08:31:05 2020 timezone = CDT (US/Central) uptime = 89 days, 13:45:48 usentpserver = enabled Commands: cd set show -> show /SP/clients/ntp/server/1 /SP/clients/ntp/server/1 Targets: Properties: address = XXX.YYY.ZZZ.AAA Commands: cd set show -> show /SP/clients/ntp/server/2 /SP/clients/ntp/server/2 Targets: Properties: address = XXX.BBB.CCC.AAA Commands: cd set show -> exit [[email protected] ~]# date Sat Aug 22 08:31:13 CDT 2020 - After enabling NTP make sure the switch is in the “standby” state before proceeding to the next switch. Do not reboot the “master” switch If there are no remaining switches in “standby” state. Check the current state with the “getmaster” command in the regular shell of the IB switch:
[[email protected] ~]# getmaster Local SM enabled and running, state STANDBY Last change in Master SubnetManager status detected at: Sun May 24 08:20:16 CDT 2020 Master SubnetManager on sm lid 1 sm guid 0x10e0dc218da0a0 : SUN DCS 36P QDR exa01sw-iba01 Master SubnetManager Activity Count: 57235558 Priority: 14
Adjusting the Subnet Manager (SM) if the clock moved forward
In our scenario, the clock moved forward causing multiple master IB switches on the fabric. This led to connectivity issues on one node. Rotating the SM to a switch that has the correct timestamp will fix this:
- Check the state of all switches in the cluster:
[[email protected] pythian]# dcli -l root -g ~/ibs_group: getmaster | grep state exa01sw-iba01: Local SM enabled and running, state STANDBY exa01sw-ibb01: Local SM enabled and running, state STANDBY exa01sw-ibs01: Local SM enabled and running, state MASTER
- Disable SM on the current “master” switch:
[[email protected] pythian]# ssh exa01sw-ibs01 You are now logged in to the root shell. It is recommended to use ILOM shell instead of root shell. All usage should be restricted to documented commands and documented config files. To view the list of documented commands, use "help" at linux prompt. [[email protected] ~]# disablesm Stopping partcfgchk [ OK ] Stopping partitiond-daemon [ OK ] Stopping IB Subnet Manager..-. [ OK ] [[email protected] ~]# getmaster Local SM not enabled
- Re-enable SM:
[[email protected] ~]# enablesm Starting IB Subnet Manager. [ OK ] Starting partitiond-daemon [ OK ] Starting partcfgchk [ OK ] [[email protected] ~]# getmaster Local SM enabled and running, state STANDBY Last change in Master SubnetManager status detected at: Sun May 24 08:20:16 CDT 2020 Master SubnetManager on sm lid 1 sm guid 0x10e0dc218da0a0 : SUN DCS 36P QDR exa01sw-iba01 XX.YY.ZZ.ZZ Master SubnetManager Activity Count: 615361 Priority: 14
- Check the state of all switches in the cluster: (When SM was disabled on ibs01, iba01 picked up as “master”)
[[email protected] pythian]# dcli -l root -g ~/ibs_group: getmaster | grep state exa01sw-iba01: Local SM enabled and running, state MASTER exa01sw-ibb01: Local SM enabled and running, state STANDBY exa01sw-ibs01: Local SM enabled and running, state STANDBY
Patching the switches with “patchmgr”
- It’s preferable to patch one switch at a time since “patchmgr” doesn’t check the state of the IB switch. Specify a single switch in the group’s file as follows:
[[email protected] ~]# cat ~/ib_group exa01sw-ibs01 [[email protected] ~]# cd /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/ [[email protected] patch_19.2.12.0.0.200317]# ./patchmgr -ibswitches ~/ib_group -upgrade 2020-08-15 15:49:38 -0500 1 of 1 :Working: DO: Initiate upgrade of InfiniBand switches to 2.2.14-1. Expect up to 40 minutes for each switch ----- InfiniBand switch update process started 2020-08-15 15:49:39 -0500 ----- [NOTE ] Log file at /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/upgradeIBSwitch.log [INFO ] List of InfiniBand switches for upgrade: ( exa01sw-ibs01 ) [SUCCESS ] Verifying Network connectivity to exa01sw-ibs01 [SUCCESS ] Validating verify-topology output [INFO ] Proceeding with upgrade of InfiniBand switches to version 2.2.14_1 [INFO ] Master Subnet Manager is set to "exa01sw-iba01" in all Switches [INFO ] ---------- Starting with InfiniBand Switch exa01sw-ibs01 [WARNING ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package. To downgrade to other versions: - Manually download the InfiniBand switch firmware package to the patch directory - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version - Run patchmgr command to initiate downgrade. [SUCCESS ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibs01. [INFO ] Starting pre-update validation on exa01sw-ibs01 [SUCCESS ] Verifying that /tmp has 150M in exa01sw-ibs01, found 492M [SUCCESS ] Verifying that / has 20M in exa01sw-ibs01, found 26M [SUCCESS ] Service opensmd is running on InfiniBand Switch exa01sw-ibs01 [SUCCESS ] NTP daemon is running on exa01sw-ibs01. [INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 15:59:12 [INFO ] Validating the current firmware on the InfiniBand Switch [SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ibs01 [SUCCESS ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibs01 through getHostByName [SUCCESS ] Execute plugin check for Patch Check Prereq on exa01sw-ibs01 [INFO ] Finished pre-update validation on exa01sw-ibs01 [SUCCESS ] Pre-update validation on exa01sw-ibs01 [INFO ] Package will be downloaded at firmware update time via scp [SUCCESS ] Execute plugin check for Patching on exa01sw-ibs01 [INFO ] Starting upgrade on exa01sw-ibs01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade [INFO ] Rebooting exa01sw-ibs01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH [SUCCESS ] Load firmware 2.2.14_1 onto exa01sw-ibs01 [SUCCESS ] Verify that /conf/configvalid is set to 1 on exa01sw-ibs01 [INFO ] Set SMPriority to 8 on exa01sw-ibs01 [INFO ] Starting post-update validation on exa01sw-ibs01 [SUCCESS ] Service opensmd is running on InfiniBand Switch exa01sw-ibs01 [SUCCESS ] NTP daemon is running on exa01sw-ibs01. [INFO ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:07:02 [INFO ] /conf/configvalid is 1 [INFO ] Validating the current firmware on the InfiniBand Switch [SUCCESS ] Firmware verification on InfiniBand switch exa01sw-ibs01 [SUCCESS ] Execute plugin check for Post Patch on exa01sw-ibs01 [INFO ] Finished post-update validation on exa01sw-ibs01 [SUCCESS ] Post-update validation on exa01sw-ibs01 [SUCCESS ] Update InfiniBand switch exa01sw-ibs01 to 2.2.14_1 ----- InfiniBand switch update process ended 2020-08-15 16:07:41 -0500 ----- 2020-08-15 16:07:41 -0500 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es). 2020-08-15 16:07:41 -0500 :SUCCESS: Completed run of command: ./patchmgr -ibswitches /root/ib_group -ibswitch_precheck -upgrade 2020-08-15 16:07:41 -0500 :INFO : upgrade attempted on nodes in file /root/ib_group: [exa01sw-ibs01] 2020-08-15 16:07:41 -0500 :INFO : For details, check the following files in /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317: 2020-08-15 16:07:41 -0500 :INFO : - upgradeIBSwitch.log 2020-08-15 16:07:41 -0500 :INFO : - upgradeIBSwitch.trc 2020-08-15 16:07:41 -0500 :INFO : - patchmgr.stdout 2020-08-15 16:07:41 -0500 :INFO : - patchmgr.stderr 2020-08-15 16:07:41 -0500 :INFO : - patchmgr.log 2020-08-15 16:07:41 -0500 :INFO : - patchmgr.trc 2020-08-15 16:07:41 -0500 :INFO : Exit status:0 2020-08-15 16:07:41 -0500 :INFO : Exiting.
- Perform a thorough check on each patched switch before moving on to the next:
# Make sure that the switch is in the "STANDBY" state after the patch [[email protected] ~]# getmaster Local SM enabled and running, state STANDBY Last change in Master SubnetManager status detected at: Sat Aug 15 17:34:55 CDT 2020 Master SubnetManager on sm lid 3 sm guid 0x10e0dc11fba0a0 : SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.BB Master SubnetManager Activity Count: 615361 Priority: 14 # Make sure that all IB switches are listed with the "ibswitches" command [[email protected] ~]# ibswitches Switch : 0x0010e0dc0f6ea0a0 ports 36 "SUN DCS 36P QDR exa01sw-iba01 10.XX.YY.ZZ" enhanced port 0 lid 2 lmc 0 Switch : 0x0010e0dc11fba0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.AA" enhanced port 0 lid 3 lmc 0 Switch : 0x0010e0dc1ceda0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibb01 10.XX.YY.BB" enhanced port 0 lid 19 lmc 0 # Check switch version [[email protected] ~]# version SUN DCS 36p version: 2.2.14-1 Build time: Aug 14 2019 08:48:04 SP board info: Manufacturing Date: N/A Serial Number: "XXXXXXXX" Hardware Revision: 0x0100 Firmware Revision: 0x0000 BIOS version: NUP1R918 BIOS date: 01/19/2016 # Check the date on all the switches [[email protected] ~]# cat ~/ibs_group exa01sw-iba01 exa01sw-ibb01 exa01sw-ibs01 [[email protected] ~]# dcli -g ~/ibs_group -l root date
Patching the switches manually
Since “patchmgr” couldn’t patch the final IB switch, Oracle recommended patching it manually. The following document describes the steps required to do so: https://docs.oracle.com/cd/E76424_01/pdf/E76431.pdf (Section: Upgrade the Switch Firmware (CLI).
- First log into the switch, check its version, SM status, and verify the firmware:
[[email protected] ~]# getmaster Local SM enabled and running, state STANDBY Last change in Master SubnetManager status detected at: Sat Aug 15 17:34:55 CDT 2020 Master SubnetManager on sm lid 3 sm guid 0x10e0dc11fba0a0 : SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.ZZ Master SubnetManager Activity Count: 615361 Priority: 14 [[email protected] ~]# fwverify -s Checking all present packages: .............................................................................................................................................................................................................................................. OK Checking if any packages are missing: ............................................................................................................................................................................................................................................. OK Checking FW Coreswitch: FW Version: 7.4.3002 OK PSID: SUN_NM2-36p_006 OK Verifying image integrity OK
- Download the required “pkg” file for your target version and compare the checksum with that provided by Oracle:
[[email protected] pythian]# cd /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317 [[email protected] patch_19.2.12.0.0.200317]# md5sum sundcs_36p_repository_2.2.14_1.pkg > pkg.md5 [[email protected] patch_19.2.12.0.0.200317]# cat pkg.md5 ce05e4e207b9718b01847634dbbd9bc4 sundcs_36p_repository_2.2.14_1.pkg
- Log into the switch, check its version, then start the upgrade using the ILOM (ilom-admin):
[[email protected] patch_19.2.12.0.0.200317]# ssh exa01sw-iba01 You are now logged in to the root shell. It is recommended to use ILOM shell instead of root shell. All usage should be restricted to documented commands and documented config files. To view the list of documented commands, use "help" at linux prompt. [[email protected] ~]# version SUN DCS 36p version: 2.2.12-2 Build time: Oct 29 2018 08:35:27 SP board info: Manufacturing Date: N/A Serial Number: "XXXXXXXX" Hardware Revision: 0x0100 Firmware Revision: 0x0000 BIOS version: NUP1R918 BIOS date: 01/19/2016 [[email protected] ~]# sudo su - ilom-admin Oracle(R) Integrated Lights Out Manager Version 2.2.12-2 ILOM 3.2.11 r127844 Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved. Warning: HTTPS certificate is set to factory default. Hostname: exa01sw-iba01.example.com -> load -source scp://root:[email protected]//u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/sundcs_36p_repository_2.2.14_1.pkg Downloading firmware image. This will take a few minutes. SUN DCS 36p version: 2.2.12-2 Build time: Oct 29 2018 08:35:27 SP board info: Manufacturing Date: N/A Serial Number: "XXXXXXXX" Hardware Revision: 0x0100 Firmware Revision: 0x0000 BIOS version: NUP1R918 BIOS date: 01/19/2016 FROM_VERSION: 2.2.12-2 TO_VERSION: 2.2.14-1 NOTE: Firmware upgrade will upgrade the SUN DCS 36p firmware. ILOM will enter a special mode to load new firmware. No other tasks should be performed in ILOM until the firmware upgrade is complete. Are you sure you want to load the specified file (y/n)? y Setting up environment for firmware upgrade. This will take a few minutes. Subnet manager has been temporarily disabled on this switch so that firmware upgrade can go on. After the upgrade is done, please make sure that the SM is running again, using commands sminfo and enablesm. Starting SUN DCS 36p FW update ========================== Performing operation: I4 A ========================== I4 A: I4 is already at the given version. ========================================= Performing operation: SUN DCS 36p firmware update ========================================= SUN DCS 36p Kontron module fw upgrade from 2.2.12-2 to 2.2.14-1: Please reboot the system to enable firmware update of Kontron module. The download of the Kontron firmware image happens during reboot. After system reboot, Kontron FW update progress can be monitored in browser using URL [http://system] OR at OS command line prompt by using command [telnet system 1234] where system is the hostname or IP address of SUN DCS 36P or GW. Firmware update is complete. -> reset /SP Are you sure you want to reset /SP (y/n)? y Performing reset on /SP Broadcast message from [email protected] (unknown) at 9:50 ... The system is going down for reboot NOW!
- Check the status of the switch, the SM, and its version to make sure the upgrade was successful:
[[email protected] ~]# version SUN DCS 36p version: 2.2.14-1 Build time: Aug 14 2019 08:48:04 SP board info: Manufacturing Date: N/A Serial Number: "XXXXXXXX" Hardware Revision: 0x0100 Firmware Revision: 0x0000 BIOS version: NUP1R918 BIOS date: 01/19/2016 [[email protected] ~]# sminfo sminfo: sm lid 3 sm guid 0x10e0dc11fba0a0, activity count 644470 priority 14 state 3 SMINFO_MASTER [[email protected] ~]# ibswitches Switch : 0x0010e0dc0f6ea0a0 ports 36 "SUN DCS 36P QDR exa01sw-iba01 10.XX.YY.AA" enhanced port 0 lid 2 lmc 0 Switch : 0x0010e0dc11fba0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.BB" enhanced port 0 lid 3 lmc 0 Switch : 0x0010e0dc1ceda0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibb01 10.XX.YY.CC" enhanced port 0 lid 19 lmc 0 [[email protected] ~]# getmaster Local SM enabled and running, state STANDBY Last change in Master SubnetManager status detected at: Sat Aug 15 17:34:55 CDT 2020 Master SubnetManager on sm lid 3 sm guid 0x10e0dc11fba0a0 : SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.CC Master SubnetManager Activity Count: 615361 Priority: 14
NOTE: Do not proceed with any other activities until the switch is in the “standby” state.
Check the status of the cluster
Finally, use Fred Denis’ amazing “rac-status.sh” script to check the status of the entire cluster:
[[email protected] pythian]# ./rac-status.sh -a Cluster exa01-clu1 is a X7-2 Elastic Rack HC 10TB Type | Name | db01 | db02 | db03 | ------------------------------------------------------------------ asm | asm | Online | Online | Online | cvu | cvu | - | - | Online | dg | DATAC1 | Online | Online | Online | dg | RECOC1 | Online | Online | Online | network | net1 | Online | Online | Online | ons | ons | Online | Online | Online | proxy_advm | proxy_advm | Offline x| Offline x| Offline x| qosmserver | qosmserver | Offline x| Offline x| Offline x| vip | db01 | Online | - | - | vip | db02 | - | Online | - | vip | db03 | - | - | Online | vip | scan1 | - | Online | - | vip | scan2 | - | - | Online | vip | scan3 | Online | - | - | ------------------------------------------------------------------ x : Resource is disabled : Has been restarted less than 24 hours ago : STATUS and TARGET are different Listener | Port | db01 | db02 | db03 | Type | --------------------------------------------------------------------------------- ASMNET1LSNR_ASM| TCP:1525 | Online | Online | Online | Listener | LISTENER | TCP:1521 | Online | Online | Online | Listener | LISTENER_LEAF | | x | x | x | Listener | LISTENER_SCAN1 | TCP:1521 | - | Online | - | SCAN | LISTENER_SCAN2 | TCP:1521 | - | - | Online | SCAN | LISTENER_SCAN3 | TCP:1521 | Online | - | - | SCAN | --------------------------------------------------------------------------------- x : Listener is disabled : Has been restarted less than 24 hours ago DB | Service | db01 | db02 | db03 | ------------------------------------------------------------------ prim01 | prim01_batch | - | - | Online | | prim01_ro | Online | Online | Online | stdby | stdby | Online | Online | Online | | stdby_ro | Online | Online | Online | ------------------------------------------------------------------ : Has been restarted less than 24 hours ago : STATUS and TARGET are different DB | Version | db01 | db02 | db03 | DB Type | --------------------------------------------------------------------------------- prim01 | 18.3.0.0 (1) | Open | Open | Open | RAC (P) | stdby | 12.2.0.1 (2) | Readonly | Readonly | Readonly | RAC (S) | --------------------------------------------------------------------------------- ORACLE_HOME references listed in the Version column ("''" means "same as above") 1 : /u01/app/oracle/product/18.3.0.0/dbhome_4 oracle oinstall 2 : /u01/app/oracle/product/12.2.0.1/dbhome_1 '' '' x : Instance is disabled : Has been restarted less than 24 hours ago
Conclusion
As demonstrated throughout this post, even though there are a number of checks to try and prevent issues while patching an Exadata component, it’s nearly impossible to cover all bases. Besides using “exachk” and patchmgr’s prechecks, it’s also important to create your own checklist to preventively detect issues not picked up by those utilities, as well as performing sanity checks after a component is patched to ensure it’s working as expected.
If you have any questions, please feel free to send me a comment and I’ll reply as soon as possible.
Related resources
Fred Denis wrote both scripts used in this post. To get the latest version, please, check Fred’s GitHub page: https://github.com/freddenis/oracle-scripts/
Details on each of the scripts used here:
- rac-status.sh: Displays the current status of the cluster and all its targets (-a).
- exa-versions.sh: Displays the current version of each component in the Exadata stack (Cell nodes (-c), DB nodes (-d), IB switches (-i)).
2 Comments. Leave new
Fantastic post with great detail and examples – thanks Fernando
Glad you liked it, Neil! Keep tuned for more! :)