Exadata’s InfiniBand Switch: Incorrect NTP settings Leading to Evictions, and Patching Recommendations

Posted in: Technical Track

Overview

This post describes a recent outage during a routine Exadata (X7-2) patching to the 19.2.12.0.0.200317 release using “patchmgr.” An incorrect configuration of NTP (Network Time Protocol) on the InfiniBand (IB) switches placed all switches in the “discover” state after the patching. Even though we were using the “-rolling” flag, it continued to patch all three IB switches present in the cluster, which led to the outage. Please read on for best practices and recommendations to avoid outages while patching InfiniBand switches.

This NTP issue and some of the steps are also listed in a document recently published on My Oracle Support (MOS): Infiniband Switch rebooted as a part of Patching can caused multiple node evictions in the Cluster (Doc ID 2703311.1)

First, let’s look at the whole Exadata. Below are all the nodes with its respective versions:

Note: The details in this post are from a live production system, with the names, IPs, domains, etc., changed.

[root@exa01db01 pythian]# ./exa-versions.sh

        Cluster is a X7-2 Elastic Rack HC 10TB

           -- Database Servers

       exa01db01           exa01db02           exa01db03
------------------------------------------------------------
 19.2.12.0.0.200317  19.2.12.0.0.200317  19.2.12.0.0.200317
------------------------------------------------------------


           -- Cells

     exa01celadm01       exa01celadm02       exa01celadm03       exa01celadm04       exa01celadm05       exa01celadm06
------------------------------------------------------------------------------------------------------------------------
 19.2.12.0.0.200317  19.2.12.0.0.200317  19.2.12.0.0.200317  19.2.12.0.0.200317  19.2.12.0.0.200317  19.2.12.0.0.200317
------------------------------------------------------------------------------------------------------------------------


         -- Infiniband Switches

      exa01sw-iba01       exa01sw-ibb01       exa01sw-ibs01
------------------------------------------------------------
       2.2.12-2            2.2.12-2            2.2.12-2
------------------------------------------------------------

Patching prerequisites

As usual, before patching any Exadata component, it’s important to review the health of the entire cluster, thus we’ve executed the “exachk” utility which reported no issues on the IB switches:

Exachk IB switch checks

The “exachk” utility checks for the NTP configurations, as highlighted above, to ensure the NTP configuration is not the default. It doesn’t check either current timestamps, or if NTP is running properly:

IB switch check details

Another mandatory step before patching any Exadata component is to execute patchmgr’s pre-upgrade checks (“precheck”). This precheck runs a series of tests to make sure the device is ready for the upgrade. Below are the details of these checks which reported no issues on the IB switches:

[root@exa01db01 patch_19.2.12.0.0.200317]# ./patchmgr -ibswitches ~/ib_group -ibswitch_precheck -upgrade

2020-08-15 15:45:31 -0500 1 of 1 :Working: DO: Initiate pre-upgrade validation check on InfiniBand switch(es).
 ----- InfiniBand switch update process started 2020-08-15 15:45:32 -0500 -----
[NOTE     ] Log file at /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/upgradeIBSwitch.log

[INFO     ] List of InfiniBand switches for upgrade: ( exa01sw-iba01 exa01sw-ibb01 exa01sw-ibs01 )
[SUCCESS  ] Verifying Network connectivity to exa01sw-iba01
[SUCCESS  ] Verifying Network connectivity to exa01sw-ibb01
[SUCCESS  ] Verifying Network connectivity to exa01sw-ibs01
[SUCCESS  ] Validating verify-topology output
[INFO     ] Master Subnet Manager is set to "exa01sw-ibs01" in all Switches

[INFO     ] ---------- Starting with InfiniBand Switch exa01sw-ibs01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
     To downgrade to other versions:
     - Manually download the InfiniBand switch firmware package to the patch directory
     - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
     - Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibs01.
[INFO     ] Starting pre-update validation on exa01sw-ibs01
[SUCCESS  ] Verifying that /tmp has 150M in exa01sw-ibs01, found 492M
[SUCCESS  ] Verifying that / has 20M in exa01sw-ibs01, found 26M
[SUCCESS  ] NTP daemon is running on exa01sw-ibs01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 15:55:04
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-ibs01
[SUCCESS  ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibs01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on exa01sw-ibs01
[INFO     ] Finished pre-update validation on exa01sw-ibs01
[SUCCESS  ] Pre-update validation on exa01sw-ibs01
[SUCCESS  ] Prereq check on exa01sw-ibs01

[INFO     ] ---------- Starting with InfiniBand Switch exa01sw-iba01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
     To downgrade to other versions:
     - Manually download the InfiniBand switch firmware package to the patch directory
     - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
     - Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-iba01.
[INFO     ] Starting pre-update validation on exa01sw-iba01
[SUCCESS  ] Verifying that /tmp has 150M in exa01sw-iba01, found 492M
[SUCCESS  ] Verifying that / has 20M in exa01sw-iba01, found 26M
[SUCCESS  ] NTP daemon is running on exa01sw-iba01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 17:34:39
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-iba01
[SUCCESS  ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-iba01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on exa01sw-iba01
[INFO     ] Finished pre-update validation on exa01sw-iba01
[SUCCESS  ] Pre-update validation on exa01sw-iba01
[SUCCESS  ] Prereq check on exa01sw-iba01

[INFO     ] ---------- Starting with InfiniBand Switch exa01sw-ibb01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
     To downgrade to other versions:
     - Manually download the InfiniBand switch firmware package to the patch directory
     - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
     - Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibb01.
[INFO     ] Starting pre-update validation on exa01sw-ibb01
[SUCCESS  ] Verifying that /tmp has 150M in exa01sw-ibb01, found 492M
[SUCCESS  ] Verifying that / has 20M in exa01sw-ibb01, found 26M
[SUCCESS  ] NTP daemon is running on exa01sw-ibb01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:24:46
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-ibb01
[SUCCESS  ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibb01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on exa01sw-ibb01
[INFO     ] Finished pre-update validation on exa01sw-ibb01
[SUCCESS  ] Pre-update validation on exa01sw-ibb01
[SUCCESS  ] Prereq check on exa01sw-ibb01
[SUCCESS  ] Overall status

 ----- InfiniBand switch update process ended 2020-08-15 15:47:41 -0500 -----
2020-08-15 15:47:41 -0500 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).
2020-08-15 15:47:41 -0500        :SUCCESS: Completed run of command: ./patchmgr -ibswitches /root/ib_group -ibswitch_precheck -upgrade
2020-08-15 15:47:41 -0500        :INFO   : upgrade attempted on nodes in file /root/ib_group: [exa01sw-iba01 exa01sw-ibb01 exa01sw-ibs01]
2020-08-15 15:47:41 -0500        :INFO   : For details, check the following files in /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317:
2020-08-15 15:47:41 -0500        :INFO   :  - upgradeIBSwitch.log
2020-08-15 15:47:41 -0500        :INFO   :  - upgradeIBSwitch.trc
2020-08-15 15:47:41 -0500        :INFO   :  - patchmgr.stdout
2020-08-15 15:47:41 -0500        :INFO   :  - patchmgr.stderr
2020-08-15 15:47:41 -0500        :INFO   :  - patchmgr.log
2020-08-15 15:47:41 -0500        :INFO   :  - patchmgr.trc
2020-08-15 15:47:41 -0500        :INFO   : Exit status:0
2020-08-15 15:47:41 -0500        :INFO   : Exiting.

Patching the switches / full outage

After validating all prerequisites, “patchmgr” successfully applied the patch to the switches in “standby” state:

[root@exa01db01 patch_19.2.12.0.0.200317]# ./patchmgr -ibswitches ~/ib_group -upgrade 

 ----- InfiniBand switch update process started 2020-08-15 15:49:39 -0500 -----
[NOTE     ] Log file at /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/upgradeIBSwitch.log

[INFO     ] List of InfiniBand switches for upgrade: ( exa01sw-iba01 exa01sw-ibb01 exa01sw-ibs01 )
[SUCCESS  ] Verifying Network connectivity to exa01sw-iba01
[SUCCESS  ] Verifying Network connectivity to exa01sw-ibb01
[SUCCESS  ] Verifying Network connectivity to exa01sw-ibs01
[SUCCESS  ] Validating verify-topology output
[INFO     ] Proceeding with upgrade of InfiniBand switches to version 2.2.14_1
[INFO     ] Master Subnet Manager is set to "exa01sw-ibs01" in all Switches

[INFO     ] ---------- Starting with InfiniBand Switch exa01sw-ibs01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
     To downgrade to other versions:
     - Manually download the InfiniBand switch firmware package to the patch directory
     - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
     - Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibs01.
[INFO     ] Starting pre-update validation on exa01sw-ibs01
[SUCCESS  ] Verifying that /tmp has 150M in exa01sw-ibs01, found 492M
[SUCCESS  ] Verifying that / has 20M in exa01sw-ibs01, found 26M
[SUCCESS  ] Service opensmd is running on InfiniBand Switch exa01sw-ibs01
[SUCCESS  ] NTP daemon is running on exa01sw-ibs01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 15:59:12
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-ibs01
[SUCCESS  ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibs01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on exa01sw-ibs01
[INFO     ] Finished pre-update validation on exa01sw-ibs01
[SUCCESS  ] Pre-update validation on exa01sw-ibs01
[INFO     ] Package will be downloaded at firmware update time via scp
[SUCCESS  ] Execute plugin check for Patching on exa01sw-ibs01
[INFO     ] Starting upgrade on exa01sw-ibs01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO     ] Rebooting exa01sw-ibs01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS  ] Load firmware 2.2.14_1 onto exa01sw-ibs01
[SUCCESS  ] Verify that /conf/configvalid is set to 1 on exa01sw-ibs01
[INFO     ] Set SMPriority to 8 on exa01sw-ibs01
[INFO     ] Starting post-update validation on exa01sw-ibs01
[SUCCESS  ] Service opensmd is running on InfiniBand Switch exa01sw-ibs01
[SUCCESS  ] NTP daemon is running on exa01sw-ibs01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:07:02
[INFO     ] /conf/configvalid is 1
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-ibs01
[SUCCESS  ] Execute plugin check for Post Patch on exa01sw-ibs01
[INFO     ] Finished post-update validation on exa01sw-ibs01
[SUCCESS  ] Post-update validation on exa01sw-ibs01
[SUCCESS  ] Update InfiniBand switch exa01sw-ibs01 to 2.2.14_1

[INFO     ] ---------- Starting with InfiniBand Switch exa01sw-ibb01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
     To downgrade to other versions:
     - Manually download the InfiniBand switch firmware package to the patch directory
     - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
     - Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibb01.
[INFO     ] Starting pre-update validation on exa01sw-ibb01
[SUCCESS  ] Verifying that /tmp has 150M in exa01sw-ibb01, found 492M
[SUCCESS  ] Verifying that / has 20M in exa01sw-ibb01, found 26M
[SUCCESS  ] Service opensmd is running on InfiniBand Switch exa01sw-ibb01
[SUCCESS  ] NTP daemon is running on exa01sw-ibb01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:45:01
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-ibb01
[SUCCESS  ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibb01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on exa01sw-ibb01
[INFO     ] Finished pre-update validation on exa01sw-ibb01
[SUCCESS  ] Pre-update validation on exa01sw-ibb01
[INFO     ] Package will be downloaded at firmware update time via scp
[SUCCESS  ] Execute plugin check for Patching on exa01sw-ibb01
[INFO     ] Starting upgrade on exa01sw-ibb01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO     ] Rebooting exa01sw-ibb01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
[SUCCESS  ] Load firmware 2.2.14_1 onto exa01sw-ibb01
[SUCCESS  ] Verify that /conf/configvalid is set to 1 on exa01sw-ibb01
[INFO     ] Set SMPriority to 5 on exa01sw-ibb01
[INFO     ] Starting post-update validation on exa01sw-ibb01
[SUCCESS  ] Service opensmd is running on InfiniBand Switch exa01sw-ibb01
[SUCCESS  ] NTP daemon is running on exa01sw-ibb01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:24:14
[INFO     ] /conf/configvalid is 1
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-ibb01
[SUCCESS  ] Execute plugin check for Post Patch on exa01sw-ibb01
[INFO     ] Finished post-update validation on exa01sw-ibb01
[SUCCESS  ] Post-update validation on exa01sw-ibb01
[SUCCESS  ] Update InfiniBand switch exa01sw-ibb01 to 2.2.14_1

After successfully patching the two “standby” switches, “patchmgr” proceeded to patch the “master” switch (note that “patchmgr” doesn’t check for the actual statuses of the IB switches after the patching). When it rebooted, all nodes lost connectivity, as there were no surviving switches in the “standby” state to manage the connections. This abruptly interrupted “patchmgr”:

[INFO     ] ---------- Starting with InfiniBand Switch exa01sw-iba01
[WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
     To downgrade to other versions:
     - Manually download the InfiniBand switch firmware package to the patch directory
     - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
     - Run patchmgr command to initiate downgrade.
[SUCCESS  ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-iba01.
[INFO     ] Starting pre-update validation on exa01sw-iba01
[SUCCESS  ] Verifying that /tmp has 150M in exa01sw-iba01, found 492M
[SUCCESS  ] Verifying that / has 20M in exa01sw-iba01, found 26M
[SUCCESS  ] Service opensmd is running on InfiniBand Switch exa01sw-iba01
[SUCCESS  ] NTP daemon is running on exa01sw-iba01.
[INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 18:12:35
[INFO     ] Validating the current firmware on the InfiniBand Switch
[SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-iba01
[SUCCESS  ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-iba01 through getHostByName
[SUCCESS  ] Execute plugin check for Patch Check Prereq on exa01sw-iba01
[INFO     ] Finished pre-update validation on exa01sw-iba01
[SUCCESS  ] Pre-update validation on exa01sw-iba01
[INFO     ] Package will be downloaded at firmware update time via scp
[SUCCESS  ] Execute plugin check for Patching on exa01sw-iba01
[INFO     ] Starting upgrade on exa01sw-iba01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
[INFO     ] Rebooting exa01sw-iba01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH

As seen above, the last operation executed by “patchmgr” was the reboot of exa01sw-iba01. As previously explained, since there were no nodes in the “standby” state, we lost connectivity to all the nodes and had to log into the DB nodes using the ILOM:

[root@test01 ~]$ ssh root@exa01dbadm01-ilom.example.com
Password:

Oracle(R) Integrated Lights Out Manager

Version 4.0.4.52 r133103

Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: exa01dbadm01-ilom

-> start /SP/Console
Are you sure you want to start /SP/console (y/n)? y

Serial console started.  To stop, type ESC (


exa01db01 login: root
Password:
Last login: Sat Aug 15 16:26:07 CDT 2020 from exa01db01.example.com on ssh
Last login: Sat Aug 15 16:31:59 on ttyS0
[root@exa01db01 ~]#

Troubleshooting

All IB switches returned to “standby” state when the clocks reached the pre-reboot timepoint. Given the complexity of this issue, we engaged Oracle to assist in the analysis.

We lost all connectivity, as the three switches were in “discover” state. By the time we logged back into the system, ibs01 had already become the “master.” The other two switches remained in the “boot in progress” state for some time:

[root@exa01sw-ibs01 ~]# ibswitches
Switch : 0x0010e0dc11fba0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.ZZ" enhanced port 0 lid 3 lmc 0
Switch : 0x0010e0dc1ceda0a0 ports 36 "BOOT IN PROGRESS" enhanced port 0 lid 0 lmc 0
Switch : 0x0010e0dc0f6ea0a0 ports 36 "BOOT IN PROGRESS" enhanced port 0 lid 0 lmc 0

[root@exa01sw-ibs01 ~]# getmaster
Local SM enabled and running, state MASTER
Last change in Master SubnetManager status detected at: Sat Aug 15 16:25:50 CDT 2020
Master SubnetManager on sm lid 3 sm guid 0x10e0dc11fba0a0 : SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.ZZ
Master SubnetManager Activity Count: 11637 Priority: 14

The Oracle engineer identified the root cause of this issue as the incorrect configuration of NTP. This configuration error led to a clock difference on the switches, fixed by Linux after the reboot, thus moving the clock backward. This caused the switch to enter “discover” state and become unusable. He pinpointed the issue by detecting a time drift on the /var/log/messages file of the IB switch:

Aug 15 18:16:50 exa01sw-iba01 envd[2532]: Connector 13B present
Aug 15 16:29:05 exa01sw-iba01 envd[2532]: Connector 14B present
The switches reverted to “standby” state when the system clock reached the same time it had before the reboot.

Solution

To avoid running into an outage, it’s preferable to patch switches one at a time and execute a series of checks on each switch after patching, before proceeding.

NOTE: Only perform changes using the “Ilom-admin” (ILOM shell) account. Do not perform any changes on the IB switches with the “root” (regular shell) account.

Configure and enable NTP

  1. Carefully review the NTP settings on each switch:
    Hostname: exa01sw-iba01.example.com
    
    -> show /SP/clock
    
     /SP/clock
        Targets:
    
        Properties:
            datetime = Mon Aug 17 08:58:58 2020
            timezone = CDT (America/Chicago)
            uptime = 0 days, 23:06:02
            usentpserver = disabled
    
        Commands:
            cd
            set
            show
    
    -> show /SP/clients/ntp/server/1
    
     /SP/clients/ntp/server/1
        Targets:
    
        Properties:
            address = 0.0.0.0
    
        Commands:
            cd
            set
            show
    
    -> show /SP/clients/ntp/server/2
    
     /SP/clients/ntp/server/2
        Targets:
    
        Properties:
            address = 0.0.0.0
    
        Commands:
            cd
            set
            show
    
    -> exit
    [root@exa01sw-iba01 ~]# ls -l /etc/ntp.conf
    lrwxrwxrwx 1 root root 21 2015-09-10 11:37 /etc/ntp.conf -> /config/conf/ntp.conf
    [root@exa01sw-iba01 ~]# ls -l /etc/default/ntpdate
    lrwxrwxrwx 1 root root 20 2016-03-31 07:39 /etc/default/ntpdate -> /config/conf/ntpdate
    [root@exa01sw-iba01 ~]# cat /config/conf/ntp.conf
    ## DO NOT EDIT THIS FILE ##
    #server none
    #server none
    driftfile /conf/ntp-drift
    pidfile /var/run/ntpd.pid
    [root@exa01sw-iba01 ~]# cat /config/conf/ntpdate
    ## DO NOT EDIT THIS FILE ##
    # servers to check.   (Separate multiple servers with spaces.)
    NTPSERVERS="XXX.YYY.ZZZ.AAA XXX.BBB.CCC.AAA"
    #
    # additional options for ntpdate
    NTPOPTIONS="-u"

    The “root” account (regular shell) changed the NTP’s settings only at the OS level. During the reboot Linux invoked “ntpdate” and adjusted the system’s clock (backward) using the OS configurations, causing a significant drift that placed the switches in “discover” state.

  2. Change to the “ilom-admin” (ILOM shell) account and adjust the configurations, before enabling NTP.
    Hostname: exa01sw-ibb01.example.com
    
    -> show /SP/clients/ntp/server/1
    
     /SP/clients/ntp/server/1
        Targets:
    
        Properties:
            address = 0.0.0.0
    
        Commands:
            cd
            set
            show
    
    -> show /SP/clients/ntp/server/2
    
     /SP/clients/ntp/server/2
        Targets:
    
        Properties:
            address = 0.0.0.0
    
        Commands:
            cd
            set
            show
    
    -> show /SP/clock
    
     /SP/clock
        Targets:
    
        Properties:
            datetime = Sat Aug 22 08:25:29 2020
            timezone = CDT (US/Central)
            uptime = 89 days, 13:42:39
            usentpserver = disabled
    
        Commands:
            cd
            set
            show
    
    ->  set /sp/clock/ usentpserver=disabled
    Set 'usentpserver' to 'disabled'
    
    -> set /sp/clients/ntp/server/1/ address=XXX.YYY.ZZZ.AAA
    Set 'address' to 'XXX.YYY.ZZZ.AAA'
    
    -> set /sp/clients/ntp/server/2/ address=XXX.BBB.CCC.AAA
    Set 'address' to 'XXX.BBB.CCC.AAA'
    
    -> set /sp/clock/ usentpserver=enabled
    Set 'usentpserver' to 'enabled'
    
    -> show /SP/clock
    
     /SP/clock
        Targets:
    
        Properties:
            datetime = Sat Aug 22 08:31:05 2020
            timezone = CDT (US/Central)
            uptime = 89 days, 13:45:48
            usentpserver = enabled
    
        Commands:
            cd
            set
            show
    
    -> show /SP/clients/ntp/server/1
    
     /SP/clients/ntp/server/1
        Targets:
    
        Properties:
            address = XXX.YYY.ZZZ.AAA
    
        Commands:
            cd
            set
            show
    
    -> show /SP/clients/ntp/server/2
    
     /SP/clients/ntp/server/2
        Targets:
    
        Properties:
            address = XXX.BBB.CCC.AAA
    
        Commands:
            cd
            set
            show
    
    -> exit
    [root@exa01sw-ibb01 ~]# date
    Sat Aug 22 08:31:13 CDT 2020
  3. After enabling NTP make sure the switch is in the “standby” state before proceeding to the next switch. Do not reboot the “master” switch If there are no remaining switches in “standby” state. Check the current state with the “getmaster” command in the regular shell of the IB switch:
    [root@exa01sw-ibb01 ~]# getmaster
    Local SM enabled and running, state STANDBY
    Last change in Master SubnetManager status detected at: Sun May 24 08:20:16 CDT 2020
    Master SubnetManager on sm lid 1 sm guid 0x10e0dc218da0a0 : SUN DCS 36P QDR exa01sw-iba01 
    Master SubnetManager Activity Count: 57235558 Priority: 14

Adjusting the Subnet Manager (SM) if the clock moved forward

In our scenario, the clock moved forward causing multiple master IB switches on the fabric. This led to connectivity issues on one node. Rotating the SM to a switch that has the correct timestamp will fix this:

  1. Check the state of all switches in the cluster:
    [root@exa01db01 pythian]# dcli -l root -g ~/ibs_group: getmaster | grep state
    exa01sw-iba01: Local SM enabled and running, state STANDBY
    exa01sw-ibb01: Local SM enabled and running, state STANDBY
    exa01sw-ibs01: Local SM enabled and running, state MASTER
  2. Disable SM on the current “master” switch:
    [root@exa01db01 pythian]# ssh exa01sw-ibs01
    You are now logged in to the root shell.
    It is recommended to use ILOM shell instead of root shell.
    All usage should be restricted to documented commands and documented
    config files.
    To view the list of documented commands, use "help" at linux prompt.
    [root@exa01sw-ibs01 ~]# disablesm
    
    Stopping partcfgchk            [ OK ]
    Stopping partitiond-daemon     [ OK ]
    Stopping IB Subnet Manager..-. [ OK ]
    
    [root@exa01sw-ibs01 ~]# getmaster
    Local SM not enabled
    
  3. Re-enable SM:
    [root@exa01sw-ibs01 ~]# enablesm
    
    Starting IB Subnet Manager. [ OK ]
    Starting partitiond-daemon  [ OK ]
    Starting partcfgchk         [ OK ]
    
    [root@exa01sw-ibs01 ~]# getmaster
    Local SM enabled and running, state STANDBY
    Last change in Master SubnetManager status detected at: Sun May 24 08:20:16 CDT 2020
    Master SubnetManager on sm lid 1 sm guid 0x10e0dc218da0a0 : SUN DCS 36P QDR exa01sw-iba01 XX.YY.ZZ.ZZ
    Master SubnetManager Activity Count: 615361 Priority: 14
  4. Check the state of all switches in the cluster: (When SM was disabled on ibs01, iba01 picked up as “master”)
    [root@exa01db01 pythian]# dcli -l root -g ~/ibs_group: getmaster | grep state
    exa01sw-iba01: Local SM enabled and running, state MASTER
    exa01sw-ibb01: Local SM enabled and running, state STANDBY
    exa01sw-ibs01: Local SM enabled and running, state STANDBY

Patching the switches with “patchmgr”

  1. It’s preferable to patch one switch at a time since “patchmgr” doesn’t check the state of the IB switch. Specify a single switch in the group’s file as follows:
    [root@exa01sw-iba01 ~]# cat ~/ib_group
    exa01sw-ibs01
    
    [root@exa01sw-iba01 ~]# cd /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/
    [root@exa01sw-iba01 patch_19.2.12.0.0.200317]# ./patchmgr -ibswitches ~/ib_group -upgrade
    2020-08-15 15:49:38 -0500 1 of 1 :Working: DO: Initiate upgrade of InfiniBand switches to 2.2.14-1. Expect up to 40 minutes for each switch
    
     ----- InfiniBand switch update process started 2020-08-15 15:49:39 -0500 -----
    [NOTE     ] Log file at /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/upgradeIBSwitch.log
    
    [INFO     ] List of InfiniBand switches for upgrade: ( exa01sw-ibs01 )
    [SUCCESS  ] Verifying Network connectivity to exa01sw-ibs01
    [SUCCESS  ] Validating verify-topology output
    [INFO     ] Proceeding with upgrade of InfiniBand switches to version 2.2.14_1
    [INFO     ] Master Subnet Manager is set to "exa01sw-iba01" in all Switches
    
    [INFO     ] ---------- Starting with InfiniBand Switch exa01sw-ibs01
    [WARNING  ] Infiniband switch meets minimal version requirements, but downgrade is only available to 2.2.13-2 with the current package.
         To downgrade to other versions:
         - Manually download the InfiniBand switch firmware package to the patch directory
         - Set export variable "EXADATA_IMAGE_IBSWITCH_DOWNGRADE_VERSION" to the appropriate version
         - Run patchmgr command to initiate downgrade.
    [SUCCESS  ] Verify SSH access to the patchmgr host exa01db01.example.com from the InfiniBand Switch exa01sw-ibs01.
    [INFO     ] Starting pre-update validation on exa01sw-ibs01
    [SUCCESS  ] Verifying that /tmp has 150M in exa01sw-ibs01, found 492M
    [SUCCESS  ] Verifying that / has 20M in exa01sw-ibs01, found 26M
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch exa01sw-ibs01
    [SUCCESS  ] NTP daemon is running on exa01sw-ibs01.
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 15:59:12
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-ibs01
    [SUCCESS  ] Verifying that the patchmgr host exa01db01.example.com is recognized on the InfiniBand Switch exa01sw-ibs01 through getHostByName
    [SUCCESS  ] Execute plugin check for Patch Check Prereq on exa01sw-ibs01
    [INFO     ] Finished pre-update validation on exa01sw-ibs01
    [SUCCESS  ] Pre-update validation on exa01sw-ibs01
    [INFO     ] Package will be downloaded at firmware update time via scp
    [SUCCESS  ] Execute plugin check for Patching on exa01sw-ibs01
    [INFO     ] Starting upgrade on exa01sw-ibs01 to 2.2.14_1. Please give upto 15 mins for the process to complete. DO NOT INTERRUPT or HIT CTRL+C during the upgrade
    [INFO     ] Rebooting exa01sw-ibs01 to complete the firmware update. Wait for 15 minutes before continuing. DO NOT MANUALLY REBOOT THE INFINIBAND SWITCH
    [SUCCESS  ] Load firmware 2.2.14_1 onto exa01sw-ibs01
    [SUCCESS  ] Verify that /conf/configvalid is set to 1 on exa01sw-ibs01
    [INFO     ] Set SMPriority to 8 on exa01sw-ibs01
    [INFO     ] Starting post-update validation on exa01sw-ibs01
    [SUCCESS  ] Service opensmd is running on InfiniBand Switch exa01sw-ibs01
    [SUCCESS  ] NTP daemon is running on exa01sw-ibs01.
    [INFO     ] Manually validate the following entries Date:(YYYY-MM-DD) 2020-08-15 Time:(HH:MM:SS) 16:07:02
    [INFO     ] /conf/configvalid is 1
    [INFO     ] Validating the current firmware on the InfiniBand Switch
    [SUCCESS  ] Firmware verification on InfiniBand switch exa01sw-ibs01
    [SUCCESS  ] Execute plugin check for Post Patch on exa01sw-ibs01
    [INFO     ] Finished post-update validation on exa01sw-ibs01
    [SUCCESS  ] Post-update validation on exa01sw-ibs01
    [SUCCESS  ] Update InfiniBand switch exa01sw-ibs01 to 2.2.14_1       
           
     ----- InfiniBand switch update process ended 2020-08-15 16:07:41 -0500 -----
    2020-08-15 16:07:41 -0500 1 of 1 :SUCCESS: DONE: Initiate pre-upgrade validation check on InfiniBand switch(es).
    2020-08-15 16:07:41 -0500        :SUCCESS: Completed run of command: ./patchmgr -ibswitches /root/ib_group -ibswitch_precheck -upgrade
    2020-08-15 16:07:41 -0500        :INFO   : upgrade attempted on nodes in file /root/ib_group: [exa01sw-ibs01]
    2020-08-15 16:07:41 -0500        :INFO   : For details, check the following files in /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317:
    2020-08-15 16:07:41 -0500        :INFO   :  - upgradeIBSwitch.log
    2020-08-15 16:07:41 -0500        :INFO   :  - upgradeIBSwitch.trc
    2020-08-15 16:07:41 -0500        :INFO   :  - patchmgr.stdout
    2020-08-15 16:07:41 -0500        :INFO   :  - patchmgr.stderr
    2020-08-15 16:07:41 -0500        :INFO   :  - patchmgr.log
    2020-08-15 16:07:41 -0500        :INFO   :  - patchmgr.trc
    2020-08-15 16:07:41 -0500        :INFO   : Exit status:0
    2020-08-15 16:07:41 -0500        :INFO   : Exiting.
    
  2. Perform a thorough check on each patched switch before moving on to the next:
    # Make sure that the switch is in the "STANDBY" state after the patch
    [root@exa01sw-iba01 ~]# getmaster
    Local SM enabled and running, state STANDBY
    Last change in Master SubnetManager status detected at: Sat Aug 15 17:34:55 CDT 2020
    Master SubnetManager on sm lid 3 sm guid 0x10e0dc11fba0a0 : SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.BB
    Master SubnetManager Activity Count: 615361 Priority: 14
    
    # Make sure that all IB switches are listed with the "ibswitches" command
    [root@exa01sw-iba01 ~]# ibswitches
    Switch  : 0x0010e0dc0f6ea0a0 ports 36 "SUN DCS 36P QDR exa01sw-iba01 10.XX.YY.ZZ" enhanced port 0 lid 2 lmc 0
    Switch  : 0x0010e0dc11fba0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.AA" enhanced port 0 lid 3 lmc 0
    Switch  : 0x0010e0dc1ceda0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibb01 10.XX.YY.BB" enhanced port 0 lid 19 lmc 0
    
    # Check switch version
    [root@exa01sw-iba01 ~]# version
    SUN DCS 36p version: 2.2.14-1
    Build time: Aug 14 2019 08:48:04
    SP board info:
    Manufacturing Date: N/A
    Serial Number: "XXXXXXXX"
    Hardware Revision: 0x0100
    Firmware Revision: 0x0000
    BIOS version: NUP1R918
    BIOS date: 01/19/2016
    
    # Check the date on all the switches
    [root@exa01sw-iba01 ~]# cat ~/ibs_group
    exa01sw-iba01
    exa01sw-ibb01
    exa01sw-ibs01
    [root@exa01sw-iba01 ~]# dcli -g ~/ibs_group -l root date

Patching the switches manually

Since “patchmgr” couldn’t patch the final IB switch, Oracle recommended patching it manually. The following document describes the steps required to do so: https://docs.oracle.com/cd/E76424_01/pdf/E76431.pdf (Section: Upgrade the Switch Firmware (CLI).

  1. First log into the switch, check its version, SM status, and verify the firmware:
    [root@exa01sw-iba01 ~]# getmaster
    Local SM enabled and running, state STANDBY
    Last change in Master SubnetManager status detected at: Sat Aug 15 17:34:55 CDT 2020
    Master SubnetManager on sm lid 3 sm guid 0x10e0dc11fba0a0 : SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.ZZ
    Master SubnetManager Activity Count: 615361 Priority: 14
    [root@exa01sw-iba01 ~]# fwverify -s
    
    Checking all present packages:
    .............................................................................................................................................................................................................................................. OK
    
    Checking if any packages are missing:
    ............................................................................................................................................................................................................................................. OK
    
    Checking FW Coreswitch:
      FW Version: 7.4.3002 OK
      PSID: SUN_NM2-36p_006 OK
      Verifying image integrity OK
    
  2. Download the  required “pkg” file for your target version and compare the checksum with that provided by Oracle:
    [root@exa01db01 pythian]# cd /u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317 
    [root@exa01db01 patch_19.2.12.0.0.200317]# md5sum sundcs_36p_repository_2.2.14_1.pkg > pkg.md5 
    [root@exa01db01 patch_19.2.12.0.0.200317]# cat pkg.md5 
    ce05e4e207b9718b01847634dbbd9bc4 sundcs_36p_repository_2.2.14_1.pkg
  3. Log into the switch, check its version, then start the upgrade using the ILOM (ilom-admin):
    [root@exa01db01 patch_19.2.12.0.0.200317]# ssh exa01sw-iba01
    You are now logged in to the root shell.
    It is recommended to use ILOM shell instead of root shell.
    All usage should be restricted to documented commands and documented
    config files.
    To view the list of documented commands, use "help" at linux prompt.
    [root@exa01sw-iba01 ~]# version
    SUN DCS 36p version: 2.2.12-2
    Build time: Oct 29 2018 08:35:27
    SP board info:
    Manufacturing Date: N/A
    Serial Number: "XXXXXXXX"
    Hardware Revision: 0x0100
    Firmware Revision: 0x0000
    BIOS version: NUP1R918
    BIOS date: 01/19/2016
    [root@exa01sw-iba01 ~]# sudo su - ilom-admin
    
    Oracle(R) Integrated Lights Out Manager
    
    Version 2.2.12-2 ILOM 3.2.11 r127844
    
    Copyright (c) 2018, Oracle and/or its affiliates. All rights reserved.
    
    Warning: HTTPS certificate is set to factory default.
    
    Hostname: exa01sw-iba01.example.com
    
    -> load -source scp://root:XXXXXXX@10.XX.YY.ZZ//u01/patches/exadata_patches/IB_PATCHING/patch_19.2.12.0.0.200317/sundcs_36p_repository_2.2.14_1.pkg
    Downloading firmware image. This will take a few minutes.
    SUN DCS 36p version: 2.2.12-2
    Build time: Oct 29 2018 08:35:27
    SP board info:
    Manufacturing Date: N/A
    Serial Number: "XXXXXXXX"
    Hardware Revision: 0x0100
    Firmware Revision: 0x0000
    BIOS version: NUP1R918
    BIOS date: 01/19/2016
    FROM_VERSION: 2.2.12-2
    TO_VERSION: 2.2.14-1
    
    NOTE: Firmware upgrade will upgrade the SUN DCS 36p firmware.
          ILOM will enter a special mode to load new firmware. No
          other tasks should be performed in ILOM until the firmware
          upgrade is complete.
    
    
    
    
    Are you sure you want to load the specified file (y/n)? y
    Setting up environment for firmware upgrade. This will take a few minutes.
    
    
    Subnet manager has been temporarily disabled on this switch so that
    firmware upgrade can go on. After the upgrade is done, please make
    sure that the SM is running again, using commands sminfo and enablesm.
    
    Starting SUN DCS 36p FW update
    
    ==========================
    Performing operation: I4 A
    ==========================
    I4 A: I4 is already at the given version.
    
    =========================================
    Performing operation: SUN DCS 36p firmware update
    =========================================
    SUN DCS 36p Kontron module fw upgrade from 2.2.12-2 to 2.2.14-1:
    Please reboot the system to enable firmware update of Kontron module. The download of the Kontron firmware image happens during reboot.
    
    After system reboot, Kontron FW update progress can be monitored in browser using URL [http://system] OR at OS command line prompt by using command [telnet system 1234] where system is the hostname or IP address of SUN DCS 36P or GW.
    
    
    Firmware update is complete.
    
    -> reset /SP
    Are you sure you want to reset /SP (y/n)? y
    Performing reset on /SP
    
    
    Broadcast message from root@exa01sw-iba01.example.com
            (unknown) at 9:50 ...
    
    The system is going down for reboot NOW!
    
  4. Check the status of the switch, the SM, and its version to make sure the upgrade was successful:
    [root@exa01sw-iba01 ~]# version
    SUN DCS 36p version: 2.2.14-1
    Build time: Aug 14 2019 08:48:04
    SP board info:
    Manufacturing Date: N/A
    Serial Number: "XXXXXXXX"
    Hardware Revision: 0x0100
    Firmware Revision: 0x0000
    BIOS version: NUP1R918
    BIOS date: 01/19/2016
    
    [root@exa01sw-iba01 ~]# sminfo
    sminfo: sm lid 3 sm guid 0x10e0dc11fba0a0, activity count 644470 priority 14 state 3 SMINFO_MASTER
    
    [root@exa01sw-iba01 ~]# ibswitches
    Switch  : 0x0010e0dc0f6ea0a0 ports 36 "SUN DCS 36P QDR exa01sw-iba01 10.XX.YY.AA" enhanced port 0 lid 2 lmc 0
    Switch  : 0x0010e0dc11fba0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.BB" enhanced port 0 lid 3 lmc 0
    Switch  : 0x0010e0dc1ceda0a0 ports 36 "SUN DCS 36P QDR exa01sw-ibb01 10.XX.YY.CC" enhanced port 0 lid 19 lmc 0
    
    [root@exa01sw-iba01 ~]# getmaster
    Local SM enabled and running, state STANDBY
    Last change in Master SubnetManager status detected at: Sat Aug 15 17:34:55 CDT 2020
    Master SubnetManager on sm lid 3 sm guid 0x10e0dc11fba0a0 : SUN DCS 36P QDR exa01sw-ibs01 10.XX.YY.CC
    Master SubnetManager Activity Count: 615361 Priority: 14
    

NOTE: Do not proceed with any other activities until the switch is in the “standby” state.

Check the status of the cluster

Finally, use Fred Denis’ amazing “rac-status.sh” script to check the status of the entire cluster:

[root@exa01db01 pythian]# ./rac-status.sh  -a

                Cluster exa01-clu1 is a X7-2 Elastic Rack HC 10TB

        Type      |      Name      |   db01   |   db02   |   db03   |
  ------------------------------------------------------------------
   asm            | asm            |  Online  |  Online  |  Online  |
   cvu            | cvu            |     -    |     -    |  Online  |
   dg             | DATAC1         |  Online  |  Online  |  Online  |
   dg             | RECOC1         |  Online  |  Online  |  Online  |
   network        | net1           |  Online  |  Online  |  Online  |
   ons            | ons            |  Online  |  Online  |  Online  |
   proxy_advm     | proxy_advm     | Offline x| Offline x| Offline x|
   qosmserver     | qosmserver     | Offline x| Offline x| Offline x|
   vip            | db01           |  Online  |     -    |     -    |
   vip            | db02           |     -    |  Online  |     -    |
   vip            | db03           |     -    |     -    |  Online  |
   vip            | scan1          |     -    |  Online  |     -    |
   vip            | scan2          |     -    |     -    |  Online  |
   vip            | scan3          |  Online  |     -    |     -    |
  ------------------------------------------------------------------
    x  : Resource is disabled
       : Has been restarted less than 24 hours ago
       : STATUS and TARGET are different

      Listener    |      Port      |   db01   |   db02   |   db03   |     Type     |
  ---------------------------------------------------------------------------------
   ASMNET1LSNR_ASM| TCP:1525       |  Online  |  Online  |  Online  |   Listener   |
   LISTENER       | TCP:1521       |  Online  |  Online  |  Online  |   Listener   |
   LISTENER_LEAF  |                |     x    |     x    |     x    |   Listener   |
   LISTENER_SCAN1 | TCP:1521       |     -    |  Online  |     -    |     SCAN     |
   LISTENER_SCAN2 | TCP:1521       |     -    |     -    |  Online  |     SCAN     |
   LISTENER_SCAN3 | TCP:1521       |  Online  |     -    |     -    |     SCAN     |
  ---------------------------------------------------------------------------------
    x  : Listener is disabled
       : Has been restarted less than 24 hours ago

         DB       |     Service    |   db01   |   db02   |   db03   |
  ------------------------------------------------------------------
   prim01         | prim01_batch   |     -    |     -    |  Online  |
                  | prim01_ro      |  Online  |  Online  |  Online  |
   stdby          | stdby          |  Online  |  Online  |  Online  |
                  | stdby_ro       |  Online  |  Online  |  Online  |
  ------------------------------------------------------------------
       : Has been restarted less than 24 hours ago
       : STATUS and TARGET are different

         DB       |     Version    |   db01   |   db02   |   db03   |    DB Type   |
  ---------------------------------------------------------------------------------
   prim01         | 18.3.0.0   (1) |   Open   |   Open   |   Open   |    RAC (P)   |
   stdby          | 12.2.0.1   (2) | Readonly | Readonly | Readonly |    RAC (S)   |
  ---------------------------------------------------------------------------------
  ORACLE_HOME references listed in the Version column ("''" means "same as above")

         1 : /u01/app/oracle/product/18.3.0.0/dbhome_4  oracle oinstall
         2 : /u01/app/oracle/product/12.2.0.1/dbhome_1    ''      ''

    x  : Instance is disabled
       : Has been restarted less than 24 hours ago

Conclusion

As demonstrated throughout this post, even though there are a number of checks to try and prevent issues while patching an Exadata component, it’s nearly impossible to cover all bases. Besides using “exachk” and patchmgr’s prechecks, it’s also important to create your own checklist to preventively detect issues not picked up by those utilities, as well as performing sanity checks after a component is patched to ensure it’s working as expected.

If you have any questions, please feel free to send me a comment and I’ll reply as soon as possible.

Related resources

Fred Denis wrote both scripts used in this post. To get the latest version, please, check Fred’s GitHub page: https://github.com/freddenis/oracle-scripts/

Details on each of the scripts used here:

  • rac-status.sh: Displays the current status of the cluster and all its targets (-a).
  • exa-versions.sh: Displays the current version of each component in the Exadata stack (Cell nodes (-c), DB nodes (-d), IB switches (-i)).
email

Interested in working with Fernando? Schedule a tech call.

Senior Oracle Database Consultant

2 Comments. Leave new

Fantastic post with great detail and examples – thanks Fernando

Reply
Fernando Salgado
September 20, 2020 6:57 pm

Glad you liked it, Neil! Keep tuned for more! :)

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *