How to install a brand new Exadata X5 (Part 1)

When you buy an Exadata machine, you can ask Oracle to proceed with the installation or let some brilliant DBAs like the one at Pythian to do it :)

I will then present in this blog how to install a brand new Exadata X5; this procedure comes from the real life as it has been applied on three X5-2 half rack that are now on production.

As the procedure is quite long, I have split it in two distinct parts :

• Part 1 (this one) is more about the settings that are needed to proceed with the installation
• Part 2 is about the installation procedure itself

 

Anonymization

I have obviously anonymized the procedures, a little word here on the conventions I used here :

• I have replaced the name of the Exadata cluster by “mycluster“
• “a_server” is a hostname outside the Exadata I used to reach it (a jump server, an administration server for example)
• “a_user” is my own non privileged user I use to reach the Exadata servers from this “jump server” (I could have wrote “fdenis” instead)
• I show the client specific IP like this one “<IP_of_an_IB_Switch>”
• Sometimes the customer specific IPs appear like that “10.XX.YY.ZZ” (no secret here, 10.x.x.x being the class A private network)
• All the clear text IPs are the Exadata defaults

Please let me know in the comments if I missed one or if something is not clear.

 

What is already configured

When Oracle delivers an Exadata, they take care of the cabling and the first boot. During this first boot, few things are configured :

• The Ethernet Switch
• The InfiniBand Switches (IB Switches)
• The PDUs
• An access to the ILOM of the first database server

 

What is needed to go further

<client>-<cluster>.xml

The DBA needs some customer specific information before heading on to the network configuration and the software installation of the new Exadata machine such as the hostnames, the IPs address of each components, the default gateway, etc…

Since Exadata X2, Oracle is kind enough to provide an easy way to do that with a software named Oracle Exadata Deployment Assistant (OEDA). The client then download OEDA on his laptop, fill all the needed information and then generate an XML file (named <client>-<cluster>.xml) that contains all the required information for the DBA to proceed with the installation. This file is vital as all the installation procedure is based on it.

 

OEDA for Linux

OEDA is not only used to fill the XML file, it is also used to proceed with the set up (and as OEDA version can move pretty quickly, it is NOT provided with the Exadata machine), we then need to upload on the first database server the following components :

• OEDA for Linux
• The <client>-<cluster>.xml file
• The databasemachine.xml file is nice to have as well as we can then use it to make an Exadata Status Script for example

There’re 2 ways described in the official documentation to do that; usually, the client plug an USB key and copy the OEDA zip file and the XML configuration file let’s say in /root/config_files.

 

Exadata set up

What’s in it ?

First, let’s have a look at what’s in this new Exadata ! we can easily know what is in the box by using the ibhosts command from any part of the IB network (here, I did it from an IB switch as there’s not much server reachable at this part of the installation)

Here, we can see that we have 5 cells (5 storage servers) :

• node1 elasticNode 172.16.2.37
• node2 elasticNode 172.16.2.38
• node3 elasticNode 172.16.2.39
• node4 elasticNode 172.16.2.40
• node6 elasticNode 172.16.2.42

And 4 database servers :

• node8 elasticNode 172.16.2.44
• node9 elasticNode 172.16.2.45
• node10 elasticNode 172.16.2.46
• node12 elasticNode 172.16.2.48

As no network configuration has been done yet, all components have the default factory hostnames and IP. Oracle uses the private 172.16 network for the factory IP settings starting with the first database node at 172.16.2.44. Oracle uses the Infiniband port number where the node is plugged (first database node is plugged on the port number 8) + 36 : 36 + 8 = 44 then the default database server node 1 is 172.16.2.44.

 

Network Configuration

Let’s then configure these IP and hostnames with the customer’s requirements in order to be able to access the database and storage servers from the customer network.

/etc/hosts of the IB Switches

There’s not much to configure on the IB Switches, just to check that the IP and the host are declared in the /etc/hosts; add it if it’s not already done :

ILOM console

To start with, we have to connect to the ILOM of the first database server (the IP is accessible from the customer network).

As we don’t need to do much things on the ILOM itself, let’s head on to the first database server using the ILOM console :

Then here press <ENTER> and login as root 

As we saw previously, the first database server is <node8> and a Linux OS is pre-installed

ReclaimSpace

Exadata database servers come with Linux pre-installed but it is also possible to switch to Oracle VM  which is also pre-installed on disk. In order to not waste any disk space, we can reclaim that space and any other unused space by executing the below command on every database server (list of IP for db servers can be taken from the ibhosts command like shown earlier) :

OneCommand

Create a OneCommand directory, copy and unzip OEDA and the XML configuration file provided by the client in it.

Note that it is not allowed to run OEDA from a root (/opt) directory then use /u01 to unzip it. Here is the error you would face if you launch OEDA from /opt :

Invoke OEDA from a non root file system. Current directory /opt/oracle.SupportTools/onecommand/linux-x64 is part of the root file system Non-root file systems with required space are: File System /u01 free space 95485 MB

applyElasticConfig

We can now proceed with the network configuration (the applyElasticConfig.sh shell script is provided by OEDA and is located under /opt/oracle.SupportTools/onecommand/linux-x64)

This will apply all the customer’s specifications to the database and storage servers and all servers will be rebooted at the end of their configuration.

Once the set-up applied, you will be able to connect to each database and storage servers from the customer network using their 10.x.y.z specific IP and their hostnames if they have already been defined in the corporate DNS.

 

An Issue with the Cells IP config

It happened that after the previous step, the cells were not well configured and then the cells configuration is different from the real network configuration shown by the “ifconfig” command. We have to correct that manually to avoid any issue later on specially when applying some patches; here are the error messages you could face if your cells are misconfigured (“CELL-01533: Unable to validate the IP addresses from the cellinit.ora file because the IP addresses may be down or misconfigured” and  “ORA-00700: soft internal error, arguments: [main_6a], [3], [IP addresses in cellinit.ora not operational], [], [], [], [], [], [], [], [], []“).

Here is the output of the ifconfig command on one cell (is pasted here only the information we need for visibility purpose) :

Here is a (bad) output of one cell (is pasted here only the information we need for visibility purpose) :

Note : “/24” is the network mask, it should be set to “/22” in the following step
Who’s right then ? well, ibhosts doesn’t lie :

We then need to update the configuration manually (even the hostname has to be set), restart the services :

And then a quick check of the new config :

Note : this procedure has to be done for every cell — if your cells were misconfigured by the ./applyElasticConfig.sh -cf <CLIENT>-mycluster.xml command only

dbs_group, cell_group and all_group

You have probablyalready  seen many Exadata documentations mentioning files named dbs_group, cell_group and all_group. But where do they come from ? Well, you, the DBA, create them:)

As these files are used on our daily basis administration tasks, it is a good idea to put them in a well known directory.

Here is few commands to create them quickly — I also like to have an ib_group file with the name of my IB Switches, this will be very useful when you will patch the IB Switches.

 

root SSH equivalence

It is better to have a root SSH equivalence to ease all the installation and the administration tasks. We will choose one server (I use to choose DB server node 1 aka “myclusterdb01″) and will deploy its SSH keys to all the other DB, cell nodes and the IB Switches.

First we need to generate the root’s SSH environment (press <ENTER> after each question) :

And we will use the wonderful dcli to deploy the ssh keys to all the database and storage servers from the database node 1:

Note : you will be prompted for the password of each server, don’t mess up, root user is locked after one failed attempt :)

Let’s do the same for the IB Switches :

You are now able to connect from the first DB server to any host without password. You can quickly test it with this command : “dcli -g ~/all_group -l root date“

 

 

All is now ready to jump to the installation part which is detailed in the part 2 of this blog !

 

Want to talk with an expert? Schedule a call with our team to get the conversation started.