I will then present in this blog how to install a brand new Exadata X5; this procedure comes from the real life as it has been applied on three X5-2 half rack that are now on production.
As the procedure is quite long, I have split it in two distinct parts :
- Part 1 (this one) is more about the settings that are needed to proceed with the installation
- Part 2 is about the installation procedure itself
I have obviously anonymized the procedures, a little word here on the conventions I used here :
- I have replaced the name of the Exadata cluster by “mycluster“
- “a_server” is a hostname outside the Exadata I used to reach it (a jump server, an administration server for example)
- “a_user” is my own non privileged user I use to reach the Exadata servers from this “jump server” (I could have wrote “fdenis” instead)
- I show the client specific IP like this one “<IP_of_an_IB_Switch>”
- Sometimes the customer specific IPs appear like that “10.XX.YY.ZZ” (no secret here, 10.x.x.x being the class A private network)
- All the clear text IPs are the Exadata defaults
Please let me know in the comments if I missed one or if something is not clear.
What is already configured
When Oracle delivers an Exadata, they take care of the cabling and the first boot. During this first boot, few things are configured :
- The Ethernet Switch
- The InfiniBand Switches (IB Switches)
- The PDUs
- An access to the ILOM of the first database server
What is needed to go further
The DBA needs some customer specific information before heading on to the network configuration and the software installation of the new Exadata machine such as the hostnames, the IPs address of each components, the default gateway, etc…
Since Exadata X2, Oracle is kind enough to provide an easy way to do that with a software named Oracle Exadata Deployment Assistant (OEDA). The client then download OEDA on his laptop, fill all the needed information and then generate an XML file (named <client>-<cluster>.xml) that contains all the required information for the DBA to proceed with the installation. This file is vital as all the installation procedure is based on it.
OEDA for Linux
OEDA is not only used to fill the XML file, it is also used to proceed with the set up (and as OEDA version can move pretty quickly, it is NOT provided with the Exadata machine), we then need to upload on the first database server the following components :
- OEDA for Linux
- The <client>-<cluster>.xml file
- The databasemachine.xml file is nice to have as well as we can then use it to make an Exadata Status Script for example
There’re 2 ways described in the official documentation to do that; usually, the client plug an USB key and copy the OEDA zip file and the XML configuration file let’s say in /root/config_files.
Exadata set up
What’s in it ?
First, let’s have a look at what’s in this new Exadata ! we can easily know what is in the box by using the ibhosts command from any part of the IB network (here, I did it from an IB switch as there’s not much server reachable at this part of the installation)
Here, we can see that we have 5 cells (5 storage servers) :
- node1 elasticNode 172.16.2.37
- node2 elasticNode 172.16.2.38
- node3 elasticNode 172.16.2.39
- node4 elasticNode 172.16.2.40
- node6 elasticNode 172.16.2.42
And 4 database servers :
- node8 elasticNode 172.16.2.44
- node9 elasticNode 172.16.2.45
- node10 elasticNode 172.16.2.46
- node12 elasticNode 172.16.2.48
As no network configuration has been done yet, all components have the default factory hostnames and IP. Oracle uses the private 172.16 network for the factory IP settings starting with the first database node at 172.16.2.44. Oracle uses the Infiniband port number where the node is plugged (first database node is plugged on the port number 8) + 36 : 36 + 8 = 44 then the default database server node 1 is 172.16.2.44.
Let’s then configure these IP and hostnames with the customer’s requirements in order to be able to access the database and storage servers from the customer network.
/etc/hosts of the IB Switches
There’s not much to configure on the IB Switches, just to check that the IP and the host are declared in the /etc/hosts; add it if it’s not already done :
To start with, we have to connect to the ILOM of the first database server (the IP is accessible from the customer network).
As we don’t need to do much things on the ILOM itself, let’s head on to the first database server using the ILOM console :
Then here press <ENTER> and login as root
As we saw previously, the first database server is <node8> and a Linux OS is pre-installed
Exadata database servers come with Linux pre-installed but it is also possible to switch to Oracle VM which is also pre-installed on disk. In order to not waste any disk space, we can reclaim that space and any other unused space by executing the below command on every database server (list of IP for db servers can be taken from the ibhosts command like shown earlier) :
Create a OneCommand directory, copy and unzip OEDA and the XML configuration file provided by the client in it.
Note that it is not allowed to run OEDA from a root (/opt) directory then use /u01 to unzip it. Here is the error you would face if you launch OEDA from /opt :
Invoke OEDA from a non root file system. Current directory /opt/oracle.SupportTools/onecommand/linux-x64 is part of the root file system Non-root file systems with required space are: File System /u01 free space 95485 MB
We can now proceed with the network configuration (the applyElasticConfig.sh shell script is provided by OEDA and is located under /opt/oracle.SupportTools/onecommand/linux-x64)
This will apply all the customer’s specifications to the database and storage servers and all servers will be rebooted at the end of their configuration.
Once the set-up applied, you will be able to connect to each database and storage servers from the customer network using their 10.x.y.z specific IP and their hostnames if they have already been defined in the corporate DNS.
An Issue with the Cells IP config
It happened that after the previous step, the cells were not well configured and then the cells configuration is different from the real network configuration shown by the “ifconfig” command. We have to correct that manually to avoid any issue later on specially when applying some patches; here are the error messages you could face if your cells are misconfigured (“CELL-01533: Unable to validate the IP addresses from the cellinit.ora file because the IP addresses may be down or misconfigured” and “ORA-00700: soft internal error, arguments: [main_6a], , [IP addresses in cellinit.ora not operational], , , , , , , , , “).
Here is the output of the ifconfig command on one cell (is pasted here only the information we need for visibility purpose) :
Here is a (bad) output of one cell (is pasted here only the information we need for visibility purpose) :
Note : “/24” is the network mask, it should be set to “/22” in the following step
Who’s right then ? well, ibhosts doesn’t lie :
We then need to update the configuration manually (even the hostname has to be set), restart the services :
And then a quick check of the new config :
Note : this procedure has to be done for every cell — if your cells were misconfigured by the ./applyElasticConfig.sh -cf <CLIENT>-mycluster.xml command only
dbs_group, cell_group and all_group
You have probablyalready seen many Exadata documentations mentioning files named dbs_group, cell_group and all_group. But where do they come from ? Well, you, the DBA, create them:)
As these files are used on our daily basis administration tasks, it is a good idea to put them in a well known directory.
Here is few commands to create them quickly — I also like to have an ib_group file with the name of my IB Switches, this will be very useful when you will patch the IB Switches.
root SSH equivalence
It is better to have a root SSH equivalence to ease all the installation and the administration tasks. We will choose one server (I use to choose DB server node 1 aka “myclusterdb01″) and will deploy its SSH keys to all the other DB, cell nodes and the IB Switches.
First we need to generate the root’s SSH environment (press <ENTER> after each question) :
And we will use the wonderful dcli to deploy the ssh keys to all the database and storage servers from the database node 1:
Note : you will be prompted for the password of each server, don’t mess up, root user is locked after one failed attempt :)
Let’s do the same for the IB Switches :
You are now able to connect from the first DB server to any host without password. You can quickly test it with this command : “dcli -g ~/all_group -l root date“
All is now ready to jump to the installation part which is detailed in the part 2 of this blog !
Very good and detailed post Fred. However this must be really old version of OEDA ? Since Oct 2015 release of OEDA you need to run onecommand from non root directory (/u01/onecommand) or you’ll get “Invoke OEDA from a non root file system” error. I suggest you also run CheckHWnFWProfile on each compute node before ApplyElasticConfig to make sure there are no issues, had few before.
Good catch ! indeed, we cannot start OEDA from /opt, I messed up a bit in my notes here. This is corrected !
-> page not found
Thanks for your comments.
I’ve added a “coming soon” for Part 2 as it is still under validation by the Pythian blog boss :) should be online very soon.
Have a good day,
Part 2 is online and can be found here : https://blog.pythian.com/install-brand-new-exadata-x5-part-2/
the following note https://www.oracle.com/technetwork/articles/oem/exadata-commands-part2-402442.html
explain that *_group files are created automatically by the ‘configuration worksheet’. However, it’s not really clear in their note when the files are created. Is it the install.sh script which create it ? Is it why you have created yourself as they are not created yet ?
Thanks for your reply
Thanks for your comment.
This is most likely because they launched OEDA from the Exadata itself (The configuration worksheet creates the following files in the directory /opt/oracle.SupportTools/onecommand.).
In this blog, I assume that someone from the client side uses OEDA from his laptop and then give the DBA what has been generated.
It looks closer to the real life to me as before the installation, the DB Servers only have the default IP, it could be messy to forward the X11 display with this default network configuration then it looks easier to me that the client downloads OEDA on his laptop and generates the XML configuration file. I also saw people generating the XML configuration file before their Exadata was delivered.
This is then why I gave the procedure to create those files.
Have a good day,
Thanks for your reply. Clear and really helpful.
Good work as usual !