Exadata is the Ferrari of the technology world. It’s considered the technology for the select few, but everyone dreams about it. There is very little information available on the Internet right now about the architecture and the administration of Exadata, though it’s a huge subject in itself, which demand numerous high quality books, dedicated blogs, websites and special interest groups.
This blog post just mentions some of very important components of Exadata Storage Server, which are physical disks, cell disks, grid disks and ASM disks and their correspondence.
The Exadata Storage Server contains 12 physical disks.
There is one to one relationship between a physical disk and the cell disk. One physical disk corresponds to a single cell disk.
Then a Grid disk comprises of many cell disks.
And a Grid disk corresponds to one ASM disk.
An ASM diskgroup comprises of many ASM disks.
On the Exadata Storage Server, We can use cellcli command line utility in Exadata to see the information about physical disks, cell disks, grid disks and the cell.
The cellcli utility works from the root, celladmin and cellmonitor (read-only) users. The best practice is actually to run it from the last two less-privileged users, and not from the root user.
Now let’s have look at some of the disk management commands using cellcli utility.
[root@mycell-net0 ~]# cellcli CellCLI: Release 11.2.1.3.1 - Production on Fri Oct 29 07:47:26 GMT 2010 Copyright (c) 2007, 2009, Oracle. All rights reserved. Cell Efficiency Ratio: 140
Just to give you an idea about what cellcli has on offer, look at the output of the help command:
CellCLI> help HELP [topic] Available Topics: ALTER ALTER ALERTHISTORY ALTER CELL ALTER CELLDISK ALTER GRIDDISK ALTER IORMPLAN ALTER LUN ALTER THRESHOLD ASSIGN KEY CALIBRATE CREATE CREATE CELL CREATE CELLDISK CREATE FLASHCACHE CREATE GRIDDISK CREATE KEY CREATE THRESHOLD DESCRIBE DROP DROP ALERTHISTORY DROP CELL DROP CELLDISK DROP FLASHCACHE DROP GRIDDISK DROP THRESHOLD EXPORT CELLDISK IMPORT CELLDISK LIST LIST ACTIVEREQUEST LIST ALERTDEFINITION LIST ALERTHISTORY LIST CELL LIST CELLDISK LIST FLASHCACHE LIST FLASHCACHECONTENT LIST GRIDDISK LIST IORMPLAN LIST KEY LIST LUN LIST METRICCURRENT LIST METRICDEFINITION LIST METRICHISTORY LIST PHYSICALDISK LIST THRESHOLD SET SPOOL START
Let’s see the output of some of the commands listed above:
CellCLI> list physicaldisk detail name: [5:3:2:0] diskType: FlashDisk id: 00000200000000000000 luns: 1_2 makeModel: "MARVELL SD88SA02" physicalFirmware: D20R physicalInsertTime: 2010-10-04T21:05:46+00:00 physicalInterface: sas physicalSerial: 0000000000000000000 physicalSize: 22.8880615234375G slotNumber: "PCI Slot: 1; FDOM: 2" status: normal <pre>
CellCLI> list cell detail name: mycellnet0 bmcType: IPMI cellVersion: OSS_11.2.0.1.0_LINUX.X64_100818.1 cpuCount: 16 fanCount: 12/12 fanStatus: normal id: 0000X00000 interconnectCount: 3 interconnect1: bond0 iormBoost: 0.0 ipaddress1: kernelVersion: 2.6.18-194.3.1.0.2.el5 makeModel: SUN MICROSYSTEMS SUN FIRE X4275 SERVER SAS metricHistoryDays: 7 notificationMethod: snmp notificationPolicy: critical,warning,clear offloadEfficiency: 47,485.1 powerCount: 2/2 powerStatus: normal smtpFrom: smtpFromAddr: smtpPort: smtpServer: smtpToAddr: smtpUseSSL: snmpSubscriber: status: online temperatureReading: 27.0 temperatureStatus: normal upTime: 16 days, 14:01 cellsrvStatus: running msStatus: running rsStatus: running <pre>
CellCLI> list celldisk detail name: FD_15_mycellnet0 comment: creationTime: 2010-07-14T02:05:06+00:00 deviceName: /dev/sdy devicePartition: /dev/sdy diskType: FlashDisk errorCount: 0 freeSpace: 0 id: 00000000-0000-0000-0000-000000000000 interleaving: none lun: 5_3 size: 22.875G status: normal <pre>
CellCLI> list griddisk detail name: CD_15_mycellnet0 availableTo: cellDisk: CD_15_mycellnet0 comment: creationTime: 2010-07-14T02:12:55+00:00 diskType: FlashDisk errorCount: 0 id: 00000000-0000-000-0000-000000000000 offset: 528.734375G size: 29.125G status: active
In the future posts, I will be touching the administration of celldisks, as how to perform operations like (import/export/create/drop/alter) on the celldisks.
Author
Interested in working with Fahd? Schedule a tech call.
10 Comments. Leave new
Why are there no device file names listed for the griddisk? This is what you make ASM disks out of right?
–R
Hi Rich,
I assume you’re referring to the deviceName and devicePartitions seen for celldisks but not for griddisks. The griddisks are Exadata storage server concepts rather than OS concepts, and can’t be accessed as OS devices. The ASM instances sit on the database servers rather than storage, so it’s a remote request over the network rather than a /dev device. For Exadata disks, the ASM diskstring looks like o/storage server IP/griddisk name
Cheers,
Marc
[…] Cellcli, an Exadata utility which is used to manage Exadata storage, is another hot topic of the week. Kerry Osborne sheds light on it here, and Pythian Blog here. […]
Good document, however I differ on one specific line which says a grid disk comprises of many cell disks. I believe a cell disk can be sliced in many grid disks or a single grid disk.
HTH.
Abhijit
>Then a Grid disk comprises of many cell disks.
No. Cell disks are the low-level physical Exadata representation of a device. Early on, we (Exadata development) considered supporting celldisk creation from HW RAID volumes but opted for a 1:1 relationship instead for many reasons. Griddisks are the virtualization of celldisks (the presentation form to ASM). To that end there is a M:M relationship between celldisks and griddisks.
>There is very little information available on the Internet right now about the architecture
…really?
@Fahd
@Kevin
Based on above I made following Assumptions.Correct me if I am wrong..
1.each storage server will have 12 cell disks
2.each storage server can have maximum 12 grid disks or less if any grid disk is made up of more than one cell disk
3.Assuming ASM will be made up of Multiple storage server.each storage server can provide maximum 12 ASM Disks to Exadata server
4.ASM diskgrup can use any number of ASM Disk, comming from any storage server residing in exadata machine
TIA
Anuj
[…] https://www.pythian.com/news/17905/physical-disk-cell-disk-grid-disk-and-asm-disk-in-exadata/ […]
Very help full information. I have also listed few CellCLI commands at a web page https://www.dbas-oracle.com/2013/05/7-Important-CellCLI-Commands-for-Exadata-DBA.html
Hello Friends,
Good Day.
Suppose there is a Half-Rack Exadata machine with 4 DB and 7 Storage Nodes (Cells).
Suppose we issue a SELECT statement to query few rows of a table. My curiosity is how would the DB/Storage know exactly which cells to look for to fetch the queried rows without having to search all the 7 cells whether smart scan is being used or whether it isn’t.
I am unable to create celldisk.
after command create celldisk all its showing following error:
missing class: oracle.osmgmt.ms.cli.cellcli
please help