A hands-on lab covering the CellDisk → GridDisk → ASM storage stack, Grid Disk failure simulation, Cell Disk recreation, and recovery validation.
1. Introduction
One of the most important concepts for an Exadata DBA is understanding what happens between the physical storage and ASM.
The storage hierarchy can be simplified as:
Physical Disk / LUN
↓
Cell Disk
↓
Grid Disk
↓
ASM Disk
↓
ASM Disk Group
In a simulator environment, the physical disks may be represented by files such as:
/opt/oracle/.../disks/raw/DISK06
This makes it possible to practice CellCLI storage administration without modifying physical Exadata hardware.
⸻
2. Understanding CellDisk and GridDisk
A Cell Disk represents the usable storage carved from a physical disk/LUN.
A Grid Disk is a logical slice of that Cell Disk that is presented to ASM or other consumers.
For example, a 5 GB Cell Disk can be divided into multiple Grid Disks:
CD_DISK06 — 5 GB
│
├── DATA
│ Offset: 48M
│ Size: 3G
│
├── RECO
│ Offset: 3.046875G
│ Size: 1.5G
│
└── DBFS
Offset: 4.546875G
Size: 464M
The important point is that offset is the starting position of a Grid Disk inside its Cell Disk.
For example:
DATA:
offset = 48M
size = 3G
means the DATA Grid Disk begins at approximately 48 MB into the Cell Disk.
⸻
3. Check the Physical Disk
On the Cell Server:
ls -lh /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/
You may see:
DISK01
DISK02
DISK03
DISK04
DISK05
DISK06
DISK07
FLASH01
FLASH02
FLASH03
FLASH04
Enter CellCLI:
cellcli
Check physical disks:
LIST PHYSICALDISK;
For a specific disk:
LIST PHYSICALDISK '/opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/DISK06' DETAIL;
Important attributes include:
name
diskType
physicalSize
status
errorCount
physicalSerial
physicalInsertTime
⸻
4. Check the Cell Disk
LIST CELLDISK;
For a specific Cell Disk:
LIST CELLDISK 'CD_DISK06_ex01celadm03' DETAIL;
Useful attributes:
LIST CELLDISK ATTRIBUTES name,size,freeSpace,status;
This allows you to determine:
- Cell Disk size
- Available space
- Parent physical disk
- Current status
⸻
5. Check the Grid Disks
List all Grid Disks:
LIST GRIDDISK;
For DISK06:
LIST GRIDDISK WHERE cellDisk='CD_DISK06_ex01celadm03'
ATTRIBUTES name,cellDisk,size,offset,status;
For detailed information:
LIST GRIDDISK 'data_CD_DISK06_ex01celadm03' DETAIL;
The important attributes are:
name
cellDisk
size
offset
status
asmDiskName
asmDiskgroupName
asmFailGroupName
⸻
6. Why Offset Matters
Suppose the Cell Disk is:
5 GB
and contains:
DATA → 3 GB
RECO → 1.5 GB
DBFS → 464 MB
The approximate layout is:
0
│
├── Reserved/metadata area
│
├── 48 MB
│
├──────── DATA 3 GB ────────┤
│ │
├──────── RECO 1.5 GB ──────┤
│ │
├──────── DBFS 464 MB ──────┤
│ │
└───────────────────────────┘
5 GB
The offsets tell Cell Services where each Grid Disk begins.
Therefore, if you’re recreating Grid Disks over existing underlying data, preserving the original:
Grid Disk name
Offset
Size
is critical.
⸻
7. Simulating Grid Disk Failure
Before making changes, record the existing configuration.
For example:
LIST GRIDDISK ATTRIBUTES name,cellDisk,size,offset,status;
Save the output.
Also record:
LIST CELLDISK ATTRIBUTES name,size,freeSpace,status;
And:
LIST PHYSICALDISK ATTRIBUTES name,physicalSize,status;
This gives you a recovery reference.
⸻
8. Make Grid Disks Inactive
In a controlled training environment, you can practice changing Grid Disk state.
Do not perform destructive operations on a production Exadata system just to reproduce this lab.
The purpose of this exercise is to understand the lifecycle:
Active
↓
Inactive
↓
Dropped
↓
Cell Disk recreated
↓
Grid Disks recreated
↓
ASM validation
The exact CellCLI syntax should always be checked against the CellCLI version in use.
⸻
9. Drop the Cell Disk
The lab scenario can then demonstrate removing the Cell Disk object from Cell Services while leaving the underlying simulator disk file untouched.
This distinction is extremely important.
You can have:
CellCLI metadata
↓
Cell Disk
↓
Grid Disks
while separately having the underlying simulator file:
/opt/oracle/.../disks/raw/DISK06
The Cell Disk object and the underlying file are not the same object.
Therefore, removing a Cell Disk object in the simulator does not automatically mean that the physical simulator file has been deleted.
⸻
10. Verify the Physical Disk File
After the Cell Disk is removed, verify at the Linux level:
ls -lh /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/DISK06
You can also check:
stat /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/DISK06
And:
du -h /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/DISK06
The purpose is to demonstrate:
CellCLI object removed
≠
Underlying simulator disk file deleted
⸻
11. Recreate the Cell Disk
In the simulator, the Cell Disk can be recreated using the underlying LUN/file reference.
The exact syntax depends on the CellCLI/simulator implementation.
The important concept is:
Existing physical/LUN reference
↓
Recreate Cell Disk
↓
Same underlying storage
Do not blindly copy a simulator command into a real Exadata environment. The storage architecture and supported commands differ.
⸻
12. Recreate the Grid Disks
Once the Cell Disk has been recreated, the next task is to reproduce the original Grid Disk layout.
For example:
CD_DISK06
│
├── DATA
│ Offset = 48M
│ Size = 3G
│
├── RECO
│ Offset = 3.046875G
│ Size = 1.5G
│
└── DBFS
Offset = 4.546875G
Size = 464M
The objective is to reproduce the original logical layout rather than simply creating three arbitrary Grid Disks.
⸻
13. Verify CellCLI Recovery
After recreation:
LIST PHYSICALDISK;
Then:
LIST CELLDISK;
Then:
LIST GRIDDISK;
And finally:
LIST GRIDDISK ATTRIBUTES
name,cellDisk,size,offset,status;
Compare the result with the configuration you recorded before the exercise.
⸻
14. Verify ASM
Move to the ASM/database host:
sqlplus / as sysasm
Check ASM diskgroups:
SELECT
name,
state,
type,
total_mb,
free_mb
FROM v$asm_diskgroup;
Check ASM disks:
SELECT
name,
path,
header_status,
mode_status,
state,
total_mb,
free_mb
FROM v$asm_disk
ORDER BY path;
If the environment uses the appropriate ASM discovery configuration, verify that the recreated Grid Disks are visible again.
⸻
15. Final Recovery Validation
The recovery is not complete simply because CellCLI shows:
status = active
You should validate every layer.
Physical layer
ls -lh .../DISK06
Cell Disk layer
LIST CELLDISK 'CD_DISK06_ex01celadm03' DETAIL;
Grid Disk layer
LIST GRIDDISK WHERE cellDisk='CD_DISK06_ex01celadm03'
ATTRIBUTES name,size,offset,status;
ASM layer
SELECT name,state,type,total_mb,free_mb
FROM v$asm_diskgroup;
and:
SELECT name,path,header_status,mode_status,state
FROM v$asm_disk;
Database layer
Finally, validate that the database can access the affected data normally.
⸻
16. What I Learned From This Lab
The biggest lesson isn’t the CellCLI syntax.
It’s understanding which layer you’re working on.
Exadata Storage Stack
Physical Disk / LUN
│
▼
Cell Disk
│
┌─────────┴─────────┐
▼ ▼ ▼
DATA RECO DBFS
Grid Disk Grid Disk Grid Disk
│ │ │
└─────────┴─────────┘
│
▼
ASM Disk
│
▼
ASM Disk Group
│
▼
Database
A Cell Disk problem is not automatically an ASM problem.
A Grid Disk problem is not automatically a physical disk failure.
And deleting a CellCLI storage object in a simulator is not necessarily equivalent to deleting the underlying storage.
That separation of layers is one of the most important concepts to understand when learning Exadata administration.
⸻
⚠️ Lab Safety
This procedure is appropriate as a controlled training/simulator exercise. Do not reproduce the drop/recreate steps against a production Exadata system without an approved recovery procedure, backups, and Oracle-supported operational guidance
Comments
Post a Comment