0% found this document useful (0 votes)
43 views4 pages

Exadata Storage Architecture Overview

The Exadata storage layout divides the system into a database layer and storage layer connected via an InfiniBand network. The storage layer uses Oracle ASM over InfiniBand and cell servers to access storage more efficiently using techniques like Smart Scans. Exadata configurations include full, half or quarter racks with varying server and core counts.

Uploaded by

Mukarram Khan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
43 views4 pages

Exadata Storage Architecture Overview

The Exadata storage layout divides the system into a database layer and storage layer connected via an InfiniBand network. The storage layer uses Oracle ASM over InfiniBand and cell servers to access storage more efficiently using techniques like Smart Scans. Exadata configurations include full, half or quarter racks with varying server and core counts.

Uploaded by

Mukarram Khan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Exadata Storage Layout

To understand Exadata lay out it will be helpful if we divide the entire system mentally into two
parts, the storge layer and the database layer.

The database layer or database servers works same as any non-exa machines with ASM to access the
storage. We can have rac or non-rac database servers on the exa machines. ASM is required even if
the databases are not configured to use RAC.

Both the layers are connected via an an InfiniBand network. InfiniBand provides a low-latency, high-
throughput switched fabric communications link. Redundancy is provided through multiple switches
and links.

iDB is network-based protocol that is implemented using InfiniBand. iDB is used to send requests for
data along with metadata about the request (including predicates) to cellsrv. In certain
situations, cellsrv is able to use the metadata to process the data before sending results back to
the database layer. When cellsrv is able to do this, it is called a Smart Scan and generally results in
a significant decrease in the volume of data that needs to be transmitted back to the database
layer. When Smart Scans are not possible, cellsrv returns the entire Oracle block(s). Note that iDB
uses the RDS(Reliable Datagram Sockets) protocol, which is a low-latency, InfiniBand-specific
protocol. In certain cases, the Oracle software can set up remote direct memory access (RDMA) over
RDS, which bypasses doing system calls to accomplish low-latency, process-to-process
communication across the InfiniBand network.

Configuration Options
Exadata is delivered as a preconfigured, integrated system, there are few options available like X8-8
or X8-2. We can have a full, half or quarter rack configuration. This configuration mainly differs in the
servers and cores. Please refer oracle documentation for more details. This document is more from
layout perspective

Oracle Exadata X8M-2 System Model Specifications


Exadata Software Architecture
As discussed above Exadata machine is mainly divided into two parts, database software and storge
software which are connect via Idb over InfiniBand . Below figure gives more systematic overview of
the architecture stack and how it spans multiple servers in both the database grid and the storage
grid.

From the storage point of view ASM is a key component, ASM is an object that cuts across, all the
communication lines between the two tiers. This is meant to indicate that ASM provides the
mapping between the files and the objects that the database knows about on the storage layer. ASM
does not actually sit between the storage and the database

The Major component in the database software is LIBCELL, which is a library that is linked with the
Oracle kernel. LIBCELL has the code that knows how to request data via iDB. This provides a very
nonintrusive mechanism to allow the Oracle kernel to talk to the storage tier via network-based calls
instead of operating system reads and writes. iDB is implemented on top of the RDS protocol.

Exadata Storage Server Architecture


The architecture on the storage servers is simple. There is one master process (cellsrv) which
provides many or majority of Exadata storage services like serving the database requests for disk
I/O , advanced SQL offload capabilities also manages the IORM functionality( I already written a post
about IORM [Link]
7145349467624325120--gUb?utm_source=share&utm_medium=member_desktop )
Management server (MS)
The Management Server implements a web service for cell management commands and runs
background monitoring threads. It works with the Exadata cell command line interface (CellCLI) and
processes most of the commands from [Link] MS is the primary interface to administer,
manage and query the status of the Exadata cell.

Restart server (RS)

Restart Server (RS) ensures the ongoing functioning of the Exadata software and services. It
monitors the heartbeat with the Management server and the Cellsrv processes. If the processes fail
to respond within the allowable heartbeat period, RS Process will restart Cellsrv and Management
server processes. So the Purpose of RS is to make sure MS and CellSrv are up and running fine.

Also, from the architecture diagram we can see cellsrv uses an [Link] file and has an alert log. In
fact, the storage software bears a striking resemblance to an Oracle database. The [Link] file
contains a set of parameters that are evaluated when cellsrv is started. The alert log is used to write
a record of notable events, much like an alert log on an Oracle database. Note also that Automatic
Diagnostic Repository (ADR) is included as part of the storage software for capturing and reporting
diagnostic information.
Exadata Disk Architecture
In Normal oracle database disks are presented directly to ASM, but Exadata database machine we
have a slightly different configuration .

Mapping Physical Disks, LUNs, and Cell Disks on the Storage Servers

Physical Disk:
The Physical disk is the Actual device within the storage cell, these physical disks are connected to
the RAID controller.

Logical Unit Number (LUN)


The Logical Unit Number (LUN) is the Logical Resource Representing Underlying Disk with the Upper
Software Layers. LUN is the lowest level of storage abstraction visible to OS layer We cannot create
or remove LUN it exists because of the presence of physical disk

Every Physical Disk mapped to LUN with 1 on 1 Mapping. Out of 12 physical disks, only on the First 2
disks, there is some area around 30G reserve for OS Storage Area to store OS image, Exadata
software binaries, config files etc. Rest of the 10 disks, the Lun will be same size of Physical disk.

Cell Disk
Then Comes the Cell disk which is based on LUN. The First 2 Disks having System Area carved out and
Rest of the storage space on those 2 disks are mapped as Cell Disks. For the Other 10 Disks, there
will be 1 cell disk for each LUN which is of same size.

Grid Disks
Cell disks can be sub-divided into Grid Disks. Multiple Grid Disks created out of Cell disks which can
be presented to ASM. If ASM Disk Group is to be named as DATA, then the grid disk is created with
prefix DATA. If size is not specified it will take all the available space in the cell disk.

Common questions

Powered by AI

Grid Disks in Exadata are subdivisions of Cell Disks, which in turn are created from LUNs based on physical disks. Grid Disks are the storage units that ASM Disk Groups use within the Exadata infrastructure. They provide a flexible way to allocate storage space, allowing administrators to partition Cell Disks according to storage requirements. Grid Disks facilitate efficient storage resource management and are critical for optimizing the use and performance of the Exadata architecture .

The Automatic Diagnostic Repository (ADR) plays a significant role in supporting the operational efficiency of Exadata's storage software by capturing and reporting diagnostic information. It maintains an alert log similar to an Oracle database, providing a record of notable events that aid in troubleshooting and maintenance. ADR's inclusion ensures that any operational issues in the storage software are promptly identified, which allows for quick rectification and helps maintain optimal system performance and stability .

Smart Scans are a feature of Exadata where the cellsrv (storage server process) uses the metadata provided with a data request to process the data directly within the storage tier. This processing can include filtering rows and columns, performing certain calculations, and returning only relevant data to the database layer, significantly reducing the volume of data that needs to be transmitted. Smart Scans leverage the high-performance capabilities of InfiniBand and iDB to improve query performance by offloading work from the database servers to the storage servers, which reduces CPU load and speeds up data retrieval .

In a normal Oracle database setup, disks are presented directly to ASM, whereas in Exadata, the configuration is more layered. Physical disks in Exadata are connected to RAID controllers and mapped to Logical Unit Numbers (LUNs), which represent the underlying disk to the upper software layers. Each physical disk maps one-to-one with a LUN. Cell Disks are then created from these LUNs, and Grid Disks are subdivided from Cell Disks for ASM presentation. This layered architecture in Exadata allows for greater flexibility and efficiency in storage management compared to a traditional setup .

Cellsrv is the master process in Exadata's storage server architecture responsible for executing multiple critical functions, including handling database requests for disk I/O and providing advanced SQL offloading capabilities. It manages I/O Resource Management (IORM) and facilitates Smart Scans by utilizing metadata to process data directly on the storage servers. These capabilities enhance the performance of Exadata by minimizing unnecessary data transfer and optimizing resource allocation .

ASM in Exadata serves as a key component by providing a mapping between files and objects that the database recognizes on the storage layer. Although ASM is not positioned between the storage and database directly, it provides a unified interface for managing storage and reduces complexity in file management. It handles redundancy, striping, and rebalancing of disks, thus abstracting the underlying physical storage details. This integration is essential in ensuring efficient storage management and performance optimization in Exadata environments .

The Exadata software architecture facilitates low-latency data requests by using the iDB protocol built over the InfiniBand network. This setup employs RDS (Reliable Datagram Sockets) for fast, low-latency communication that avoids traditional operating system calls. The iDB protocol integrates with the Oracle kernel through LIBCELL, a library that leverages network-based calls for direct interaction with the storage tier. This enables efficient data processing and retrieval, reducing latency compared to conventional methods where each data request requires multiple system calls and read/write operations .

The Restart Server (RS) in Exadata is responsible for ensuring the continued functioning of Exadata software and services. It does this by monitoring the heartbeat of the Management Server (MS) and cellsrv processes. If these processes fail to respond within the acceptable heartbeat interval, the RS will automatically restart them to ensure system reliability and uptime. This functionality is critical in maintaining the stability and resilience of the Exadata system, reducing downtime, and ensuring consistent performance .

The Management Server (MS) in Exadata facilitates access to cell management commands and runs background monitoring threads. It serves as the primary interface for administering, managing, and querying the status of Exadata cells via the CellCLI (Cell Command Line Interface). The MS interacts closely with the Restart Server (RS) to ensure that cellsrv and MS processes are running smoothly by monitoring their heartbeat and restarting them if necessary. This monitoring and management capability is crucial for maintaining the reliability and uptime of the Exadata environment .

The InfiniBand network in Exadata provides a low-latency, high-throughput switched fabric communications link that connects the database and storage layers. It facilitates communication by using the iDB protocol, which is implemented over InfiniBand using RDS (Reliable Datagram Sockets) for low-latency, process-to-process communication. The use of iDB with InfiniBand allows the Oracle kernel to make network-based calls to the storage tier, bypassing traditional operating system reads and writes, which enhances performance. Additionally, RDMA (remote direct memory access) can be set up over RDS in certain cases to further reduce latency in data transfer .

You might also like