Considerations for Deploying RAC
As we have shown in this chapter, RAC embodies a great technology solution for achieving
HA and scalability of Oracle
database services. However, this solution itself has a complex hardware and software
technology stack. Before an IT
organization decides to adapt Oracle RAC technology for its database architecture, it
should be aware of the advantages
and potential disadvantages of the Oracle RAC technology, and its implications for the
organization’s business
goals. This will help to justify the adoption of Oracle RAC for the business. The next section
highlights some related
considerations that will help you decide whether or not Oracle RAC should be used as the
database architecture.
Cost of Ownership
One of the possible reasons that IT departments are looking at Oracle RAC is to reduce
the cost of ownership of the
database infrastructure. This cost saving is relative, depending on what you are comparing.
The cost of Oracle RAC
implementation includes three parts: hardware infrastructure cost, Oracle software cost,
and management cost.
The hardware stack consists of multiple servers, redundant networks, and shared storage.
The software cost mainly
includes Oracle RAC license and the Oracle Database license. For Oracle Database
Enterprise Edtion, the Oracle RAC
license is separate from Oracle Database license, while for Oracle Database standard
edition, the Oracle Database
license already includes the Oracle RAC license which you don’t have to pay for separately.
One of the limitations of Oracle Standard Edition is that the total number of CPU sockets
of all the servers in the
cluster can not go beyond 4. A CPU socket is a connection that allows a computer
processor to be connected to a
motherboard. A CPU socket can have multilple CPU cores. For example, a Dell R820
server has four CPU sockets while
a Dell R720 server has two sockets. Since each socket can have 8 cores, an R820 server
can have up to 4 * 8 = 32
CPU cores, and a Dell R720 server can have up to 16 CPU cores. Using Oracle Standard
Edition with a maximum
capacity of 4 CPU sockets, you can make a two-node Oracle RAC cluster with Dell R720
servers, but only a one-node
cluster with a Dell R820 server. For Oracle Enterprise Edition, the RAC license can be
based on the total number
of processors. This is based on the total cores of the servers in the cluster. For example, for
a two-node Oracle RAC
configuration using Dell R720s with 8 core CPU sockets, the total number of the CPU
cores can be 2 * 2 * 8 = 32.
Management staff cost is related to the cost of training and attracting individuals with
the skills needed (system
admins, network admins, and DBAs) to manage it. The hardware and software costs
include the initial purchase cost
as well as the ongoing support cost. Although this cost is higher than a simple database
solution like a single-node MS
SQL server, the RAC solution is cheaper than typical complex mission-critical databases
running on big SMP servers,
as Oracle RAC is mainly implemented on Linux and industry-standard low-cost
commodity hardware. In the last
decade, these industry-standard servers running Linux have become much cheaper, and
offer a powerful and reliable
solution widely accepted for enterprise systems.
Another cost-saving factor is that Oracle RAC can be implemented as a shared resource
pool to consolidate many
databases. This can significantly reduce the costs of hardware, software, and management
by reducing the number of
systems. In the Oracle E-Business database consolidation example mentioned in the last
section, 100 databases were
consolidated onto a 16-node RAC. The number of database servers was reduced from
100 or more to 16. The reduction led
to huge savings in hardware, software, and management. As already mentioned, long-
term operating costs are also cut by
reducing the need for support, maintenance, and even powering and cooling 100 systems
in a data center for the entire life
cycle of the environment. For full details of this example, refer to my technical
presentation at Oracle OpenWorld:
[Link]
[Link].
25CHAPTER 1 ■ OVERVIEW OF ORACLE RAC
26
High Availability Considerations
Oracle RAC provides HA of the database service by reducing unplanned and planned
downtime caused by server
failure. But RAC itself doesn’t protect the database against other failures, such as storage
failure, data corruption,
network failure, human operation error, or even data center failure. To provide complete
protection against these
failures, additional measures need to be taken. Oracle MAA (Maximal Availability
Architecture) lists the guidelines
and related Oracle technologies needed to protect databases against those failures.
During the deployment of RAC, it is critical to follow HA practices to ensure the stability
of the RAC. The most
important hardware components that Oracle RAC relies on are the private network and
shared storage. The private
network should be based on a redundant network with two dedicated switches. Chapter 9
discusses the RAC network
in detail. The shared storage access should be based on multiple I/O paths, and the
storage disk drives should be set
up with a RAID configuration and Oracle ASM disk mirroring to ensure redundancy.
Chapter 5 discusses storage best
practices in detail.
In theory, Oracle RAC protects the database service against failure of up to N-1 servers
(where N is the total
number of servers). In reality, if all of the N-1 servers fail, the workloads of the entire
clusterware will be on the
only surviving node, and the performance will definitely suffer unless each server leaves
N-1/N headroom. For
example, for a four-node RAC, leaving 3/4 (75%) headroom would not be realistic. A
realistic approach is to
ensure that each server in the cluster can handle the failed-over workload in case of single
server failure. This
requires each server to leave only 1/N headroom. And the bigger N is, the less headroom
is needed. The worst
case is a two-node RAC, where each server needs to reserve 1/2 (50%) headroom. For a
four-node RAC,
only 1/4 = 25% headroom is needed.
■ Note CPU headroom is the CPU resource that we have to leave unused in case
of server failure. The less headroom,
the better resource utilization on each node.
Scalability Considerations
Oracle RAC provides database scalability. With the addition of each extra RAC node, the
cluster is expected to
increase database performance capability: handling larger workloads or more concurrent
users, performing more TPS
(transactions per second) for OLTP, or reducing the average transaction/query response
time. However, many RAC
databases may not show linear scalability when adding more RAC nodes. This is because
there are many other factors
that are related to database scalability:
1.
Poor database design and poorly tuned SQL queries can lead to very costly query plans
that may kill database throughput and significantly increase query response time. Poorly
tuned queries will run just as badly (or even worse) in RAC compared to a single-node
database.
2.
There may be quite costly performance overhead caused by Oracle cache fusion, and
excessive wait time on data blocks transferring on interconnects between RAC nodes
during query executions and database transactions. These wait events are called cluster
wait events. Cache fusion overhead and cluster wait events may increase when multiple
RAC instances access the same data blocks more frequently. A higher number of RAC
nodes also contributes to cluster waits and slows down the interconnect. The number
of RAC nodes is limited by the bandwidth of the interconnect network, which is less of
an issue with the introduction of high-speed networks such as InfiniBand and 10-40GB
[Link] 1 ■ OVERVIEW OF ORACLE RAC
27
3.
In some database environments with I/O-intensive workloads, most performance
bottlenecks are on storage I/O with lower CPU utilization. Such environments will not
scale
well just by adding more RAC nodes. Therefore, it is important to understand the
workload
characteristics and potential performance bottlenecks before we opt for the scalable
solution. Storage performance capacity is measured in IOPS (I/O operations per second)
for OLTP workloads and throughput MB/second for DSS workloads. Since the speed of
hard
disks is limited by physics (drive seek time and latency), they tend to impose an upper
limit
on IOPS and throughput. One way to scale storage performance is to add more disk
drives
and stripe the data files with either RAID or Oracle ASM striping. Another option is to
move
frequently accessed data (hot data) to solid disk drives (SSDs). SSDs provide much higher
IOPS, especially for the random small I/O operations which dominate OLTP workloads, as
SSDs have no moving parts and hence no mechanical delays. Using SSDs is a very viable
option to scale storage IOPS performance for OLTP workloads. For DSS workloads, one
option is based on the building block concept. Each building block is composed of a RAC
node plus additional storage and network based on the balance between CPU processing
power and storage throughput for a DSS/Data warehouse–type workload. Scalability is
based on the building block instead of just a server.
RAC or Not
When IT organizations need to decide whether to deploy an Oracle RAC as their database
architecture, IT architects
and DBAs need to make decisions based on many factors.
1.
The High availability SLA: How much database downtime is acceptable for both
unplanned and planned downtime? Without RAC, planned downtime for hardware
and software maintenance may vary from a few minutes to as much as a few hours. And
the unplanned time for hardware and software problems can also vary from minutes
to hours. If a database server is completely lost, it will take a longer time to rebuild it,
although some downtimes, like the complete loss of the server, may occur only rarely,
Are these potential downtimes acceptable to the business according to the SLA? If not,
can downtime prevention justify the cost of the RAC deployment? And furthermore, a
loss of the entire database system including the storage may take hours or days to recover
from the backup. Does this justify a Disaster Recovery (DR) solution which consists of a
completely duplicated system in another data center? Some mission-critical databases
may be equipped with the combination of RAC and Data Guard DR solution to protect
the database from server failure as well as storage and site failure. However, the cost and
technical complexity need to be justified by business needs.
2.
Scalability Requirement: What kind of workloads are there and where is the performance
bottleneck: CPU intensive and/or storage I/O intensive? If there is no need to scale out
CPU/memory resources or if we can scale up by adding additional CPUs or memory,
Oracle RAC One Node (instead of multiple RAC RAC) may be a good way of providing HA
without the need to pay for an RAC license. RAC One Node also has the flexibility to be
easily upgraded to a full RAC solution any time there is a need to scale out to multi-node
RAC in the future.
3.
Database Consolidation Requirements: If the organization has a business requirement
to consolidate many database services together, a multi-node RAC can offer significant
advantages in terms of cost of ownership wjile providing HA and scalability to all the
[Link] 1 ■ OVERVIEW OF ORACLE RAC
28
4.
If the organization decides to deploy the RAC solution, it should fulfil the hardware
requirements and follow configuration and management best practices to ensure that the
RAC provides all the potential advantages.
5.
Many companies have carried out successful migration of their mission-critical databases
from big SMP Unix machines to multi-node Oracle RAC clusters based on lower-cost
industry-standard commodity X86-64 servers running Linux. In the last decade, these
servers have advanced significantly in terms of reliability as well as processing power.
Practical experience has shown that the new architecture can provide a highly available
and scalable infrastructure for enterprise-level applications