Conference on File and Storage Technologies (FAST’02), pp.

189–201,
28–30 January 2002, Monterey, CA. (USENIX, Berkeley, CA.)

Selecting RAID levels for disk arrays

Eric Anderson, Ram Swaminathan, Alistair Veitch, Guillermo A. Alvarez and John Wilkes
Storage and Content Distribution Department,
Hewlett-Packard Laboratories, 1501 Page Mill Road, Palo Alto, CA 95014
fanderse,swaram,aveitch,galvarez,wilkesg@[Link]

Abstract Workload

Disk arrays have a myriad of configuration parameters

that interact in counter-intuitive ways, and those interac- Tagger
tions can have significant impacts on cost, performance,
and reliability. Even after values for these parameters (Tagged) workload
have been chosen, there are exponentially-many ways to
map data onto the disk arrays’ logical units. Meanwhile,
Initial
the importance of correct choices is increasing: stor-
assignment
age systems represent an growing fraction of total sys-
Solver
tem cost, they need to respond more rapidly to changing Initial design
needs, and there is less and less tolerance for mistakes.
We believe that automatic design and configuration of
storage systems is the only viable solution to these is- Reassignment
sues. To that end, we present a comparative study of a
range of techniques for programmatically choosing the Final solution
RAID levels to use in a disk array.

Our simplest approaches are modeled on existing, man- Figure 1: The decision flow in making RAID level selections,
ual rules of thumb: they “tag” data with a RAID level be- and mapping stores to devices. If a tagger is present, it irre-
fore determining the configuration of the array to which vocably assigns a RAID level to each store before the solver is
run; otherwise, the solver assigns RAID levels as it makes data
it is assigned. Our best approach simultaneously deter-
layout decisions. Some variants of the solver allow revisiting
mines the RAID levels for the data, the array configura-
this decision in a final reassignment pass; others do not.
tion, and the layout of data on that array. It operates as an
optimization process with the twin goals of minimizing
array cost while ensuring that storage workload perfor-
mance requirements will be met. This approach produces for many configuration parameters must be specified:
robust solutions with an average cost/performance 14– achieving the right balance between cost, availability,
17% better than the best results for the tagging schemes, and application performance needs depends on many
and up to 150–200% better than their worst solutions. correct decisions. Unfortunately, the tradeoffs between
We believe that this is the first presentation and system- the choices are surprisingly complicated. We focus here
atic analysis of a variety of novel, fully-automatic RAID- on just one of these choices: which RAID level, or data-
level selection techniques. redundancy scheme, to use.
The two most common redundancy schemes are
RAID 1/0 (striped mirroring), where every byte of data is
1 Introduction kept on two separate disk drives, and striped for greater
I/O parallelism, and RAID 5 [20], where a single parity
Disk arrays are an integral part of high-performance stor- block protects the data in a stripe from disk drive failures.
age systems, and their importance and scale are growing RAID 1/0 provides greater read performance and failure
as continuous access to information becomes critical to tolerance—but requires almost twice as many disk drives
the day-to-day operation of modern business. to do so. Much prior work has studied the properties of
Before a disk array can be used to store data, values different RAID levels (e.g., [2, 6, 20, 11]).
Disk arrays organize their data storage into Logical even more important is the uncertainty that surrounds a
Units, or LUs, which appear as linear block spaces to manually-designed system: (how well) will it meet its
their clients. A small disk array, with a few disks, might performance and availability goals?
support up to 8 LUs; a large one, with hundreds of disk We believe that automatic methods for storage system
drives, can support thousands. Each LU typically has a design [1, 5, 7, 4] can overcome these limitations, be-
given RAID level—a redundancy mapping onto one or cause they can consider a wider range of workload in-
more underlying physical disk drives. This decision is teractions, and explore a great deal more of the search
made at LU-creation time, and is typically irrevocable: space than any manual method. To do so, these auto-
once the LUhas been formatted, changing its RAID level matic methods need to be able to make RAID-level se-
requires copying all the data onto a new LU. lection decisions, so the question arises: what is the best
Following previous work [7, 25], we describe the work- way to do this selection? This paper introduces a variety
loads to be run on a storage system as sets of stores and of approaches for answering this question.
streams. A store is a logically contiguous array of bytes, The rest of the paper is organized as follows. In Section 2
such as a file system or a database table, with a size typ- we describe the architecture of our RAID level selection
ically measured in gigabytes; a stream is a set of access infrastructure. We introduce the schemes that operate
patterns on a store, described by attributes such as re- on a per-store basis in Section 3, and in Section 4 we
quest rate, request size, inter-stream phasing informa- present a family of methods that simultaneously account
tion, and sequentiality. A RAID level must be decided for prior data placement and RAID level selection deci-
for each store in the workload; if there are k RAID levels sions. In Section 5, we compare all the schemes by do-
to choose from and m stores in the workload, then there
2
are k m feasible configurations. Since k  and m is
ing experiments with synthetic and realistic workloads.
We conclude in Sections 6 and 7 with a review of related
usually over a hundred, this search space is too large to work, and a summary of our results and possible further
explore exhaustively by hand. research.
Host-based logical volume managers (LVMs) complicate
matters by allowing multiple stores to be mapped onto 2 Automatic selection of RAID levels
a single LU, effectively blending multiple workloads to-
gether. Our approach to automating storage system design relies
There is no single best choice of RAID level: the right on a solver: a tool that takes as input (1) a workload de-
choice for a given store is a function of the access pat- scription and (2) information about the target disk array
terns on the store (e.g., reads versus writes; small versus types and their configuration choices. The solver’s out-
large; sequential versus random), the disk array’s char- put is a design for a storage system capable of supporting
acteristics (including optimizations such as write buffer that workload.
merging [23], segmented caching [26], and parity log- In the results reported in this paper, we use our third-
ging [21]), and the effects of other workloads and stores generation solver, Ergastulum [5] (prior solver gener-
assigned to the same array [18, 27]. ations were called Forum [7] and Minerva [1]). Our
In the presence of these complexities, system adminis- solvers are constraint-based optimization systems that
trators are faced with the tasks of (1) selecting the type use analytical and interpolation-based performance mod-
and number of arrays; (2) selecting the size and RAID els [3, 7, 18, 23] to determine whether performance con-
level for each LU in each disk array; and (3) placing straints are being met by a tentative design. Although
stores on the resulting LUs. The administrators’ goals such models are less accurate than trace-driven simula-
are operational in nature, such as minimum cost, or max- tions, they are much faster, so the solver can rapidly eval-
imum reliability for a given cost—while satisfying the uate many potential configurations.
performance requirements of client applications. This is As illustrated in Figure 1, the solver designs configura-
clearly a very difficult task, so manual approaches ap- tions for one or more disk arrays that will satisfy a given
ply rules of thumb and gross over-provisioning to sim- workload. This includes determining the array type and
plify the problem (e.g., “stripe each database table over count, selecting the configuration for each LU in each ar-
as many RAID 1/0 LUs as you can”). Unfortunately, this ray, and assigning the stores onto the LUs. In general,
paper shows that the resulting configurations can cost as solvers rely on heuristics to search for the solution that
much as a factor of two to three more than necessary. minimizes some user-specified goal or objective. All ex-
This matters when the cost of a large storage system can periments in this paper have been run with the objective
easily be measured in millions of dollars and represents of minimizing the hardware cost of the system being de-
more than half the total system hardware cost. Perhaps signed, while satisfying the workload’s performance re-
quirements. type. In some cases we use the same performance
During the optional tagging phase, the solver examines models as we later apply in the solver.
each store, and tags it with a RAID level based on the 2. Solver-based, or integrated, approaches: These
attributes of the store and associated streams. omit the tagging step, and defer the choice of RAID
During the initial assignment phase, Ergastulum ex- level until data-placement decisions are made by the
plores the array design search space by first randomiz- solver. This allows the RAID level decision to take
ing the order of the stores, and then running a best-fit into account interactions with the other stores and
search algorithm [10, 15, 16] that assigns one store at a streams that have already been assigned.
time into a tentative array design. Given two possible as- We explored two variants of this approach: a par-
signments of a store onto different LUs, the solver uses tially adaptive one, in which the RAID level of an LU
an externally-selected goal function to choose the “best” is chosen when the first store is assigned to it, and
assignment. While searching for the best placement of cannot subsequently be changed; and a fully adap-
a store, the solver will try to assign it onto the existing tive variant, in which any assignment pass can re-
LUs, to purchase additional LUs on existing arrays, and
visit the RAID level decision for an LU at any time
to purchase additional arrays. A goal function that favors during its best-fit search. In both cases, the reassign-
lower-cost solutions will bias the solver towards using ment pass can still change the bindings of stores to
existing LUs where it can. LUs, and even move a store to an LU of a different
At each assignment, the solver uses its performance RAID level.
models to perform constraint checks. These checks en- Neither variant requires any ad hoc constants, and
sure that the result is a feasible, valid solution that can ac- both can dynamically select RAID levels. The fully
commodate the capacity and performance requirements adaptive approach has greater solver complexity
of the workload. and longer running times, but results in an explo-
The reassignment phase of the solver algorithm attempts ration of a larger fraction of the array design search
to improve on the solution found in the initial phase. The space.
solver randomly selects a complete LU from the existing
set, removes all the stores from it, and reassigns them, Table 1 contrasts the four families of RAID level selection
just as in the first phase. It repeats this process until ev- methods we studied.
ery single LUs has been reassigned a few times (a con-
figurable parameter that we set to 3). The reassignment We now turn to a detailed description of these ap-
phase is designed to help the solver avoid local minima proaches.
in the optimization search space. This phase produces a
near-optimal assignment of stores to LUs. For more de- 3 Tagging schemes
tails on the optimality of the assignments and on the op-
eration of the solver, we refer the interested reader to [5]. Tagging is the process of determining, for each store in
isolation, the appropriate RAID level for it. The solver
2.1 Approaches to RAID level selection must later assign that store to an LU with the required
RAID level. The tagger operates exactly once on each
We explore two main approaches to selecting a RAID
store in the input workload description, and its decisions
level:
are final. We followed this approach in previous work
[1] because the decomposition into two separate stages
1. Tagging approaches: These approaches perform a is natural, is easy to understand, and limits the search
pre-processing step to tag stores with RAID levels space that must be explored when designing the rest of
before the solver is invoked. Once tagged with a the storage system.
RAID level, a store cannot change its tag, and it must
We explore two types of taggers: one type based on rules
be assigned to an LU of that type. Tagging decisions of thumb and the other based on performance models.
consider each store and its streams in isolation. We
consider two types of taggers: rule-based, which 3.1 Rule-based taggers
examine the size and type of I/Os; and model-based,
which use performance models to make their deci- These taggers make their decisions using rules based on
sions. The former tend to have many ad hoc pa- the size and type of I/Os performed by the streams. This
rameter settings; the latter have fewer, but also need is the approach implied by the original RAID paper [20],
performance-related data for a particular disk array which stated, for example, that RAID 5 is bad for “small”
 Approach Goal functions Solver Summary
Rule-based tagging no change simple many constants, variable results
Model-based tagging no change simple fewer constants, variable results
Partially-adaptive solver special for initial assignment simple good results, limited flexibility
Fully-adaptive solver no change complex good results, flexible but slower

Table 1: The four families of RAID-level selection methods studied in this paper. The two tagging families use either rule-based
or model-based taggers. The model-based taggers use parameters appropriate for the array being configured. The fully adaptive
family uses a substantially more complex solver than the other families. The Goal functions column indicates whether the same
goal functions are used in both solver phases: initial assignment and reassignment. The Summary column provides an evaluation
of their relative strengths and weaknesses.

writes, but good for “big” sequential writes. This ap- required additional constants, with units of bytes-per-
proach leads to a large collection of device-specific con- second/GB and seeks-per-second/GB; these values had to
stants, such as the number of seeks per second a de- be computed independently for each array. (Also, it is
vice can perform, and device-specific thresholds, such as unclear what to do if an array can support different disk
where exactly to draw the line between a “mostly-read” types with different capacity/performance ratios.)
and a “mostly-write” workload. These thresholds could, We also evaluated each of these taggers without the
in principle, be workload-independent, but in practice, capacity-bound rule. These variations are shown in the
we found it necessary to tune them experimentally to our graphs in Section 5 by appending Simple to each of the
test workloads and arrays, which means that there is no tagger names.
guarantee they will work as well on any other problem.
The rules we explored were the following. The first three 3.2 Model-based taggers
taggers help provide a measure of the cost of the laissez- The second type of tagging methods we studied used
faire approaches. The remaining ones attempt to specify array-type-specific performance models to estimate the
concrete values for the rules of thumb proposed in [20]. effect of assigning a store to an LU, and made a selection
based on that result.
1. random: pick a RAID level at random.
The first set of this type use simple performance mod-
2. allR10: tag all stores RAID 1/0. els that predict the number of back-end I/Os per sec-
ond (IOPS) that will result from the store being tagged
3. allR5: tag all stores RAID 5. at each available RAID level, and then pick the RAID
4. R5BigWrite: tag a store RAID 1/0 unless it has level that minimizes that number. This removes some ad
“mostly” writes (the threshold we used was at least hoc thresholds such as the size of a “big” write, but still
2/3 of the I/Os), and the writes are also “big” requires array-specific constants to compute the IOPS
(greater than 200 KB, after merging sequential I/O estimates. These taggers still need the addition of the
requests together). capacity-bound rule to get decent results. The IOPS-
based taggers we study are:
5. R5BigWriteOnly: tag a store RAID 1/0 unless it has
“big” writes, as defined above. 8. IOPS: tag a store RAID 1/0 if the estimated IOPS
would be smaller on a RAID 1/0 than on a RAID 5
6. R10SmallWrite: tag a store RAID 5 unless it has
LU. Otherwise tag it as RAID 5.
“mostly” writes and the writes are “small” (i.e., not
“big”). 9. IOPS-disk: as IOPS except the IOPS estimates are
divided by the number of disks in the LU, resulting
7. R10SmallWriteAggressive: as R10SmallWrite, but
in a per-disk IOPS measure, rather than a per-LU
with the threshold for number of writes set to 1/10
measure. The intent is to reflect the potentially dif-
of the I/Os rather than 2/3.
ferent number of disks in RAID 1/0 and RAID 5 LUs.
In practice, we found these rules needed to be aug- 10. IOPS-capacity: as IOPS except the IOPS estimates
mented with an additional rule to determine if a store was are multiplied by the ratio of raw (unprotected) ca-
capacity-bound (i.e., if space, rather than performance, pacity divided by effective capacity. This measure
was likely to be the bottleneck resource). A capacity- factors in the extra capacity cost associated with
bound store was always tagged as RAID 5. This rule RAID 1/0.
The second set of model-based taggers use the same per- the first store is assigned to it. Adding another goal
formance models that needed to be constructed and cal- function to the solver proved easy, so we tried several
ibrated for the solver anyway, and does not depend on in a search for one that worked well. We refer to the
any ad hoc constants. These taggers use the models to second option as fully-adaptive because the RAID level
compute, for each available RAID level, the percentage of the store and the LUs can be changed at almost any
changes in the LU’s utilization and capacity that will re- time. It is more flexible than the partially-adaptive one,
sult from choosing that level, under the simplifying as- but required more extensive modifications to the solver’s
sumption that the LU is dedicated solely to the store being search algorithm.
tagged. We then form a 2-dimensional vector from these
two results, and then pick the RAID level that minimizes: 4.1 Partially-adaptive schemes

11. PerfVectLength: the length (L2 norm) of the vector; The partially-adaptive approach works around the prob-
lem of the solver always choosing the cheaper, RAID 1/0
12. PerfVectAvg: the average magnitude (L1 norm) of LUs, by ignoring cost considerations in the initial selec-
the components; tion – thereby avoiding local cost-derived minima – and
the maximum component (L1
reintroducing cost in the reassignment stage. By allow-
13. PerfVectMax:
ing more LUs with more-costly RAID levels, the reas-
norm);
signment phase would have a larger search space to work
14. UtilizationOnly: just the utilization component, ig- within, thereby producing a better overall result.
noring capacity. Even in this scheme, the solver still needs to decide
whether a newly-created LU should be labeled as RAID 5
4 Solver-based schemes or RAID 1/0 during the initial assignment pass. It does
this by means of a goal function. The goal function can
When we first tried using the solver to make all RAID- take as input the performance, capacity, and utilization
level decisions, we discovered it worked poorly for two metrics for all the array components that would be in-
related reasons: volved in processing accesses to the store being placed
into the new LU. We devised a large number of possible
1. The solver’s goal functions were cost-based, and us- initial goal functions, based on the combinations of these
ing an existing LU is always cheaper than allocating metrics that seemed reasonable. While it is possible that
a new one. there are other, better initial goal functions, we believe
2. The solver chooses a RAID level for a new LU we have good coverage of the possibilities. Here is the
when it places the first store onto it – and a 2- set we explored:
disk RAID 1/0 LU is always cheaper than a 3- or
more-disk RAID 5 LU. As a result, the solver 1. allR10: always use RAID 1/0.
would choose a RAID 1/0 LU, fill it up, and then
repeat this process, even though the resulting sys- 2. allR5: always use RAID 5.
tem would cost more because of the additional disk
space needed for redundancy in RAID 1/0. (Our 3. AvgOfCapUtil: minimize the average of capacities
tests on the FC-60 array (described in Section 5.2) and utilizations of all the disks (the L1 norm).
did not have this discrepancy because we arranged
for the RAID 1/0 and RAID 5 LUs to contain six disks 4. LengthOfCapUtil: minimize the sum of the squares
each, to take best advantage of the array’s internal of capacities and utilizations (the L2 norm) of all
bus structure.) the disks.

We explored two options for addressing these difficulties. 5. MaxOfCapUtil: minimize the maximum of capaci-
First, we used a number of different initial goal functions ties and utilizations of all the disks (the L1 norm).
that ignored cost, in the hope that this would give the
6. MinAvgUtil: minimize the average utilizations of all
reassignment phase a better starting point. Second, we
the array components (disks, controllers and inter-
extended the solver to allow it to change the RAID level
nal buses).
of an LU even after stores had been assigned to it.
We refer to the first option as partially-adaptive, because 7. MaxAvgUtil: maximize the average utilizations of
it can change the RAID level associated with an indi- all the array components (disks, controllers and in-
vidual store—but it still fixes an LU’s RAID level when ternal buses).


8. MinAvg Util: minimize the arithmetic mean of the To address these concerns, we extended the search al-
change in utilizations of all the array components gorithm to let it dynamically switch the RAID level of a
(disks, controllers and internal buses). given LU. Every time the solver considers assigning a


9. MinAvg UtilPerRAIDdisk: as with scheme (8), but
store to an LU (that may already have stores assigned to
it), it evaluates whether the resulting LU would be better
first divide the result by the number of physical
off with a RAID 1/0 or RAID 5 layout.
disks used in the LU.
The primary cost of the fully-adaptive approach is that


10. MinAvg UtilPerDATAdisk: as with scheme (8), but it requires more CPU time than the partially-adaptive ap-
first divide the result by the number of data disks proach, which did not revisit RAID-level selection deci-
used in the LU. sions. In particular, the fully-adaptive approach roughly


11. MinAvg UtilTimesRAIDdisks: as with scheme (8), doubles the number of performance-model evaluations,
which are relatively expensive operations. But fully-
but first multiply the result by the number of physi-
cal disks used in the LU. adaptive approach has several advantages: the solver is
no longer biased towards a given RAID level, because it


12. MinAvg UtilTimesDATAdisks: as with scheme (8), can identify the best choice at all stages of the assignment
but first multiply the result by the number of data process. Adding more RAID levels to choose from is
disks used in the LU. also possible, although the total computation time grows
roughly linearly with the number of RAID levels. And
The intent of the various disk-scaling schemes (9–12) there no longer is a need for a special goal function dur-
was to explore ways of incorporating the size of an LU ing the initial assignment phase.
into the goal function.
Our experiments showed that, with two exceptions, the
Goal functions for the reassignment phase make minimal PriceThenMinAvgUtil and PriceThenMaxAvgUtil goal
system cost the primary decision metric, while selecting functions produced identical results for all the fully-
the right kind of RAID level is used as a tie-breaker. As a adaptive schemes. Each was better for one particular
result, there are fewer interesting choices of goal function workload; we selected PriceThenMaxAvgUtil for our ex-
during this phase, and we used just two: periments, as it resulted in the lowest average cost. We
found that it was possible to improve the fully-adaptive
1. PriceThenMinAvgUtil: lowest cost, ties resolved results slightly (so that they always produced the lowest
using scheme (6). cost) by increasing the number of reassignment passes to
5, but we did not do so to keep the comparison with the
2. PriceThenMaxAvgUtil: lowest cost, ties resolved
partially-adaptive solver as fair as possible.
using scheme (7).

During our evaluation, we tested each of the reassign- 5 Evaluation

ment goal functions in combination with all the initial-
assignment goal functions listed above. In this section, we present an experimental evaluation of
4.2 Fully-adaptive approach the effectiveness of the RAID level selection schemes dis-
cussed above.
As we evaluated the partly-adaptive approach, we found We took workload specifications from [1] and from
several drawbacks that led us to try the more flexible, traces of a validated TPC-D configuration. We used the
fully-adaptive approach: Ergastulum solver to design storage systems to support
 After the goal functions had become cost-sensitive
these workloads, ensuring for each design that the per-
formance and capacity needs of the workload would be
in the reassignment phase, new RAID 5 LUs would met. To see if the results were array-specific, we con-
not be created. Solutions would suffer if there were structed designs for two different disk array types.
too few RAID 5 LUs after initial assignment.
The primary evaluation criterion for the RAID-level se-
 It was not clear how well the approach would extend lection schemes was the cost of the generated configura-
to more than two RAID levels. tions, because our performance models [3, 18] predicted
 Although we were able to achieve good results with that all the generated solutions would support the work-
load performance requirements. The secondary criterion
the partially-adaptive approach, the reasons for the
results were not always obvious, hinting at a possi- was the CPU time taken by each approach.
ble lack of robustness. We chose not to run the workloads on the target phys-
 Workload Capacity #stores #streams Access size Run count %reads
filesystem 0.09 TB 140 140 20.0 KB (13.8) 2.6 (1.3) 64.2%
scientific 0.19 TB 100 200 640.0 KB (385.0) 93.5 (56.6) 20.0%
oltp 0.19 TB 194 182 2.0 KB (0.0) 1.0 (0.0) 66.0%
fs-light 0.16 TB 170 170 14.8 KB (7.3) 2.1 (0.7) 64.1%
tpcd30 0.05 TB 316 224 27.6 KB (19.3) 57.7 (124.8) 98.0%
tpcd30-2x 0.10 TB 632 448 27.6 KB (19.3) 57.7 (124.8) 98.0%
tpcd30-4x 0.20 TB 1264 896 27.6 KB (19.3) 57.7 (124.8) 98.0%
tpcd300-1 1.95 TB 911 144 53.5 KB (12.8) 1.13 (0.1) 98.3%
tpcd300-5 1.95 TB 935 374 49.1 KB (10.6) 1.23 (1.9) 92.7%
tpcd300-7 1.95 TB 941 304 51.1 KB (10.7) 1.12 (0.1) 95.0%
tpcd300-9 1.95 TB 933 399 49.8 KB (10.6) 1.20 (1.9) 85.6%
tpcd300-10 1.95 TB 910 321 45.3 KB (12.3) 1.28 (2.2) 80.3%

Table 2: Characteristics of workloads used in experiments. “Run count” is the mean number of consecutive sequential accesses
made by a stream. Thus workloads with low run counts (filesystem, oltp, fs-light) have essentially random accesses, while workloads
with high run counts (scientific) have sequential accesses. tpcd has both streams with random and sequential accesses. The access
size and run count columns list the mean and (standard deviation) for these values across all streams in the workload.

ical arrays because it was not feasible. First, we did 5.2 Disk arrays
not have access to the applications used for some of the
workloads—just traces of them running. Second, there
were too many of them. We evaluated over a thou-
sand configurations for the results presented; many of
the workloads run for hours. Third, some of the resulting We performed experiments using two of the arrays sup-
configurations were too large for us to construct. Fortu- ported by our solver: the Hewlett-Packard SureStore
nately, previous work [1] with the performance models Model 30/FC High Availability Disk Array (FC-30, [12])
we use indicated that their performance predictions are and the Hewlett-Packard SureStore E Disk Array FC-60
sufficiently accurate to allow us to feel confident that our (FC-60, [13]), as these are the ones for which we have
comparisons were fair, and that the configurations de- calibrated models.
signed would indeed support the workloads. The FC-30 is characteristic of a low-end, stand-alone
disk array of 3–4 years ago. An FC-30 has up to 30 disks
of 4 GB each, two redundant controllers (to survive a
5.1 Workloads controller failure) and 60 MB of battery-backed cache
(NVRAM). Each of the two array controllers is connected
to the client host(s) over a 1 Gb/s FibreChannel network.
To evaluate the RAID-level selection schemes, we used Our FC-30 performance models [18] have an average er-
a number of different workloads that represented both ror of 6% and a worst-case error of 20% over a rea-
traces of real systems and models of a diverse set of sonable range of LU sizes.
applications: an active file system (filesystem), a scien- The FC-60 is characteristic of modern mid-range arrays.
tific application (scientific), an on-line transaction pro- An FC-60 array can have up to 60 disks, placed in up to
cessing benchmark (oltp), a lightly-loaded filesystem (fs- six disk enclosures. Each of the two array controllers is
light), a 30 GB TPC-D decision-support benchmark, run- connected to the client host(s) over a 1 Gb/s FibreChan-
ning three queries in parallel until all of them com- nel network. Each controller may have up to 512 MB
plete (tpcd30), the tpcd30 workload duplicated (as if they of NVRAM. The controller enclosure contains a back-
were independent, but simultaneous runs) 2 and 4 times plane bus that connects the controllers to the disk enclo-
(tpcd30-2x and tpcd30-4x), and the most I/O-intensive sures, via six 40 MB/s ultra-wide SCSI busses. Disks of
queries (i.e., 1, 5, 7, 9 and 10) of the 300 GB TPC-D up to 72 GB can be used, for a total unprotected capac-
benchmark run one at a time on a validated configuration ity of 4.3 TB. Dirty blocks are mirrored in both con-
(tpcd300-query-N). troller caches, to prevent data loss if a controller fails.
Table 2 summarizes their performance characteristics. Our interpolation-based FC-60 performance models [3]
Detailed information on the derivations of these work- have an average error of about 10% over a fairly wide
loads can be found in [1]. range of configurations.
 1.8 2.5
maximum maximum
average average
1.6 200% 200%
150% 150%
110% 2 110%
1.4 100% 100%

1.2
1.5
1

0.8
1
0.6

0.4
0.5

0.2

0 0
allR10
allR5
random
IOPS
IOPScapacity
IOPSdisk
PerfVectAvg
PerfVectLength
PerfVectMax
UtilizationOnly
R10SmallWrite
R10SmallWriteSimple
R10SmallWriteAggressive
R10SmallWriteAggressiveSimple
R5BigWrite
R5BigWriteSimple
R5BigWriteOnly
R5BigWriteOnlySimple

allR10
allR5
random
IOPS
IOPScapacity
IOPSdisk
PerfVectAvg
PerfVectLength
PerfVectMax
UtilizationOnly
R10SmallWrite
R10SmallWriteSimple
R10SmallWriteAggressive
R10SmallWriteAggressiveSimple
R5BigWrite
R5BigWriteSimple
R5BigWriteOnly
R5BigWriteOnlySimple
(a) FC-30 array (b) FC-60 array

Figure 2: Tagger results for the FC-30 and FC-60 disk arrays. The results for each tagger are plotted within a single bar of the
graph. Over all workloads, the bars show the proportion of time each tagger resulted in a final solution with the lowest cost (as
measured over all varieties of RAID level selection), within 110% of the lowest cost, within 150% of the lowest cost and within
200% of the cost. The taller and darker the bar, the better the tagger. Above each bar, the points show the maximum (worst) and
average results for the tagger, as a multiple of the best cost. The allR10 and allR5 taggers tag all stores as RAID 1/0 or RAID 5
respectively. The random tagger allocates stores randomly to either RAID level. The IOPS models are based on very simple array
models. The PerfVect... and UtilizationOnly taggers are based on the complete analytical models as used by the solver. The
remaining taggers are rule-based.

5.3 Comparisons is even less clear—the sets of best taggers for each ar-
ray are completely disjoint. Hence, the optimal choice
As described above, the primary criteria for comparison of RAID level varies widely from array to array, and no
for all schemes is that of total system cost. single set of rules seems to work well for all array types,
even when a subset of all array-specific parameters (such
as the test for capacity-boundedness) is used in addition.
5.3.1 Tagger results
Second, the results for the FC-60 are, in general, worse
than for the FC-30. In large part, this is due to the rela-
Figure 2 shows the results for each of the taggers for the
tive size and costs of the arrays. Many of the workloads
FC-30 and FC-60 arrays. There are several observations
require a large number (more than 20) of the FC-30 ar-
and conclusions we can draw from these results.
rays; less efficient solutions—even those that require a
First, there is no overall winner. Within each array type, few more complete arrays—add only a small relative in-
it is difficult to determine what the optimal choice is. For crement to the total price. Conversely, the same work-
instance, compare the PerfVectMax and IOPS taggers for loads for the FC-60 only require 2–3 arrays, and the rela-
the FC-30 array. IOPS has a better average result than tive cost impact of a solution requiring even a single extra
PerfVectMax, but performs very badly on one workload array is considerable. Another reason for the increased
(filesystem), whereas PerfVectMax is much better in the FC-60 costs is that many of the taggers were hand-tuned
worst case. Depending on the user’s expected range of for the FC-30 in an earlier series of experiments [1].
workloads, either one may be the right choice.
With a different array, which has very different perfor-
When comparing results across array types, the situation
 1.8 2.2
maximum maximum
average average
1.6 200% 2 200%
150% 150%
110% 110%
1.4 100% 1.8 100%

1.2 1.6

1 1.4

0.8 1.2

0.6 1

0.4 0.8

0.2 0.6

0 0.4
allR10

allR5

AvgOfCapUtil

LengthOfCapUtil

MaxOfCapUtil

MinAvgDeltaUtil

MinAvgDeltaUtilPerDataDisk

MinAvgDeltaUtilPerLUNdisk

MinAvgDeltaUtilTimesDataDisks

MinAvgDeltaUtilTimesLUNdisks

MinimumAverageUtil

allR10

allR5

AvgOfCapUtil

LengthOfCapUtil

MaxOfCapUtil

MinAvgDeltaUtil

MinAvgDeltaUtilPerDataDisk

MinAvgDeltaUtilPerLUNdisk

MinAvgDeltaUtilTimesDataDisks

MinAvgDeltaUtilTimesLUNdisks

MinimumAverageUtil
(a) FC-30 array (b) FC-60 array

Figure 3: Partially-adaptive results for the FC-30 and FC-60 disk arrays. There are two bars for each initial assignment goal
function: the one on the left uses the PriceThenMinAvgUtil reassignment goal function, the one on the right PriceThenMaxAvgUtil.

mance characteristics, the decisions as to what consti- adaptive rule for the FC-30 but not for the FC-60. How-
tutes a “large write” become invalid. For example, con- ever the rules based on norms (AvgOfCapUtil, MaxOf-
sider the stripe size setting for each of the arrays we used. CapUtil and LengthOfCapUtil) seem to perform fairly
The FC-60 uses a default of 16 KB, whereas the FC-30 well for both arrays—an improvement over the tagging
uses 64 KB, which results in different performance for schemes. The family of partially-adaptive rules based on
I/O sizes between these values. change in utilization seems to perform reasonably for the
FC-30, but poorly for the FC-60—with one exception,
Third, even taggers based completely on the solver mod-
els perform no better, and sometimes worse, than taggers

MinAvg UtilTimesDataDisks, that performed as well as
based only on simple rules. This indicates that tagging the norm-rules.
solutions are too simplistic; it is necessary to take into
account the interactions between different streams and
stores mapped to the same LU or array when selecting
RAID levels. This can be done through the use of adap-
tive algorithms, as shown in the following sections.
5.3.3 Fully-adaptive results

5.3.2 Partially-adaptive results

Tables 3 and 4 show, for each workload, the best re-
sults achieved for each family of RAID level selection
Figure 3 shows results for each of the partially-adaptive methods. As can be seen, the fully-adaptive approach
rules for the FC-30 and FC-60 arrays. Our results show finds the best solution in all but one case, indicating that
that the partly adaptive solver does much better than the this technique better searches the solution space than the
tagging approaches. In particular, minimizing the aver- partly adaptive and tagging techniques. Although the
age capacity and utilization works well for both arrays fully-adaptive approach needs more modifications to the
and all the workloads. solver, a single goal function performs nearly perfectly
From the data, it is clear that allR5 is the best partially- on both arrays, and it is more flexible.
 Taggers Partly adaptive Fully
Workload PerfVectMax IOPSdisk R10SmallWriteAggressive AllR5 AvgOfCapUtil adaptive
filesystem 1% 1% 1% 0% 0% 0%
filesystem-lite 1% 1% 3% 0% 0% 0%
oltp 1% 1% 1% 0% 0% 0%
scientific 2% 2% 2% 0% 2% 0%
tpcd30-1x 7% 8% 8% 0% 0% 0%
tpcd30-2x 4% 2% 2% 2% 2% 0%
tpcd30-4x 1% 44% 44% 2% 2% 2%
tpcd300-query-1 0% 0% 0% 0% 0% 0%
tpcd300-query-5 18% 4% 4% 0% 0% 0%
tpcd300-query-7 25% 0% 0% 0% 4% 0%
tpcd300-query-9 22% 4% 8% 0% 4% 0%
tpcd300-query-10 4% 8% 8% 0% 0% 0%
average 7.2% 6.3% 6.8% 0.3% 1.2% 0.12%

Table 3: Cost overruns for the best solution for each workload and RAID selection method for the FC-30 array. Values are in percent
above the best cost over all results for that array—that is, if the best possible result cost $100, and the given method resulted in a
system costing $115, then the cost overrun is 15%. Increasing the number of reassignment passes to 5 results in the fully-adaptive
scheme being best in all cases; we do not report those numbers to present a fair comparison with the other schemes.

Taggers Partly adaptive Fully

Workload FC60UtilizationOnly IOPScapacity allR10 AvgOfCapUtil MaxOfCapUtil adaptive
filesystem 0% 44% 0% 0% 0% 0%
filesystem-lite 24% 0% 24% 0% 0% 0%
oltp 0% 2% 0% 0% 0% 0%
scientific 0% 108% 0% 0% 0% 0%
tpcd30-1x 12% 12% 0% 0% 0% 0%
tpcd30-2x 9% 9% 0% 0% 0% 0%
tpcd30-4x 7% 7% 0% 7% 0% 0%
tpcd300-query-1 7% 2% 50% 5% 0% 0%
tpcd300-query-5 61% 2% 37% 0% 9% 0%
tpcd300-query-7 32% 1% 0% 0% 20% 0%
tpcd300-query-9 36% 2% 10% 0% 0% 0%
tpcd300-query-10 12% 2% 50% 9% 0% 0%
average 16.7% 15.9% 14.3% 1.75% 2.4% 0%

Table 4: Cost overruns for the best solution for each workload and RAID selection method for the FC-60 array. All values are
percentages above the best cost seen across all the methods.
5.3.4 CPU time comparison given access patterns.
The HP AutoRAID disk array [24] side-steps the issue
The advantage of better solutions does not come without by dynamically, and transparently, migrating data blocks
a cost: Table 5 shows that the CPU time to calculate a between RAID 1/0 and RAID 5 storage as a result of data
solution increases for the more complex algorithms, be- access patterns. However, the AutoRAID technology is
cause they explore a larger portion of the search space. not yet widespread, and even its remapping algorithms
In particular, tagging eliminates the need to search any are themselves based on simple rules of thumb that could
solution that uses an LU with a different tag, and makes perhaps be improved (e.g., “put as much recently written
selection of a new LU’s type trivial when it is created, data in RAID 1/0 as possible”).
whereas both of the adaptive algorithms have to perform
a model evaluation and a search over all of the LU types. In addition to RAID levels, storage systems have multiple
other parameters that system administrators are expected
The fully-adaptive algorithm searches all the possibili- to set. Prior studies examined how to choose the num-
ties that the partially-adaptive algorithm does, and also ber of disks per LU [22], and the optimal stripe unit size
looks at the potential benefit of switching the LU type for RAID 0 [9], RAID 5 [8], and other layouts [19]. The
on each assignment. It takes considerably longer to run. RAID Configuration Tool [27] allows system administra-
Even so, this factor is insignificant when put into con- tors to run simple, synthetic variations on a user-supplied
text: our solver has completely designed enterprise stor- I/O trace against a simulator, to help visualize the perfor-
age systems containing $2–$5 million of storage equip- mance consequences of each parameter setting (includ-
ment in under an hour of CPU time. We believe that the ing RAID levels). Although it assists humans in explor-
advantages of the fully-adaptive solution will outweigh ing the search space by hand, it does not automatically
its computation costs in almost all cases. search the parameter space itself.
Apart from the HP AutoRAID, none of these systems
5.3.5 Implementation complexity provide much, if any, assistance with mixed workloads.
The work described here is part of a larger research pro-
A final tradeoff that might be considered is the imple- gram at HP Laboratories with the goal of automating the
mentation complexity. The modifications to implement design, construction, and management of storage sys-
partially-adaptive schemes on the original solver took a tems. In the scheme we have developed for this, we run
few hours of work. The fully-adaptive approach took a our solver to develop a design for a storage system, then
few weeks of work. Both figures are for a person thor- implement that design, monitor it under load, analyze the
oughly familiar with the solver code. However, the fully- result, and then re-design the storage system if neces-
adaptive approach clearly gives the best results, and is in- sary, to meet changes in workload, available resources,
dependent of the devices and workloads being used; the or even simple mis-estimates of the original requirements
development investment is likely to pay off very quickly [4]. Our goal is to do this with no manual intervention at
in any production environment. all – we would like the storage system to be completely
self-managing. An important part of the solution is the
6 Related work ability to design configurations and data layouts for disk
arrays automatically, which is where the work described
The published literature does not seem to report on sys- in this paper contributes.
tematic, implementable criteria for automatic RAID level
selection. In their original paper [20], Patterson, Gib- 7 Summary and conclusions
son and Katz mention some selection criteria for RAID 1
through RAID 5, based on the sizes of read and write In this paper, we presented a variety of methods for se-
accesses. Their criteria are high-level rules of thumb lecting RAID levels, running the gamut from the ones
that apply to extreme cases, e.g., “if a workload contains that consider each store in isolation and make irrevocable
mostly small writes, use RAID 1/0 instead of RAID 5”. decisions to the ones that consider all workload interac-
No attempt is made to resolve contradictory recommen- tions and can undo any decision. We then evaluated all
dations from different rules, or to determine thresh- schemes for each family in isolation, and then compared
old values for essential definitions like “small write” the cost of solutions for the best representative from each
or “write-mostly”. Simulation-based studies [2, 14, 17] family. A set of real workload descriptions and models
quantify the relative strengths of different RAID levels of commercially-available disk arrays was used for the
(including some not mentioned in this paper), but do not performance study. To the best of our knowledge, this is
derive general guidelines for choosing a RAID level for the first systematic, automatable attempt to select RAID
 Workload Taggers Partly adaptive Fully adaptive
filesystem 92 (14) 131 (42) 273 (53)
filesystem-lite 51 (3) 85 (33) 232 (28)
oltp 212 (29) 279 (46) 669 (155)
scientific 66 (5) 116 (49) 277 (55)
tpcd30-1x 44 (10) 85 (23) 782 (197)
tpcd30-2x 265 (49) 393 (92) 3980 (1414)
tpcd30-4x 1098 (159) 2041 (739) 24011 (7842)
tpcd300-query-1 689 (44) 1751 (2719) 1541 (300)
tpcd300-query-5 1517 (85) 2907 (3593) 4572 (1097)
tpcd300-query-7 1556 (90) 2401 (2126) 5836 (1345)
tpcd300-query-9 1680 (73) 2693 (2362) 6647 (2012)
tpcd300-query-10 1127 (77) 2144 (1746) 2852 (563)
mean 700 (633) 1252 (2016) 4306 (6781)

Table 5: Mean and (standard deviation) of the CPU time in seconds, for each workload and RAID selection method for the FC-60
array.

levels in the published literature. Acknowledgements: We thank Arif Merchant, Susan

The simpler tagging schemes are similar to accepted Spence and Mustafa Uysal for their comments on earlier
knowledge and to the back-of-the-envelope calculations drafts of the paper.
that system designers currently rely upon. However, they
are highly dependent on particular combinations of de- References
vices and workloads, and involve hand-picking the right
values for many constants, so they are only suitable for [1] G. A. Alvarez, E. Borowsky, S. Go, T. H.
limited combinations of workloads and devices. Further- Romer, R. Becker-Szendy, R. Golding, A. Mer-
more, because they put restrictions on the choices the chant, M. Spasojevic, A. Veitch, and J. Wilkes.
solver can make, they result in poorer solutions. Minerva: an automated resource provisioning tool
for large-scale storage systems. ACM Transactions
Integrating RAID level selection into the store-to-device on Computer Systems, 19(4), November 2001.
assignment algorithm led to much better results, with the
best results being obtained from allowing the solver to [2] G. A. Alvarez, W. Burkhard, L. Stockmeyer, and
revise its RAID-level selection decision at any time. F. Cristian. Declustered disk array architectures
We showed that the benefits of the fully-adaptive scheme with optimal and near-optimal parallelism. In Pro-
outweigh its additional costs in terms of computation ceedings of International Symposium on Computer
time and complexity. Analysis of the utilization data Architecture (ISCA), pages 109–20, June 1998.
from the fully-adaptive solver solutions showed that [3] E. Anderson. Simple table-based modeling of
some of the solutions it generated in our experiments storage devices. Technical report HPL–SSP–
were provably of the lowest possible cost (e.g., when the 2001–4, Hewlett-Packard Laboratories, July 2001.
capacity of every disk, or the bandwidth of all but one [Link]
array, were fully utilized).
[4] E. Anderson, M. Hobbs, K. Keeton, S. Spence,
For future work, we would like to explore the implica-
M. Uysal, and A. Veitch. Hippodrome: running
tions of providing reliability guarantees in addition to
circles around storage administration. In File and
performance; we believe that the fully-adaptive schemes
Storage Technologies Conference (FAST), Mon-
would be suitable for this, at the cost of increased run-
terey, January 2002.
ning times. We would also like to automatically choose
components of different cost for each individual LU [5] E. Anderson, M. Kallahalla, S. Spence, R. Swami-
within the arrays, e.g., decide between big/slow and nathan, and Q. Wang. Ergastulum: An ap-
small/fast disk drives according to the workload being proach to solving the workload and device con-
mapped onto them; and to extend automatic decisions to figuration problem. Technical report HPL–SSP–
additional parameters such as LU stripe size and disks 2001–5, Hewlett-Packard Laboratories, June 2001.
used in an LU. [Link]
 [6] M. Blaum, J. Brady, J. Bruck, and J. Menon. Even- 4th Conference on Architectural Support for Pro-
odd – an efficient scheme for tolerating double-disk gramming Languages and Operating Systems (AS-
failures in RAID architectures. IEEE Transactions PLOS), pages 190–199, Santa Clara, CA, April
on Computers, 44(2):192–202, February 1995. 1991.

[7] E. Borowsky, R. Golding, A. Merchant, L. Schreier, [18] A. Merchant and G. A. Alvarez. Disk array mod-
E. Shriver, M. Spasojevic, and J. Wilkes. Us- els in Minerva. Technical Report HPL–2001–
ing attribute-managed storage to achieve QoS. In 118, Hewlett-Packard Laboratories, April 2001.
5th International Workshop on Quality of Service, [Link]
Columbia university, new York, NY, June 1997.
[19] C.-I. Park and T.-Y. Choe. Striping in disk array
[8] P. Chen and E. Lee. Striping in a RAID level 5 disk RM2 enabling the tolerance of double disk failures.
array. In International Conference on Measurement In Supercomputing, November 1996.
and Modeling of Computer Systems (ACM SIG- [20] D.A. Patterson, G.A. Gibson, and R.H. Katz. A
METRICS), pages 136–145, May 1995. case for redundant arrays of inexpensive disks
[9] P. Chen and D. Patterson. Maximizing performance (RAID). In SIGMOD international Conference on
in a striped disk array. In International Symposium the Management of Data, pages 109–116, Chicago,
on Computer Architecture (ISCA), pages 322–331, IL, 1988.
May 1990. [21] D. Stodolsky, M. Holland, W.V. Courtright II, and
G. Gibson. Parity-logging disk arrays. ACM Trans-
[10] W. Fernandez de la Vega and G. Lueker. Bin pack-
ing can be solved within 1+ in linear time. Com-
actions on Computer Systems, 12(3):206–35, Au-
gust 1994.
binatorica, 1(4):349–355, 1981.
[22] P. Triantafillou and C. Faloutsos. Overlay striping
[11] L. Hellerstein, G. Gibson, R. Karp, R. Katz, and optimal parallel I/O for modern applications.
and D. Patterson. Coding techniques for han- Parallel Computing, 24(1):21–43, January 1998.
dling failures in large disk arrays. Algorithmica,
12(2/3):182–208, 1994. [23] M. Uysal, G. A. Alvarez, and A. Merchant. A
modular, analytical throughput model for modern
[12] Hewlett-Packard Company. HP SureStore E disk arrays. In International Symposium on Mod-
Model 30/FC High Availability Disk Array—User’s eling, Analysis and Simulation on Computer and
Guide, August 1998. Publication A3661–90001. Telecommunications Systems (MASCOTS), August
2001.
[13] Hewlett-Packard Company. HP SureStore E Disk
Array FC60—User’s guide, December 2000. Pub- [24] J. Wilkes, R. Golding, C. Staelin, and T. Sulli-
lication A5277–90001. van. The HP AutoRAID hierarchical storage sys-
tem. ACM Transactions on Computer Systems,
[14] M. Holland and G. A. Gibson. Parity declustering 14(1):108–36, February 1996.
for continuous operation in redundant disk arrays.
In 5th Conference on Architectural Support for Pro- [25] John Wilkes. Traveling to Rome: QoS specifica-
gramming Languages and Operating Systems (AS- tions for automated storage system management.
PLOS), volume 20, pages 23–35, October 1992. In Proceedings of the International Workshop on
Quality of Service (IWQoS’2001), Karlsruhe, Ger-
[15] D. S. Johnson, A. Demers, J. D. Ullman, M. R. many, June 2001. Springer-Verlag.
Garey, and R. L. Graham. Worst-case performance
bounds for simple one-dimensional packing algo- [26] B. Worthington, G. Ganger, Y. Patt, and J. Wilkes.
rithms. SIAM Journal on Computing, 3(4):299– On-line extraction of SCSI disk drive parameters.
325, December 1974. In International Conference on Measurement and
Modeling of Computer Systems (ACM SIGMET-
[16] C. Kenyon. Best-fit bin-packing with random order. RICS), pages 146–56, May 1995.
In Symposium on Discrete Algorithms, pages 359–
364, January 1996. [27] P. Zabback, J. Riegel, and J. Menon. The
RAID configuration tool. Technical Report RJ
[17] E.K. Lee and R.H. Katz. Performance conse- 10055 (90552), IBM Research, November 1996.
quences of parity placement in disk arrays. In