Chapter 4

CHAPTER FOUR

Predicting Trends

I
’M ASSUMING YOU’VE MADE A FEW PASSES THROUGH CHAPTER 3 AND HAVE JUST DEPLOYED A SUPER-AWESOME,
totally amazing, monitoring, trending, graphing, and measurement system. You’re graph-
ing everything you can get your hands on, as often as you can. You probably didn’t gain
anything from graphing the peak barking periods of your neighbor’s dog—but hey, you
did it, and I’m proud of you.

Now you’ll be able to use this data (excluding the barking statistics) like a crystal ball, and
predict the future like Nostradamus. But let’s stop here for a moment to remember an irri-
tating little detail: it’s impossible to accurately predict the future.

Forecasting capacity needs is part intuition, and part math. It’s also the art of slicing and
dicing up your historical data, and making educated guesses about the future. Outside of
those rare bursts and spikes of load on your system, the long-term view is hopefully one of
steadily increasing usage. By putting all of this historical data into perspective, you can
generate estimates for what you’ll need to sustain the growth of your website. As we’ll see
later, the key to making accurate predictions is having an adjustable forecasting process.

63
 DON’T BUY BEFORE YOU NEED IT
Before you get too excited about charting massive growth and putting new servers in place to handle
the deluge, let me remind you of one of the key economic factors you need to deal with: buying equip-
ment too early is wasteful.

This rule is derived directly from the obvious trend in computing costs: all forms of hardware are
becoming cheaper, even as they become faster and more reliable. Whether or not Moore’s Law
(Gordon E. Moore’s now-famous axiom in 1965 that postulates the number of transistors on an inte-
grated circuit approximately doubles every eighteen months) holds true forever, we can predict that
manufacturers will continue to lower costs over time. If you can wait six months before buying a piece
of equipment, you will likely end up with faster and less expensive equipment at that time.

Certainly, you don’t want to be caught unprepared when growth takes place—this book is all about
saving you from that career-threatening situation. Conversely, the company financial officers will not
hold you in high regard either when you’ve purchased a lot of equipment that lay idle, only to see its
price drop a few months later.

Riding Your Waves

A good capacity plan depends on knowing your needs for your most important resources,
and how those needs change over time. Once you have gathered historical data on capac-
ity, you can begin analyzing it with an eye toward recognizing any trends and recurring
patterns.

For example, in the last chapter I recounted how at Flickr, we discovered Sunday has been
historically the highest photo upload day of the week. This is interesting for many reasons.
It may also lead us to other questions: has that Sunday peak changed over time, and if so,
how has it changed with respect to the other days of the week? Has the highest upload
day always been Sunday? Does that change as we add new members residing on the other
side of the International Date Line? Is Sunday still the highest upload day on holiday
weekends? These questions can all be answered once you have the data, and the answers
in turn could provide a wealth of insight with respect to planning new feature launches,
operational outages, or maintenance windows.

Recognizing trends is valuable for many reasons, not just for capacity planning. When we
looked at disk space consumption in Chapter 3, we stumbled upon some weekly upload
patterns. Being aware of any recurring patterns can be invaluable when making decisions
later on. Trends can also inform community management, customer care and support,
product management, and finance. Some examples of how metrics measurement can be
useful include:

64 CHAPTER FOUR
• Your operations group can avoid scheduling maintenance that could affect image pro-
cessing machines on a Sunday, opting for a Friday instead, to minimize any adverse
effects on users.
• If you deploy any new code that touches the upload processing infrastructure, you
might want to pay particular attention the following Sunday to see whether everything
is holding up well when the system experiences its highest load.
• Making customer support aware of these peak patterns allows them to gauge the effect
of any user feedback regarding uploads.
• Product management might want to launch new features based on the low or high
traffic periods of the day. A good practice is to make sure everyone on your team
knows where these metrics are located and what they mean.
• Your finance department might also want to know about these trends because it can
help them plan for capital expenditure costs.

Trends, Curves, and Time

Let’s take a look back at the daily storage consumption data we collected in the last chap-
ter and apply it to make a forecast of future storage needs. We already know the defining
metric: total available disk space. Graphing the cumulative total of this data provides the
right perspective from which to predict future needs. Taking a look at Figure 4-1, we can
see where we’re headed with consumption, how it’s changing over time, and when we’re
likely to run out of space.

F I G U R E 4 - 1 . Total disk consumption: cumulative view

Now, let’s add our constraint: the total currently available disk space. Let’s assume for this
example we have a total of 20 TB (or 20,480 GB) installed capacity. From the graph, we see
we’ve consumed about 16 TB. Adding a solid line extending into the future to represent

PREDICTING TRENDS 65
 the total space we have installed, we obtain a graph that looks like Figure 4-2. This illus-
tration demonstrates a fundamental principal of capacity planning: predictions require
two essential bits of information, your ceilings and your historical data.

F I G U R E 4 - 2 . Cumulative disk consumption and available space

Determining when we’re going to reach our space limitation is our next step. As I just sug-
gested, we could simply draw a straight line that extends from our measured data to the
point at which it intersects our current limit line. But is our growth actually linear? It may
not be.

Excel calls this next step “adding a trend line,” but some readers might know this process
as curve fitting. This is the process by which you attempt to find a mathematical equation
that mimics the data you’re looking at. You can then use that equation to make educated
guesses about missing values within the data. In this case, since our data is on a time line,
the missing values in which we’re interested are in the future. Finding a good equation to
fit the data can be just as much art as science. Fortunately, Excel is one of many programs
that feature curve fitting.

To display the trend using a more mathematical appearance, let’s change the Chart Type
in Excel from Line to XY (Scatter).

XY (Scatter) changes the date values to just single data points. We can then use the trend-
ing feature of Excel to show us how this trend looks at some point in the future. Right-
click the data on the graph to display a drop-down menu. From that menu, select Add
Trendline. A dialog box will open, as shown in Figure 4-3.

66 CHAPTER FOUR
F I G U R E 4 - 3 . Add Trendline Type dialog box

Next, select a trend line type. For the time being, let’s choose Polynomial, and set Order to
2. There may be good reasons to choose another trend type, depending on how variable
your data is, how much data you have, and how far into the future you want to extrapo-
late. For more information, see the upcoming sidebar, “Fitting Curves.”

In this example, the data appears about as linear as can be, but since I already know this
data isn’t linear over a longer period of time (it’s accelerating), I’ll pick a trend type that
can capture some of the acceleration we know will occur.

After selecting a trend type, click the Options tab to bring up the Add Trendline options
dialog box, as shown in Figure 4-4.

To show the equation that will be used to mimic our disk space data, click the checkbox
for “Display equation on chart.” We can also look at the R2 value for this equation by
clicking the “Display R-squared value on chart” checkbox.

The R2 value is known in the world of statistics as the coefficient of determination. Without
going into the details of how this is calculated, it’s basically an indicator of how well an
equation matches a certain set of data. An R2 value of 1 indicates a mathematically perfect
fit. With the data we’re using for this example, any value above 0.85 should be sufficient.
The important thing to know is, as your R2 value decreases, so too should your confidence
in the forecasts. Changing the trend type in the previous step affects the R2 values—some-
times for better, sometimes for worse—so some experimentation is needed here when
looking at different sets of data.

PREDICTING TRENDS 67
 FITTING CURVES
In capacity planning, curve fitting is where the creative can collide with the scientific. In most cases,
capacity planning is used to stay ahead of growth, which is generally represented through time-series
data that extends upward and to the right in some form or manner.

Figuring out how and when the data gets there is the challenge, and we aim to use extrapolation to
solve it. Extrapolation is the process of constructing new data points beyond a set of known data
points. In our case, we’re going to be defining new data points that exist in the future of time-series
data.

The difficulty with curve fitting and extrapolation is you need to reconcile what you know about the
source of your data with the apparent best-fit equation. Simply because you find a curve that fits the
data with 99.999% mathematical accuracy doesn’t mean it’s going to be an accurate picture of the
future. Your data will almost always have context outside of the mathematical equation. For example,
forecasting the sale of snow shovels must include considerations for time of year (winter versus sum-
mer) or geography (Alaska or Arizona).

When finding equations to fit your data, it’s also best to stay away from higher-order polynomial
equations. They’re tempting because their fit (coefficient of determination) is so good, but anything
higher than a 2nd order polynomial can exhibit dramatic fluctuations outside of the dataset you’re
looking at.

The moral of the story is to use a good deal of common sense when curve-fitting your data. Don’t insist
on elegantly perfect fits, as they are quite often the result of questionable assumptions.

We’ll want to extend our trend line into the future, of course. We want to extend it far
enough into the future such that it intersects the line corresponding to our total available
space. This is the point at which we can predict we’ll run out of space. Under the Forecast
portion of the dialog box, enter 25 units for a value. Our units in this case are days. After
you hit OK, you’ll see our forecast looks similar to Figure 4-5.

The graph indicates that somewhere around day 37, we run out of disk space. Luckily, we
don’t need to squint at the graph to see the actual values; we have the equation used to
plot that trend line. As detailed in Table 4-1, plugging the equation into Excel, and using
the day units for the values of X, we find the last day we’re below our disk space limit is
8/30/05.

68 CHAPTER FOUR
F I G U R E 4 - 4 . Add Trendline Options dialog box

F I G U R E 4 - 5 . Extending the trend line

PREDICTING TRENDS 69
 T A B L E 4 - 1 . Determining the precise day you will run out of disk space

y=0.7675x2 + 146.96x +
Date Disk available (GB) 14147
33 08/27/05 20480.00 19832.49
34 08/28/05 20480.00 20030.87
35 08/29/05 20480.00 20230.79
36 08/30/05 20480.00 20432.24
37 08/31/05 20480.00 20635.23
38 09/01/05 20480.00 20839.75
39 09/02/05 20480.00 21045.81

Now we know when we’ll need more disk space, and we can get on with ordering and
deploying it.

This example of increasing disk space is about as simple as they come. But as the metric is
consumption-driven, every day has a new value that contributes to the definition of our
curve. We also need to factor in the peak-driven metrics that drive our capacity needs in
other parts of our site. Peak-driven metrics involve resources that are continually regener-
ated, such as CPU time and network bandwidth. They fluctuate more dramatically and
thus are more difficult to predict, so curve fitting requires more care.

Tying Application Level Metrics to System Statistics: Database Example

In Chapter 3, we went through the exercise of establishing our database ceiling values. We
discovered (through observing our system metrics) that 40 percent disk I/O wait was a
critical value to avoid, because it’s the threshold at which database replication begins
experiencing disruptive lags.

How do we know when we’ll reach this threshold? We need some indication when we are
approaching our ceiling. It appears the graphs don’t show a clear and smooth line just
bumping over the 40 percent threshold. Instead, our disk I/O wait graph shows our data-
base doing fine until a 40 percent spike occurs. We might deem occasional (and recover-
able) spikes to be acceptable, but we need to track how our average values change over
time so the spikes aren’t so close to our ceiling. We also need to somehow tie I/O wait
times to our database usage, and ultimately, what that means in terms of actual applica-
tion usage.

To establish some control over this unruly data, let’s take a step back from the system sta-
tistics and look at the purpose this database is actually serving. In this example, we’re
looking at a user database. This is a server in our main database cluster, wherein a segment
of Flickr users store the metadata associated with their user account: their photos, their
tags, the groups they belong to, and more. The two main drivers of load on the databases
are, of course, the number of photos and the number of users.

This particular database has roughly 256,000 users and 23 million photos. Over time, we
realized that neither the number of users nor the number of photos is singularly responsible

70 CHAPTER FOUR
for how much work the database does. Taking only one of those variables into account
meant ignoring the effect of the other. Indeed, there may be many users who have few, or
no photos; queries for their data is quite fast and not at all taxing. On the flip side, there
are a handful of users who maintain enormous collections of photos.

We can look at our metrics for clues on our critical values. We have all our system metrics,
our application metrics, and the historical growth of each.

We then set out to find the single most important metric that can define the ceiling for
each database server. After looking at the disk I/O wait metric for each one, we were
unable to distinguish a good correlation between I/O wait and the number of users on the
database. We had some servers with over 450,000 users that were seeing healthy, but not
dangerous, levels of I/O wait. Meanwhile, other servers with only 300,000 users were
experiencing much higher levels of I/O wait. Looking at the number of photos wasn’t
helpful either—disk I/O wait didn’t appear to be tied to photo population.

As it turns out, the metric that directly indicates disk I/O wait is the ratio of photos-to-users
on each of the databases.

As part of our application-level dashboard, we measure on a daily basis (collected each

night) how many users are stored on each database along with the number of photos asso-
ciated with each user. The photos-to-user ratio is simply the total number of photos
divided by the number of users. While this could be thought of as an average photos per
user, the range can be quite large, with some “power” Flickr users having many thousands
of photos while a majority have only tens or hundreds. By looking at how the peak disk I/O
wait changes with respect to this photos per user ratio, we can get an idea of what sort of
application-level metrics can be used to predict and control the use of our capacity (see
Figure 4-6).

This graph was compiled from a number of our databases, and displays the peak disk I/O
wait values against their current photos-to-user ratios. With this graph, we can ascertain
where disk I/O wait begins to jump up. There’s an elbow in our data around the 85–90
ratio when the amount of disk I/O wait jumps above the 30 percent range. Since our ceiling
value is 40 percent, we’ll want to ensure we keep our photos-to-user ratio in the 80–100
range. We can control this ratio within our application by distributing photos for high-
volume users across many databases.

I want to stop here for a moment to talk a bit about Flickr’s database architecture. After
reaching the limits of the more traditional Master/Slaves MySQL replication architecture
(in which all writes go to the master and all reads go to the slaves), we redesigned our
database layout to be federated, or sharded. This evolution in architecture is becoming
increasingly common as site growth reaches higher levels of changing data. I won’t go into
how that architectural migration came about, but it’s a good example of how architecture
decisions can have a positive effect on capacity planning and deployment. By federating
our data across many servers, we limit our growth only by the amount of hardware we
can deploy, not by the limits imposed by any single machine.

PREDICTING TRENDS 71
 F I G U R E 4 - 6 . Database—photo:user ratio versus disk I/O wait percent

Because we’re federated, we can control how users (and their photos) are spread across
many databases. This essentially means each server (or pair of servers, for redundancy)
contains a unique set of data. This is in contrast to the more traditional monolithic data-
base that contains every record on a single server. More information about federated data-
base architectures can be found in Cal Henderson’s book, Building Scalable Web Sites
(O’Reilly).

OK, enough diversions—let’s get back to our database capacity example and summarize
where we are to this point. Database replication lag is bad and we want to avoid it. We hit
replication lag when we see 40 percent disk I/O wait, and we reach that threshold when
we’ve installed enough users and photos to produce a photos-to-user ratio of 110. We
know how our photo uploads and user registrations grow, because we capture that on a
daily basis (Figure 4-7). We are now armed with all the information we need to make
informed decisions regarding how much database hardware to buy, and when.

We can extrapolate a trend based on this data to predict how many users and photos we’ll
have on Flickr for the foreseeable future, then use that to gauge how our photos/user ratio
will look on our databases, and whether we need to adjust the maximum amounts of
users and photos to ensure an even balance across those databases.

We’ve found where the elbow in our performance (Figure 4-6) exists for these data-
bases—and therefore our capacity—but what is so special about this photos/users ratio for
our databases? Why does this particular value trigger performance degradation? It could
be for many reasons, such as specific hardware configurations, or the types of queries that
result from having that much data during peak traffic. Investigating the answers to these

72 CHAPTER FOUR
F I G U R E 4 - 7 . Photos uploaded and user registrations

questions could be a worthwhile exercise, but here again I’ll emphasize that we should sim-
ply expect this effect will continue and not count on any potential future optimizations.

Forecasting Peak-Driven Resource Usage: Web Server Example

When we forecast the capacity of a peak-driven resource, we need to track how the peaks
change over time. From there, we can extrapolate from that data to predict future needs.
Our web server example is a good opportunity to illustrate this process.

In Chapter 3, we identified our web server ceilings as 85 percent CPU usage for this partic-
ular hardware platform. We also confirmed CPU usage is directly correlated to the amount
of work Apache is doing to serve web pages. Also as a result of our work in Chapter 3, we
should be familiar with what a typical week looks like across Flickr’s entire web server
cluster. Figure 4-8 illustrates the peaks and valleys over the course of one week.

This data is extracted from a time in Flickr’s history when we had 15 web servers. Let’s
suppose this data is taken today, and we have no idea how our activity will look in the
future. We can assume the observations we made in the previous chapter are accurate
with respect to how CPU usage and the number of busy apache processes relate—which
turns out to be a simple multiplier: 1.1. If for some reason this assumption does change,
we’ll know quickly, as we’re tracking these metrics on a per-minute basis. According to
the graph in Figure 4-8, we’re seeing about 900 busy concurrent Apache processes during
peak periods, load balanced across 15 web servers. That works out to about 60 processes
per web server. Thus, each web server is using approximately 66 percent total CPU (we
can look at our CPU graphs to confirm this assumption).

PREDICTING TRENDS 73
 F I G U R E 4 - 8 . Busy Apache processes: weekly view

The peaks for this sample data are what we’re interested in the most. Figure 4-9 presents
this data over a longer time frame, in which we see these patterns repeat.

F I G U R E 4 - 9 . Weekly web server peaks across six weeks

It’s these weekly peaks that we want to track and use to predict our future needs. As it
turns out, for Flickr, those weekly peaks almost always fall on a Monday. If we isolate
those peak values and pull a trend line into the future as we did with our disk storage
example above, we’ll see something similar to Figure 4-10.

74 CHAPTER FOUR
F I G U R E 4 - 1 0 . Web server peak trend

If our traffic continues to increase at the current pace, this graph predicts in another eight
weeks, we can expect to experience roughly 1,300 busy Apache processes running at
peak. With our 1.1 processes-to-CPU ratio, this translates to around 1,430 percent total
CPU usage across our cluster. If we have defined 85 percent on each server as our upper
limit, we would need 16.8 servers to handle the load. Of course, manufacturers are reluc-
tant to sell servers in increments of tenths, so we’ll round that up to 17 servers. We cur-
rently have 15 servers, so we’ll need to add 2 more.

The next question is, when should we add them? As I explained in the sidebar “Don’t Buy
Before You Need It,” we can waste a considerable amount of money if we add hardware
too soon.

Fortunately, we already have enough data to calculate when we’ll run out of web server
capacity. We have 15 servers, each currently operating at 66 percent CPU usage at peak.
Our upper limit on web servers is set at 85 percent, which would mean 1,275% CPU
usage across the cluster. Applying our 1.1 multiplier factor, this in turn would mean 1,160
busy Apache processes at peak. If we trust the trend line shown in Figure 4-11, we can
expect to run out of capacity sometime between the 9th and 10th week.

Therefore, the summary of our forecast can be presented succinctly:

• We’ll run out of web server capacity three to four weeks from now.
• We’ll need two more web servers to handle the load we expect to see in eight weeks.

Now we can begin our procurement process with detailed justifications based on hardware
usage trends, not simply a wild guess. We’ll want to ensure the new servers are in place
before we need them, so we’ll need to find out how long it will take to purchase, deliver,
and install them.

PREDICTING TRENDS 75
 F I G U R E 4 - 1 1 . Capacity of 15 web servers

This is a simplified example. Adding two web servers in three to four weeks shouldn’t be
too difficult or stressful. Ideally, you should have more than six data points upon which to
base your forecast, and you likely won’t be so close to your cluster’s ceiling as in our
example. But no matter how much capacity you’ll need to add, or how long the time-
frame actually is, the process should be the same.

Caveats Concerning Small Data Sets

When you’re forecasting with peak values as we’ve done, it’s important to remember the
more data you have to fit a curve, the more accurate your forecast will be. In our example,
we based our hardware justifications on six weeks worth of data. Is that enough data to
constitute a trend? Possibly, but the time period on which you’re basing your forecasts is
of great importance as well. Maybe there is a seasonal lull or peak in traffic, and you’re on
the cusp of one. Maybe you’re about to launch a new feature that will add extra load to
the web servers within the timeframe of this forecast. These are only a few considerations
for which you may need to compensate when you’re making justifications to buy new
hardware. A lot of variables can come into play when predicting the future, and as a
result, we have to remember to treat our forecasts as what they really are: educated
guesses that need constant refinement.

Automating the Forecasting

Our use of Excel in the previous examples was pretty straightforward. But you can auto-
mate that process by using Excel macros. And since you’ll most likely be doing the same
process repeatedly as your metric collection system churns out new usage data, you can
benefit greatly by introducing some automation into this curve-fitting business. Other
benefits can include the ability to integrate these forecasts into a dashboard, plug them
into other spreadsheets, or put them into a database.

76 CHAPTER FOUR
An open source program called fityk ([Link] does a great job of curve-
fitting equations to arbitrary data, and can handle the same range of equation types as
Excel. For our purposes, the full curve-fitting abilities of fityk are a distinct overkill. It was
created for analyzing scientific data that can represent wildly dynamic datasets, not just
growing and decaying data. While fityk is primarily a GUI-based application (see
Figure 4-12), a command-line version is also available, called cfityk. This version accepts
commands that mimic what would have been done with the GUI, so it can be used to
automate the curve fitting and forecasting.

F I G U R E 4 - 1 2 . The fityk curve-fitting GUI tool

The command file used by cfityk is nothing more than a script of actions you can write
using the GUI version. Once you have the procedure choreographed in the GUI, you’ll be
able to replay the sequence with different data via the command-line tool.

If you have a carriage return–delimited file of x-y data, you can feed it into a command
script that can be processed by cfityk. The syntax of the command file is relatively straight-
forward, particularly for our simple case. Let’s go back to our storage consumption data for
an example.

In the code example that follows, we have disk consumption data for a 15-day period, pre-
sented in increments of one data point per day. This data is in a file called storage-
[Link], and appears as displayed here:

1 14321.83119
2 14452.60193
3 14586.54003

PREDICTING TRENDS 77
 4 14700.89417
5 14845.72223
6 15063.99681
7 15250.21164
8 15403.82607
9 15558.81815
10 15702.35007
11 15835.76298
12 15986.55395
13 16189.27423
14 16367.88211
15 16519.57105

The cfityk command file containing our sequence of actions to run a fit (generated using
the GUI) is called [Link], and appears as shown below:

# Fityk script. Fityk version: 0.8.2

@0 < '/home/jallspaw/[Link]'
guess Quadratic
fit
info formula in @0
quit

This script imports our x-y data file, sets the equation type to a second-order polynomial
(quadratic equation), fits the data, and then returns back information about the fit, such
as the formula used. Running the script gives us these results:

jallspaw:~]$cfityk ./[Link]
1> # Fityk script. Fityk version: 0.8.2
2> @0 < '/home/jallspaw/[Link]'
15 points. No explicit std. dev. Set as sqrt(y)
3> guess Quadratic
New function %_1 was created.
4> fit
Initial values: lambda=0.001 WSSR=464.564
#1: WSSR=0.90162 lambda=0.0001 d(WSSR)=-463.663 (99.8059%)
#2: WSSR=0.736787 lambda=1e-05 d(WSSR)=-0.164833 (18.2818%)
#3: WSSR=0.736763 lambda=1e-06 d(WSSR)=-2.45151e-05 (0.00332729%)
#4: WSSR=0.736763 lambda=1e-07 d(WSSR)=-3.84524e-11 (5.21909e-09%)
Fit converged.
Better fit found (WSSR = 0.736763, was 464.564, -99.8414%).
5> info formula in @0
# storage-consumption
14147.4+146.657*x+0.786854*x^2
6> quit

bye...

We now have our formula to fit the data:

0.786854x2 + 146.657x + 14147.4

Note how the result looks almost exactly as Excel’s for the same type of curve. Treating
the values for x as days and those for y as our increasing disk space, we can plug in our 25-
day forecast, which yields the same results as the Excel exercise. Table 4-2 lists the results
generated by cfityk.

78 CHAPTER FOUR
T A B L E 4 - 2 . Same forecast as Table 4-1, curve-fit by cfityk

y=0.786854x2 + 146.657x
Date Disk Available (GB) + 14147.4
33 08/27/05 20480.00 19843.97
34 08/28/05 20480.00 20043.34
35 08/29/05 20480.00 20244.29
36 08/30/05 20480.00 20446.81
37 08/31/05 20480.00 20650.91
38 09/01/05 20480.00 20856.58
39 09/02/05 20480.00 21063.83

Being able to perform curve-fitting with a cfityk script allows you to carry out forecasting
on a daily or weekly basis within a cron job, and can be an essential building block for a
capacity planning dashboard.

Safety Factors
Web capacity planning can borrow a few useful strategies from the older and better-
researched work of mechanical, manufacturing, and structural engineering. These disci-
plines also need to base design and management considerations around resources and
immutable limits. The design and construction of buildings, bridges, and automobiles
obviously requires some intimate knowledge of the strength and durability of materials,
the loads each component is expected to bear, and what their ultimate failure points are.
Does this sound familiar? It should, because capacity planning for web operations shares
many of those same considerations and concepts.

Under load, materials such as steel and concrete undergo physical stresses. Some have
elastic properties that allow them to recover under light amounts of load, but fail under
higher strains. The same concerns exist in your servers, network, or storage. When their
resources reach certain critical levels—100 percent CPU or disk usage, for example—they
fail. To pre-empt this failure, engineers apply what is known as a factor of safety to their
design. Defined briefly, a factor of safety indicates some margin of resource allocated
beyond the theoretical capacity of that resource, to allow for uncertainty in the usage.

While safety factors in the case of mechanical or structural engineering are usually part of
the design phase, in web operations they should be considered as an amount of available
resources that you leave aside, with respect to the ceilings you’ve established for each class
of resource. This will enable those resources to absorb some amount of unexpected
increased usage. Resources with which you should calculate safety factors include all the
those discussed in Chapter 3: CPU, disk, memory, network bandwidth, even entire hosts
(if you run a very large site).

For example, in Chapter 3 we stipulated 85 percent CPU usage as our upper limit for web
servers, in order to reserve “enough headroom to handle occasional spikes.” In this case,
we’re allowing a 15 percent margin of “safety.” When making forecasts, we need to take
these safety factors into account and adjust the ceiling values appropriately.

PREDICTING TRENDS 79
 Why a 15 percent margin? Why not 10 or 20 percent? Your safety factor is going to be
somewhat of a slippery number or educated guess. Some resources, such as caching sys-
tems, can also tolerate spikes better than others, so you may want to be less conservative
with a margin of safety. You should base your safety margins on “spikes” of usage that
you’ve seen in the past. See Figure 4-13.

F I G U R E 4 - 1 3 . Spike in traffic from Yahoo Front Page

Figure 4-13 displays the effect of a typically-sized traffic spike Flickr experiences on a reg-
ular basis. It’s by no means the largest. Spikes such as this one almost always occur when
the front page of [Link] posts a prominent link to a group, a photo, or a tag
search page on Flickr. This particular spike was fleeting; it lasted only about two hours
while the link was up. It caused an eight percent bump in traffic to our photo servers. See-
ing a 5–15 percent increase in traffic like this is quite common, and confirms that our 15
percent margin of safety is adequate.

Procurement
As we’ve demonstrated, with our resource ceilings pinpointed, we can predict when we’ll
need more of a particular resource. When we complete the task of predicting when we’ll
need more, we can use that timeline to gauge when to trigger the procurement process.

Your procurement pipeline is the process by which you obtain new capacity. It’s usually
the time it takes to justify, order, purchase, install, test, and deploy any new capacity.
Figure 4-14 illustrates the procurement pipeline.

The tasks outlined in Figure 4-14 vary from one organization to another. In some large
organizations, it can take a long time to gain approvals to buy hardware, but delivery can
happen quickly. In a startup, approvals may come quickly, but the installation likely pro-
ceeds more slowly. Each situation will be different, but the challenge will remain the same:
estimate how long the entire process will take, and add some amount of comfortable buffer
to account for unforeseen problems. Once you have an idea of what that buffer timeline
is, you can then work backward to plan capacity.

80 CHAPTER FOUR
F I G U R E 4 - 1 4 . Typical procurement pipeline

In our disk storage consumption example, we have current data on our disk consumption
up to 8/15/05, and we estimate we’ll run out of space on 8/30/05. You now know you
have exactly two weeks to justify, order, receive, install, and deploy new storage. If you
don’t, you’ll run out of space and be forced to trim that consumption in some way. Ideally,
this two-week deadline will be long enough for you to bring new capacity online.

Procurement Time: The Killer Metric

Obviously, the when of ordering equipment is just as important as the what and how much.
Procurement timelines outlined above hint at how critical it is to keep your eye on how
long it will take to get what you need into production. Sometimes external influences,
such as vendor delivery times and physical installation at the data center can ruin what
started out to be a perfectly timed integration of new capacity.

Startups routinely order servers purely out of the fear they’ll be needed. Most newly
launched companies have developers to work on the product and don’t need to waste
money on operations-focused engineers. The developers writing the code are most likely
the same people setting up network switches, managing user accounts, installing software,
and wearing whatever other hats are necessary to get their company rolling. The last thing
they want to worry about is running out of servers when they launch their new, awesome
website. Ordering more servers as needed can be rightly justified in these cases, because
the hardware costs are more than offset by the costs of preparing a more streamlined and
detailed capacity plan.

But as companies mature, optimizations begin to creep in. Code becomes more refined.
The product becomes more defined. Marketing starts to realize who their users are. The
same holds true for the capacity management process; it becomes more polished and accu-
rate over time.

PREDICTING TRENDS 81
 Just-In-Time Inventory
Toyota Motors developed the first implementations of a just-in-time inventory practice. It
knew there were large costs involved to organize, store, and track excess inventory of
automobile parts, so it decided to reduce that “holding” inventory and determine exactly
when it needed parts. Having inventory meant wasting money. Instead of maintaining a
massive warehouse filled with the thousands of parts to make its cars, Toyota would only
order and stock those parts as they were needed. This reduced costs tremendously and
gave Toyota a competitive advantage in the 1950s. Just-in-time inventory practice is now
part of any modern manufacturing effort.

The costs associated with having auto parts lying around in a warehouse can be seen anal-
ogous to having servers installed before you really need them. Rack space and power con-
sumption in a data center cost money, as does the time spent installing and deploying code
on the servers. More important, you risk suffering economically as a result of the afore-
mentioned Moore’s Law, which if your forecasts allow it, should motivate you to buy
equipment later, rather than sooner.

Once you know when your current capacity will top out, and how much capacity you’ll
need to get through to the next cycle of procurement, you should take a few lessons from
the just-in-time inventory playbook, whose sole purpose it is to eliminate waste of time
and money in the process.

Here are some of the steps in our typical procurement process you’ll want to pay attention
to, and streamline:

1. Determine your needs

You know how much load your current capacity can handle, because you’ve followed
the advice in Chapter 3 to find their ceilings and are measuring their usage
constantly. Take these numbers to the curve-fitting table and start making crystal ball
predictions. This is fundamental to the capacity planning process.
2. Justify purchases
Add some color and use attention-grabbing fonts on the graphs you just made in the
previous step, because you’re going to show them to the people who will approve the
hardware purchases you’re about to make. Spend as much time as you need to
ensure your money-handling audience understands why you’re asking for capacity,
why you’re asking for it now, and why you’ll be coming back later asking for more.
Be very clear in your presentations about the downsides of insufficient capacity.
3. Solicit quotes from vendors
Vendors want to sell you servers and storage; you want to buy servers and storage—
all is balanced in the universe. Why would you choose vendor A over vendor B?
Because vendor A might help alleviate some of the fear normally associated with
ordering servers, through such practices as quick turnarounds on quotes and
replacements, discounts on servers ordered later, or discounts tied to delivery times.

82 CHAPTER FOUR
 4. Order equipment
Can you track your order online? Do you have the phone number (gasp!) of a reliable
human who can tell you where your equipment is at all times? Does the data center
know the machines are coming, and have they factored that into their schedule?
5. Physical installation
How long will it take for the machines to make the journey from a loading dock into
a rack, and cabled up to a working switch? Does the data center staff need to get
involved, or are you racking machines yourself? Are there enough rack screws?
Power drill batteries? Crossover cables? How long is this entire process going to take?
6. OS/application/configuration installation
In the next chapter, we’ll talk about deployment scenarios that involve automatic OS
installation, software deployment, and configuration management. However, just
because it’s automated doesn’t mean it doesn’t take time and that you shouldn’t be
aware of any problems that can arise.
7. Testing
Do you have a QA team? Do you have a QA environment? Testing your application
means having some process by which you can functionally test all the bits you need
to make sure everything is in its right place. Entire books are written on this topic; I’ll
just remind you that it’s a necessary step in the journey toward production life as a
server.
8. Deploy your new equipment
It’s not over until the fat server sings. Putting a machine into production should be
straightforward. When doing so, you should use the same process to measure the
capacity of your new servers as outlined in the Chapter 3. Maybe you’ll want to ramp
up the production traffic the machine receives by increasing its weight in the load-
balanced pool. If you know this new capacity relieves a bottleneck, you’ll want to
watch any effect that has on your traffic.

The Effects of Increasing Capacity

All of the segments within your infrastructure interact in various ways. Clients make
requests to the web servers, which in turn make requests to databases, caching servers,
storage, and all sorts of miscellaneous components. Layers of infrastructure work together
to respond to users by providing web pages, pieces of web pages, or confirmations that
they’ve performed some action, such as uploading a photo.

When one or more of those layers encounters a bottleneck, you bring your attention to
bear, figure out how much more capacity you need, and then deploy it. Depending on
how bottlenecked that layer or cluster is, you may find you’ll see second-order effects of
that new deployment, and end up simply moving the traffic jam to yet another part of
your architecture.

PREDICTING TRENDS 83
 For example, let’s assume your website involves a web server and a database. One of the
ways organizations can help scale their application is to cache computationally expensive
database results. Deploying something like memcached can allow you to do this. In a nut-
shell, it means for certain database queries you choose, you can consult an in-memory
cache before hitting the database. This is done primarily for the dual purpose of speeding
up the query and reducing load on the database server for results that are frequently
returned.

The most noticeable benefit is queries that used to take seconds to process might take as
little as a few milliseconds, which means your web server will be able to send the response
to the client more quickly. Ironically, there’s a side effect to this; when users are not wait-
ing for pages as long, they have a tendency to click on links faster, causing more load on
the web servers. It’s not uncommon to see memcached deployments turn into web server
capacity issues rather quickly.

Long-Term Trends
Now you know how to apply the statistics collected in Chapter 3 to immediate needs. But
you may also want to view your site from a more global perspective—both in the literal
sense (as your site becomes popular internationally), and in a figurative sense, as you look
at the issues surrounding the product and the site’s strategy.

Traffic Pattern Changes

As mentioned earlier, getting to know the peaks and valleys of your various resources and
application usage is paramount to predicting the future. As you gain more and more his-
tory with your metrics, you may be able to perceive more subtle trends that will inform
your long-term decisions.

For example, let’s take a look at Figure 4-15, which illustrates a typical traffic pattern for a
web server.

Figure 4-15 shows a pretty typical U.S. daily traffic pattern. The load rises slowly in the
morning, East Coast time, as users begin browsing. These users go to lunch as West Coast
users come online, keeping up the load, which finally drops off as people leave work. At
this point, the load drops to only those users browsing over night.

As your usage grows, you can expect this graph to grow vertically as more users visit your
site during the same peaks and valleys. But if your audience grows more internationally,
the bump you see every day will widen as the number of active user time zones increases.
As seen in Figure 4-16, you may even see distinct bumps after the U.S. drop-off if your
site’s popularity grows in a region further away than Europe.

Figure 4-16 displays two daily traffic patterns, taken one year apart, and superimposed
one on top of the other. What once was a smooth bump and decline has become a two-
peak bump, due to the global effect of popularity.

84 CHAPTER FOUR
F I G U R E 4 - 1 5 . Typical daily web server traffic pattern

F I G U R E 4 - 1 6 . Daily traffic patterns grow wider with increasing international usage

Of course, your product and marketing people are probably very aware of the demograph-
ics and geographic distribution of your audience, but tying this data to your system’s
resources can help you predict your capacity needs.

Figure 4-16 also shows that your web servers must sustain their peak traffic for longer
periods of time. This will indicate when you should schedule any maintenance windows
to minimize the effect of downtime or degraded service to the users. Notice the ratio

PREDICTING TRENDS 85
 between your peak and your low period has changed as well. This will affect how many
servers you can stand to lose to failure during those periods, which is effectively the ceiling
of your cluster.

It’s important to watch the change in your application’s traffic pattern, not only for opera-
tional issues, but to drive capacity decisions, such as whether to deploy any capacity into
international data centers.

FORECASTING CONSIDERATIONS FOR MULTIPLE DATA

CENTERS
When your site’s growth dictates serving content out of multiple data centers, you might notice cer-
tain geographic traffic patterns can emerge you may not have seen with a single data center. At the
time of this writing, Flickr serves photos from eight different locations in the U.S., and will soon be
serving photos from European and Asian locations as well. Our data centers in the U.S. are split
between the east and west coasts, and users pull content from the data center to which they are geo-
graphically closest. Photos are distributed into what we term photo farms. A farm is made up of a mir-
rored pair of data centers, one on each coast. We currently have four farms (thus eight different data
centers), with each farm containing a unique set of photos. We store each photo in the two locations
for redundancy, in the event of emergency, or if one of a farm’s data centers needs to be taken offline
for maintenance.

When we first launched our second location, we noticed the east coast data center received as much
as 65–70 percent more traffic at peak than its west coast counterpart. This was easily explained, as at
the time our European users were a much more engaged audience than our Asian users, and since
the U.S. east coast is closer to Europe, the usage was commensurately higher. In addition, we noticed
the west coast data centers received considerably more requests for the larger, original sizes of pho-
tos than the east coast. We’ve attributed this to home broadband connections in Asia having higher
bandwidth limits, so they’re accustomed to downloading larger amounts of data. The usage gap has nar-
rowed as the Asian user base has become more engaged, but overall activity is still higher in the east.

What this meant was our peaks and valleys differed for each side of each farm, and therefore our
forecasts needed to be adjusted when we planned for growth. As just mentioned, the architecture for
each data center in a farm dictates it must be able to handle 100 percent of the traffic for entire farm
in the event its partner data center falls out of operation. Therefore, capacity forecasts need to be
based on the cumulative peaks of both data centers in a farm.

As I alluded to in Chapter 3 (“Knowing Your Waves”), when you deploy capacity to multiple data cen-
ters, usage patterns can become more complex. You’ll need to take that into consideration when fore-
casting capacity.

86 CHAPTER FOUR
Application Usage Changes and Product Planning
A good capacity plan not only relies on system statistics such as peaks and valleys, but user
behavior as well. How your users interact with your site is yet another valuable vein of
data you should mine for information to help keep your crystal ball as clear as possible.

If you run an online community, you might have discussion boards in which users create
new topics, make comments, and upload media such as video and photos. In addition to
the previously discussed system-related metrics, such as storage consumption, video and
photo processing CPU usage, and processing time, some other metrics you might want to
track are:

• Discussion posts per minute

• Posts per day, per user
• Video uploads per day, per user
• Photo uploads per day, per user

Application usage is just another way of saying user engagement, to borrow a term from the
product and marketing folks.

Recall back to our database-planning example. In that example, we found our database
ceiling by measuring our hardware’s resources (CPU, disk I/O, memory, and so on), relat-
ing them to the database’s resources (queries per second, replication lag) and tying those
ceilings to something we can measure from the user interaction perspective (how many
photos per user are on each database).

This is where capacity planning and product management tie together. Using your system
and application statistics histories, you can now predict with some (hopefully increasing)
degree of accuracy what you’ll need to meet future demand. But your history is only part
of the picture. If your product team is planning new features, you can bet they’ll affect
your capacity plan in some way.

Historically, corporate culture has isolated product development from engineering. Prod-
uct people develop ideas and plans for the product, while engineering develops and main-
tains the product once it’s on the market. Both groups make forecasts for different ends,
but the data used in those forecasts should tie together.

One of the best practices for a capacity planner is to develop an ongoing conversation
with product management. Understanding the timeline for new features is critical to
guaranteeing capacity needs don’t interfere with product improvements. Having enough
capacity is an engineering requirement, in the same way development time and
resources are.

PREDICTING TRENDS 87
 Iteration and Calibration
Producing forecasts by curve-fitting your system and application data isn’t the end of your
capacity planning. In order to make it accurate, you need to revisit your plan, re-fit the
data, and adjust accordingly.

Ideally, you should have periodic reviews of your forecasts. You should check how your
capacity is doing against your predictions on a weekly, or even daily, basis. If you know
you’re nearing capacity on one of your resources and are awaiting delivery of new hard-
ware, you might keep a much closer eye on it. The important thing to remember is your
plan is going to be accurate only if you consistently re-examine your trends and question
your past predictions.

As an example, we can revisit our simple storage consumption data. We made a forecast
based on data we gleaned for a 15-day period, from 7/26/05 to 8/09/05. We also discov-
ered that on 8/30/2005 (roughly two weeks later), we expected to run out of space if we
didn’t deploy more storage. More accurately, we were slated to reach 20,446.81 GB of
space, which would have exceeded our total available space is 20,480 GB.

How accurate was that prediction? Figure 4-17 shows what actually happened.

F I G U R E 4 - 1 7 . Disk consumption: predicted trend versus actual

As it turned out, we had a little more time than we thought—about four days more. We
made a guess based on the trend at the time, which ended up being inaccurate but at least
in favor of allowing more time to integrate new capacity. Sometimes, forecasts can either
widen the window of time (as in this case) or tighten that window.

88 CHAPTER FOUR
This is why the process of revisiting your forecasts is critical; it’s the only way to adjust
your capacity plan over time. Every time you update your capacity plan, you should go
back and evaluate how your previous forecasts fared.

Since your curve-fitting and trending results tend to improve as you add more data points,
you should have a moving window with which you make your forecasts. The width of
that forecasting window will vary depending on how long your procurement process
takes.

For example, if you know that it’s going to take three months on average to order, install,
and deploy capacity, then you’d want your forecast goal to be three months out, each
time. As the months pass, you’ll want to add the influence of most recent events to your
past data and recalculate your predictions, as is illustrated in Figure 4-18.

F I G U R E 4 - 1 8 . A moving window of forecasts

Best Guesses
This process of plotting, prediction, and iteration can provide a lot of confidence in how
you manage your capacity. You’ll have accumulated a lot of data about how your current
infrastructure is performing, and how close each piece is to their respective ceilings, taking
into account comfortable margins of safety. This confidence is important because the
capacity planning process (as we’ve seen) is just as much about educated guessing and
luck as it is about hard science and math. Hopefully, the iterations in your planning pro-
cess will point out any flawed assumptions in the working data, but it should also be said
the ceilings you’re using could become flawed or obsolete over time as well.

Just as your ceilings can change depending on the hardware specifications of a server, so
too can the actual metric you’re assuming is your ceiling. For example, the defining metric
of a database might be disk I/O, but after upgrading to a newer and faster disk subsystem,
you might find the limiting factor isn’t disk I/O anymore, but the single gigabit network
card you’re using. It bears mentioning that picking the right metric to follow can be diffi-
cult, as not all bottlenecks are obvious, and the metric you choose can change as the archi-
tecture and hardware limitations change.

During this process you might notice seasonal variations. College starts in the fall, so there
might be increased usage as students browse your site for materials related to their studies
(or just to avoid going to class). As another example, the holiday season in November and
December almost always witness a bump in traffic, especially for sites involving retail
sales. At Flickr, we see both of those seasonal effects.

PREDICTING TRENDS 89
 Taking into account these seasonal or holiday variations should be yet another influence
on how wide or narrow your forecasting window might be. Obviously, the more often
you recalculate your forecast, the better prepared you’ll be, and the sooner you’ll notice
variations you didn’t expect.

Diagonal Scaling Opportunities

As I pointed out near the beginning of the chapter, predicting capacity requires two essen-
tial bits of information: your ceilings and your historical data. Your historical data is etched
in stone. Your ceilings are not, since each ceiling you have is indicative of a particular
hardware configuration. Performance tuning can hopefully change those ceilings for the
better, but upgrading the hardware to newer and better technology is also an option.

As I mentioned at the beginning of the book, new technology (such as multicore proces-
sors) can dramatically change how much horsepower you can squeeze from a single
server. The forecasting process shown in this chapter allows you to not only track where
you’re headed on a per-node basis, but also to think about which segments of the archi-
tecture you might possibly move to new hardware options.

As discussed, due to the random access patterns of Flickr’s application, our database hard-
ware is currently bound by the amount of disk I/O the servers can manage. Each database
machine currently comprises six disks in a RAID10 configuration with 16 GB of RAM. A
majority of that physical RAM is given to MySQL, and the rest is used for a filesystem
cache to help with disk I/O. This is a hardware bottleneck that can be mitigated in a vari-
ety of ways, including at least:

• Spreading the load horizontally across many six-disk servers (current plan)
• Replacing each six-disk server with hardware containing more disk spindles
• Adding more physical RAM to assist both the filesystem and MySQL
• Using faster I/O options such as Solid-State Disks (SSDs)

Which of these options is most likely to help our capacity footprint? Unless we have to
grow the number of nodes very quickly, we might not care right now. If we have accurate
forecasts for how many servers we’ll need in our current configuration, we’re in a good
place to evaluate the alternatives.

Another long-term option may be to take advantage of the bottlenecks we have on those
machines. Since our disk-bound boxes are not using many CPU cycles, we could put those
mostly idle CPUs to use for other tasks, making more efficient use of the hardware. We’ll
talk more about efficiency and virtualization in Appendix A.

Summary
Predicting capacity is an ongoing process that requires as much intuition as it does math to
help you make accurate forecasts. Even simple web applications need to be attended, and
some of this crystal ball work can be tedious. Automating as much of the process as you

90 CHAPTER FOUR
can will help you stay ahead of the procurement process. Taking the time to connect your
metric collection systems to trending software, such as cfityk will prove to be invaluable as
you develop a capacity plan that is easily adaptable. Ideally, you’ll want some sort of a
capacity dashboard that can be referred to at any point in time to inform purchasing,
development, and operational decisions.

The overall process in making capacity forecasts is pretty simple:

1. Determine, measure, and graph your defining metric for each of your resources.
Example: disk consumption
2. Apply the constraints you have for those resources.
Example: total available disk space
3. Use trending analysis (curve fitting) to illustrate when your usage will exceed your
constraint.
Example: find the day you’ll run out of disk space

PREDICTING TRENDS 91
 Chapter 5
CHAPTER FIVE

Deployment

ONCE YOU HAVE AN IDEA OF HOW MUCH CAPACITY YOU’LL NEED FOR FUTURE GROWTH AND HAVE PURCHASED
the hardware, you’ll need to physically install it and deploy it into production.

Historically, deployment has been viewed as a headache. Installing the operating system
and application software, making sure all of the right settings are in place, and loading
your website’s data—all these tedious steps must be done in order to integrate new hard-
ware that’s fresh out of the crate. Fortunately, the pain of repeating these steps over and
over has inspired an entire category of software: automated installation and configuration
tools.*

Automated Deployment Philosophies

Although various automatic installation and configuration tools differ in their implemen-
tation and execution, most of them share the same design philosophy. Just as with moni-
toring and metric-collection tools, many of these concepts and designs originated in the

* A substantial part of this chapter was written by my colleague Kevin Murphy at Flickr.

93
 high-performance computing (HPC) field. Because HPC and web operations have similari-
ties in their infrastructure, the web operations community has adopted many of these
tools and approaches.

Goal: Minimize Time to Provision New Capacity

The time needed to acquire, install, and provision new hardware must be factored into
your calculations as you determine when you’re going to run out of capacity. If your
capacity will be exhausted in six weeks, and it takes you three weeks to add new hard-
ware, you only have three weeks of breathing room. Automated deployment and configu-
ration minimizes the time spent on the phase of the process over which you have the most
control—integrating machines onto your network and beginning operations.

Goal: All Changes Happen in One Place

When making changes to hosts, it’s preferable to have a central location from which to
push changes appropriate to the servers you’re affecting. Having a central location pro-
vides a “control tower” from which to manage all aspects of your infrastructure. Unlike
server architectures, in which distributed resources help with horizontal scaling, central-
ized configuration and management environments yield several advantages:

• Version control can be used for all configurations: OS, application, or otherwise. RCS/
CVS/Subversion and others are used to track the “who, what, when, and why” of each
change to the infrastructure.
• Replication and backup of installation and configuration files is easier to manage.
• An aggregated configuration and management logging system is an ideal troubleshoot-
ing resource.
• This centralized management environment makes an ideal place to keep hardware
inventory, particularly if you want to have different configuration settings for different
hardware.

This is not to suggest that your configuration, installation, monitoring, and management
setup should be kept on a single server. Each of these deployment components demands
specific resources. Growth over time would simply overwhelm a single machine, render-
ing it a potential single point of failure. Separate these components from the rest of your
infrastructure. Monitoring and metric collection can reside on one server; configuration
management and log aggregation on another. See Figure 5-1 for an example of a typical
installation, configuration, and management architecture.

Goal: Never Log In to an Individual Server (for Management)

Restrict logging into an individual server to troubleshooting or forensic work only. All
configuration or management should be done from the centralized management servers.
This keeps your production changes in an auditable and controlled environment.

94 CHAPTER FIVE
F I G U R E 5 - 1 . Typical cluster management scenario

Goal: Have New Servers Start Working Automatically

This is the holy grail of all deployment systems. You should be able to order hardware, add
it to your inventory-management system, turn it on, have it automatically install all
required software and register itself with your monitoring and trending systems, and then
get down to work, all without administrator intervention.

Maintain Consistency for Easier Troubleshooting

Coordination of troubleshooting efforts during outages or emergencies is critical. Configu-
ration management tools effectively enable that coordination. Consistent OS and software
configuration across a cluster of identically deployed servers eliminates configuration
inconsistencies as a source of problems. Troubleshooting in large production environ-
ments is difficult enough without needing to isolate configuration discrepancies between a
handful of servers.

DEPLOYMENT 95
 When tracking down a bug, the ability to quickly view all the changes that occurred
between the last known good state and the present (bad) state can be invaluable. When
more than one person is working on an issue, a centralized and consistent overview of all
of the changes is crucial. With version control, changes are immediately apparent and
tasks aren’t repeated. Nor are they assumed to have been completed or otherwise mixed
up with other tasks.

HOMOGENIZE HARDWARE TO HALT HEADACHES

In addition to making your software configurations consistent between servers that perform the
same role, it’s valuable to have consistency at the hardware level as well. At Flickr, we have two basic
server types: a multiple-disk machine for I/O intensive tasks, and a single-disk machine for CPU-
intensive jobs (for which most of the working set fits in memory). Limiting the number of hardware
types has a number of advantages:

• It reduces variability between machines, which simplifies troubleshooting.

• It minimizes the number of different spare parts you need to keep on hand and facilitates canni-
balizing dead machines for parts.
• It simplifies automated deployment and configuration because you don’t need to create numer-
ous special-case configurations for different hardware.

Virtualized infrastructures can take this a step further. You can buy racks of identical servers and allocate
memory, CPU, and high-performance remote storage to virtual machines based on application need.

Automated Installation Tools

Before you can even begin to worry about configuration management, you need to get
your servers to a state in which they can be configured. You want a system that can auto-
matically (and repetitively) install your OS of choice. Many such systems have been devel-
oped over the years, all employing similar techniques.

There are two basic approaches to the task of imaging new machines. Most OS vendors offer
a package-based installer option, which performs the normal installation process in a non-
interactive fashion. It provides the installer with a configuration file that specifies the pack-
ages to be installed. Examples include Solaris Jumpstart, Red Hat Kickstart, and Debian FAI.

Many third-party products take a disk-image approach. A gold client image is prepared on
one machine and replicated byte-for-byte onto newly imaged hosts. Often, a single image
is used for every server in the infrastructure, with hosts only differing in the services that
are configured and running. SystemImager is a product that uses this approach.

96 CHAPTER FIVE
Each method has advantages. Package-based systems provide accountability; every file
installed is guaranteed to belong to a package, and package management tools make it
easy to quickly see what’s installed. You can get the same result with disk image systems
by installing only packaged files. The temptation to muck about with the gold client file-
system directly can lead to confusion down the road.

On the other hand, image-based systems tend to be faster to install. The installer merely
has to create a filesystem and dump the image onto it, rather than download many packages,
calculate dependencies, and install them one by one. Some products, such as SystemImager,
even support parallel installs to multiple clients by streaming the disk images via multicast
or BitTorrent.

Preparing the OS Image

Most organizations aren’t happy with the operating system their vendor installs. Default
OS installs are notoriously inappropriate for production environments because they are
designed to run on as many different hardware platforms as possible. They usually contain
packages you don’t need and typically are missing those that you do. As a result, most
companies create custom OS images suitable for their specific needs.

For a package-based system, the OS image is specified by preparing a configuration file

that lists the set of packages to be installed. You can simply dump the package list from an
existing server and tweak it as desired. Then you put the configuration file in the appropri-
ate place on the boot server, and you’re ready to go.

For an image-based system, it’s slightly more involved. Normally, a server is set aside to be
the gold client that provides the template for the deployed image. This can be a physical
server set aside for the purpose, or even a virtual machine. You perform a normal OS
installation and install all the software required in your infrastructure. The client image is
then copied to the deployed server.

The Installation Process

If you’re installing more than a handful of machines, installations with physical boot media
such as CD-ROMs quickly become tedious and require someone to be physically present at
the data center. You’re going to want to set up a network boot server, which in the case of
PC hardware usually means PXE (Figure 5-2), or Pre-boot Execution Environment.

F I G U R E 5 - 2 . Basic steps of the PXE booting process

DEPLOYMENT 97
 PXE-based installs are performed by a number of services working together. PXE firmware
on the client requests its environment via DHCP. A DHCP server provides the information
required to fetch the boot image (IP address, boot server, and image name). Finally, the
client fetches the boot image from the boot server via TFTP.

For automated installation systems, the boot image consists of a kernel, plus a ramdisk
containing the installer program. (This differs from diskless client setups, where the boot
image contains the production kernel the client uses to mount a network volume as its
root filesystem.)

The installer then determines what type of host is being installed and formats the local
filesystems appropriately. There are several methods for mapping hardware profiles to
hosts, typically they involve assigning configurations based on hostname or MAC address.
For example, Kickstart passes the configuration file name as part of the DHCP options and
fetches it from the TFTP server, while SystemImager has a configuration file stored on the
image server that maps image types to hostnames.

The installer then installs the OS from the network onto the newly formatted volumes. As
it pertains to a package-based installer, this means copying package files from a network
repository (for example, apt or yum). For image-based systems, the OS image is dumped
directly onto the local volumes, usually via rsync or a similar program.

Once the OS image is installed, the PXE server marks the host as installed. Typically, this is
done by replacing the host’s bootfile with a stub that instructs the host to boot from the
local disk. The machine is restarted, and boots normally.

As shipped, most automated deployment systems require some manual configuration.

DHCP server configurations need to be created, and hostnames mapped to roles. However,
your inventory management system should have all the information about a machine
required to boot it: MAC address, role, IP address, and hostname. With a bit of clever script-
ing, you can automatically generate the required configuration from your asset database.

Once this is set up, provisioning new servers is as simple as entering their details into
inventory management, racking them, and powering them up. Reformatting a machine is
as simple as assigning it a new role in the database and rebooting it. (Normally, a reinstall is
required only if the disk layout changes. Otherwise, the server can simply be reconfigured.)

The deployment beast has been tamed.

Automated Configuration
Now that your machines are up on the network, it’s time to configure them to do their
jobs. Configuration management systems help with this task in the following ways:

• They let you organize your configuration files into useful subsystems, which you can
combine in various ways to build production systems.

98 CHAPTER FIVE
 INVENTORY MANAGEMENT
Once your site grows beyond a handful of boxes, you will need a system to keep track of your hard-
ware. Inventory management systems can be as simple as a spreadsheet or as elaborate as a full-
blown web application backed by a database. These systems have a frustrating tendency to drift out
of sync with reality as hardware is moved, decommissioned, or repurposed.

A way out of this trap is to make your inventory management system a part of your deployment infra-
structure. By using information from your inventory management to direct your installs and configure
deployments, you change it from a snapshot of your infrastructure as you think it is, to a source of
truth that guides and instructs your systems.

One system to take this approach is iClassify. It supports automatic registration, as well as automated
and manual classification. iClassify also integrates with Puppet for configuration management, and
Capistrano for ad-hoc management of servers.

• They put all the information about your running systems in one place, from which it
can easily be backed up or replicated to another site.
• They extract institutional knowledge out of your administrator’s head and place it into
a form that can be documented and reused.

A typical configuration management system consists of a server in which configurations

are stored, and a client process, which runs on each host and requests a configuration
from the server. In an infrastructure with automated deployment, the client is run as part
of the initial install or immediately after the initial boot into the new OS.

After the initial configuration, a scheduled task on the client host periodically polls the
server to see if any new configuration is available. Automating these checks ensures every
machine in your infrastructure is always running the latest configuration.

Defining Roles and Services

You have a shiny new configuration management system installed. How do you actually
use it? The best way to attack the problem is to divide and conquer. Services (collections of
related software and configurations) are the atoms, and roles (the types of machines in
your infrastructure) are the molecules. Once you have a robust set of services defined, it
will be easy to shuffle existing services into alternative combinations to serve new roles or
to split an existing role into more specialized configurations.

DEPLOYMENT 99
 First, go through all of the machines in your infrastructure (or planned infrastructure) and
identify the roles present. A role is a particular type of machine that performs a particular
task. For a website, your list will include roles like “web server” and “database.”

Next, go through each role and determine which services need to be present on each
instance of the role for the instance to be able to do its job. A service in this sense is not
just an OS-level program like “httpd.” For example, the http server service would include
not only the httpd package and its configuration, but also settings for any metrics, health
checks, or associated software that runs on a machine serving web pages.

As you go through your roles, try to identify services that may be common to multiple
roles. For example, every server is likely to require a remote login service such as sshd. By
identifying these common services, you can create a single set of configuration routines
that can be used over and over in roles you’re deploying now, and in new ones that will
emerge as your site grows.

An Example: Splitting Off Static Web Content

Suppose you have a cluster of four web servers. Each machine serves a combination of static
content and dynamic pages generated by PHP. You have Apache’s MaxClients setting tuned
down to 80 simultaneous connections to ensure the machines don’t overrun available mem-
ory and initiate swapping. The web server role might look similar to Figure 5-3.

F I G U R E 5 - 3 . The web server role and associated services

You realize in a flash of insight that you could serve more simultaneous clients by splitting
your cluster into two roles: one with the current configuration for dynamic content, and
one with a stripped-down Apache with a larger MaxClients to serve only static content.

100 CHAPTER FIVE

First, you split out services common to both roles into a separate service called base_http.
This service includes settings that any web server should have, such as an HTTP health
check and metrics pulled from Apache’s status module.

Next, you create two new services. The dynamic http service contains your original
Apache and PHP configurations. The static http service is configured with a simplified
[Link] with a larger client cap and no PHP module.

You then define roles for each server type by combining these services. The new roles are
depicted in Figure 5-4.

F I G U R E 5 - 4 . Static web server role and associated services

Now that the new role is defined, you can either go into your inventory management sys-
tem and assign one or more existing web server machines to the static_webserver role, or
deploy new hardware with the new role. If you decide to add more metrics or health-
checks, which are applicable to both roles in the future, you can put them in the base_http
service, and both roles will inherit them.

User Management and Access Control

User management and access control require special consideration. How do you control
which users have access to which portions of your system?

In fact, there are several ways. Network-based authentication and authorization services
such as LDAP are popular. Users and groups can be granted access to individual machines
or host groups in one place. Permission and password changes are propagated automati-
cally. On the downside, these systems represent yet another service that needs to be provi-
sioned, monitored, and scaled as your infrastructure grows.

DEPLOYMENT 101
 Alternately, it’s possible to use your configuration management system to install user
accounts on a host or role basis by defining services that make the appropriate changes to
the system authentication databases. This is straightforward if you already have configura-
tion management in place.

However, with such a system, password changes and access additions and revocations may
not be applied to all servers simultaneously. Additionally, if automated configuration
updates are broken on a host, that host may not receive the latest access configuration at
all, which is an obvious security concern.

Both of these setups can be made to work. Which one is most appropriate for your infra-
structure depends on your existing authentication systems, the number of users involved,
and the frequency with which changes are made. If you are already using LDAP elsewhere
in your organization, that may be the natural choice. If you have a small number of users
and changes are infrequent, a package-based system may be appropriate.

Ad Hockery
So, you’ve achieved the dream of configuration management—an infrastructure full of
properly configured servers that you never need to log into to manage.

But what if you want to? There are times when you may like to log into all of the mem-
bers of a particular role and run a command. Fortunately, there are tools to make this task
easier. These tools run the gamut from simple “run ssh in a for loop” scripts, to sophisti-
cated remote scripting systems, like Capistrano.

Ideally, such a tool should integrate with your configuration management system. You
want to be able to log into groups of servers by role, or even by service. This may require
some scripting on your part to convert your role and service definitions into a format the
tool understands. Alternately, the tool may be provided as a component of your configura-
tion management or monitoring system (such as the gexec utility provided with Ganglia).

You can run commands on multiple servers. Should you?

In general, a good use for these utilities is to gather ad hoc data about your systems—per-
haps information you’re not measuring with your trending tools. They’re also useful for
debugging and forensics. The rule of thumb should be: If it’s not something that you
should be collecting as a metric, and it won’t affect server state, it’s OK.

When is it a bad idea? You should hesitate any time it would be more appropriate to use
configuration management. There’s always the possibility of forgetting ad hoc changes
you made. You will regret forgetting.

102 CHAPTER FIVE

Example 2: Multiple Data Centers
Eventually, you’ll want to attack the greatest single point of failure of them all—the data
center. You want to be able to continue to serve traffic even if your data center experi-
ences a catastrophic power failure or other calamity. When you expand your infrastruc-
ture into multiple physical sites, you begin to realize the gains of automation on an even
larger scale.

Bringing up another data center can look like a logistical nightmare on paper. It took you
months or years to get your current systems in place. How will you be able to rebuild
them in another location quickly? Automated deployment can make the prospect of
bringing up an entire facility from bare metal much less daunting.

Rather than replicate each system in the original site on a host-by-host basis, the process
unfolds as such:

• Set up management hosts in the new data center. The base installs may be manual, but
the configuration is not—your management host configurations should be in configu-
ration management as well!
• Tweak the base configurations to suit the new environment. For example, DNS config-
uration and routing information will differ.
• Allocate roles to the new machines on the boot server (or in inventory management).
• Boot the hosts and allow them to install and configure themselves.

To simplify synchronization of settings between data centers, it’s best to keep all data cen-
ter-specific configurations in a separate service or set of services. This allows you to attain
maximum reuse out of your service definitions.

Summary
Knowing how much hardware you need does little good if you can’t get that hardware
into service quickly. Automating your infrastructure with tools like configuration man-
agement and automated installation, ensures your deployment processes are efficient and
repeatable. Automation converts system administration tasks from one-off efforts into
reusable building blocks.

DEPLOYMENT 103
Appendix A
APPENDIX A

Virtualization and Cloud Computing

T
WO OF THE GOALS OF CAPACITY PLANNING ARE TO EMPLOY THE RESOURCES YOU HAVE ON HAND IN THE
most efficient manner, and to predict future needs based on the patterns of current use.
For those well-defined workloads, you can get pretty close to utilizing most of the hard-
ware resources for each class of server you have, such as databases, web servers, and stor-
age devices. Unfortunately, web application workloads are rarely (if ever) perfectly aligned
with the available hardware resources.

In those circumstances, you end up with inefficiencies in your capacity. For example, if
you know your database’s specific ceiling (limit) is determined by its memory or disk
usage, but meanwhile it uses very little CPU, then there’s no reason to buy servers with
two quad-core CPUs. That resource (and investment) will simply be wasted unless you
direct the server to work on other CPU-intensive tasks. Even buying a single CPU may be
overkill. But often, that’s all that’s available, so you end up with idle resources.

It’s the continual need to balance correct resources to workload demand that makes
capacity planning so important, and in recent years some technologies and approaches
have emerged that render this balance easier to manage, with ever-finer granularity.

105
 Server virtualization and cloud computing are two such approaches, and it’s worth exploring
what they mean in the context of capacity planning.

Virtualization
There are many definitions of virtualization. In general, virtualization is the abstraction of
computing resources at various levels of a computer. Hardware, application, and operating
system levels are some of the few places in which this abstraction can take place, but in
the context of growing web operations, virtualization is generally used to describe OS
abstraction, otherwise known as server virtualization.

An example of this is the Xen virtual machine monitor, or VMWare’s ESX server, where a
bottom-level OS functions with guest operating systems running on top of it. The bottom-
level OS, known as the hypervisor, can be thought of as the brains of the virtualization. It
allows the guest operating systems to share resources and easily be created, destroyed, or
migrated to other hosts.

Entire books are written on the topic of virtualization. As it relates to capacity planning,
virtualization allows for more granular control of how resources are used at the bare metal
level. Figure A-1 illustrates this concept.

F I G U R E A - 1 . Virtual servers running on bare-metal hardware

Figure A-1 shows multiple guest operating systems running on the same server. There are
many advantages to employing this abstraction:

106 APPENDIX A
Efficient use of resources
There’s no reason to waste an entire server to run small tasks like corporate email. If
there are spare CPU, memory, or disk resources, you can pile on other services to that
resource to make better use it. Because of this, organizations use virtualization to con-
solidate many servers to run on a single piece of hardware.
Portability and fault tolerance
When a physical host is reaching known (or perhaps unknown) limits, or suffers a
hardware failure, a guest OS (and its associated load) can be safely migrated to another
host.
Development sandboxes
Because entire operating systems can be created and destroyed without harming the
underlying host environment, virtualization is ideal for building multiple development
environments that demand different operating systems, kernels, or system configura-
tions. If there’s a major bug that causes the entire test-bed to explode, no problem—it
can be easily recreated.
Less management overhead
Virtualization allows you to consolidate several individual servers with idle resources
into fewer servers with higher resource utilization. This can translate into reduced
power consumption, as well as a smaller data center footprint. Another benefit of less
hardware is there are fewer parts subject to failure, such as disk drives, CPUs, and
power supplies. Of course, the counterpoint to this is consolidation can increase your
exposure to a single-point-of-failure (SPOF) as many services are dependent on the
same physical hardware. Virtualization packages solve this potential problem by allow-
ing virtual machines to easily migrate from server to server for disaster recovery, and
for rebalancing workloads.

Virtualization essentially allows you to do more work with less hardware. These efficien-
cies have a tradeoff in that they can complicate measurements. Identifying which resource
is virtual usage, and which is physical can be confusing, as the abstraction layer introduces
another level of metric collection and measurement.

One additional advantage to virtualization is you can separate out application ceilings on a
role-by-role basis, even when you are only running on a single physical server. For exam-
ple, let’s say you’re consolidating email, backup, and logging services onto a single server.
You may allocate more memory to the logging services for buffering the log writes to disk,
and you may allocate more disk space to the backup application so it has room to grow.

As long as you can keep track of the virtual and the physical, the capacity planning process
is roughly the same. Consider your physical servers as generic containers in which you
can run a limited number of virtual servers.

VIRTUALIZATION AND CLOUD COMPUTING 107

 Cloud Computing
The concept of packaging up computing resources (computation, storage) into rentable
units, like power and telephone utilities, isn’t a new one. Virtualization technologies have
spawned an entire industry of computing “utility” providers, who leverage the efficiencies
inherent in virtualization to build what are known as clouds. Cloud service providers then
make those resources available on a cost-per-usage basis via an API, or other means. Since
cloud computing and storage essentially takes some of the infrastructure deployment and
management out of the hands of the developer, using cloud infrastructure can be an
attractive alternative to running your own servers. But as with virtualization, you lose
some of the ability to monitor and precisely measure your usage.

Computing Resource Evolutions

No matter how you look at it, your website needs computing and storage resources.
Somewhere—whether in your direct and total control or not—a server will need to
respond to client requests for data, and those requests may need some amount of compu-
tation and data retrieval.

Virtualization has been around almost as long as computing. At one time, computers were
seen as equipment only managed by large financial, educational, or research institutions.
Since computers were extremely expensive, IBM and other manufacturers built large-
scale minicomputers and mainframes to handle processing for multiple users at once, uti-
lizing many of the virtualization concepts still in use today. Users would be granted slices
of computation time from mainframe machines, accessing them from thin, or dumb, termi-
nals. Users submitted jobs whose computation contended for resources. The centralized
system was managed via queues, virtual operating systems, and system accounting that
governed resource allocation. All of the heavy lifting of computation was handled by the
mainframe and its operators, and was largely invisible to the end users. The design of
these systems was largely driven by security and reliability, so considerable effort was
applied to containing user environments and data redundancy. Figure A-2 illustrates the
client-sever structure in a mainframe environment.

Mixed Definitions
Grid computing, cloud computing and services, managed and virtual hosting, utility ser-
vices: as with many emerging technologies, terminology can sometimes be confusing—
especially when they become popular. The concept of cloud computing is not immune
from appellation bewilderment. Vendors, marketers, consultants, and service providers
stretch the term to include nearly any scenario wherein a generic and scalable infrastruc-
ture is made available to many users on a pay-per-resource basis.

Even the marketing for various cloud computing providers acknowledges this confusion.
Some of the more broad categories for these are:

108 APPENDIX A
F I G U R E A - 2 . Mainframe computing and the client-server paradigm

Cloud infrastructure
These are services such as Amazon Web Services and FlexiScale. They provide the
lower-level elements that most resemble a managed hosting environment, with API
access to control many of the services, including the ability to increase or decrease the
capacity. Computation and storage are currently the main offerings.
Platform clouds
Google’s AppEngine is an example of a service that, within certain constraints, allows cus-
tomers to write code that runs in a specialized environment, abstracted away from the
underlying operating system and hardware resources. Web applications have access to a
non-relational database that will expand or contract as usage dictates, up to prescribed
limits.
Application clouds
[Link]’s browser-based hosted applications are specific applications written by
a vendor for specific purposes, and allow API access to applications built upon them.
This is also known as SaaS, or “software-as-a-service.”

For the most part, growing web applications have been looking to the first category for
solutions to scaling their backend infrastructure, but as I said, this is an emerging and
evolving technology. The levels of abstraction, development environments, and deploy-
ment constraints for each type of cloud computing will be right for some applications, but
not for others.

There’s been enough history in the cloud infrastructure category that a number of busi-
nesses have moved some of their mission-critical operations to this model with success.
We’ll look at some example cases a little later.

VIRTUALIZATION AND CLOUD COMPUTING 109

 Cloud infrastructure is in many ways the next step in the evolution of the computing para-
digm, which began with the mainframe. As shown in Figure A-3, cloud providers build,
manage, and maintain all the physical hardware, organized in a virtualized cloud, while the
customers of those computation and storage services can be both individuals and businesses.

F I G U R E A - 3 . Basic cloud infrastructure

The underlying architectures behind cloud computing is essentially virtualization

expanded on a large scale. While there are varying types of cloud infrastructures, the pre-
vailing variety are compute instances and cloud storage.

Compute instances are essentially virtual OS environments in which you can run your
application’s code. These instances can be built, torn down, and rebuilt at the customer’s
request. They may or may not have any persistent storage of their own to work with, so
deleting an instance may wipe out the data you’ve written there.

On the other hand, cloud storage is basically a place to store data, and customers are billed
a monthly usage rate along with transfer charges for reading and writing to the storage.

Farms of different classes of servers comprise each tier of cloud service, whether it be pure
storage (in the case of a distributed file system), or a varying clusters of compute nodes
that are built to house virtualized guest instances of the customer’s operating system.

Cloud Capacity
One of the most touted advantages of the cloud-computing paradigm is the reduction of
hardware deployment and installation times. Indeed, a server that takes as little as 20
minutes to deploy with your automated installation and configuration process may take a
minute or less with a cloud service provider. As far as planning goes, cloud storage and
compute instances should be viewed as just another type of resource at your disposal. Just
like a single server, for each instance of cloud-based computing, you have some amount of:

• CPU
• RAM
• Disk (persistent or non-persistent)
• Network transfer (in and out)

110 APPENDIX A
Each cloud resource still has its ceilings and costs, just as with your existing infrastructure.
In many ways, the capacity planning process is exactly the same:

• Measure what you have consumed already (number of instances, CPU, or storage)
• Find your ceilings (when do you need to launch/tear down a new instance?)
• Forecast based on past usage

Why make forecasts if your deployment timeline is a matter of minutes? For one, forecast-
ing isn’t solely about staying ahead of procurement and deployment timelines.

The promise of cloud computing is that you can increase capacity “on-demand” easily, not
necessarily automatically. Since every instance is essentially a purchase, many cloud pro-
viders put the control of those instances in the hands of their customers, and deciding
when to launch new instances (and how many) can be crucially important in the face of
spiking traffic. An evolved operation that is using cloud computing might automate the
process, but the automation should be tuned carefully to react not only to the load behav-
ior of their website, but the load behavior of the cloud itself. We’ll see an example of this
“load feedback” capacity tuning a little later.

As with your own servers, cloud resources cost money. One important thing to remember
is the costs of cloud computing can rise surprisingly fast if you don’t pay attention. Track-
ing those costs will become important when you have the power to launch 100 instances
at once.

Use it or lose it (your wallet)

There’s also an additional variable with cloud infrastructure that you don’t normally need
to consider when deploying your own equipment: metered costs. CPU-driven cloud com-
puting is based on the premise that you only use what you need, when you need it. Eco-
nomically, it might not make sense to turn on CPU instances and simply let them run at
anything less than peak capacity.

Most cloud infrastructure providers offer a “menu” of compute instance items, ranging
from lower-powered CPU and memory platforms to large-scale, multi-core CPU systems
with massive amounts of memory. These choices aren’t as customizable as off-the-shelf
systems you own yourself, so when you determine your needs you’ll have to fit them into
these more coarse-grained options. You may find many smaller-sized instances are prefer-
able to having fewer large-sized ones, but each application is different. See Figure A-4 for
an example of a cloud computing pricing menu.

As we talked about in Chapter 4, in the traditional sense of running your own hardware,
the purchasing and approval process in an organization might take some time, as buying
hardware is capital investment in the business. Since cloud infrastructure splits that con-
cept into tiny chunks of investment here and there, does the approval process remain the
same? Your systems administrator might not have had the authority to purchase 10 serv-
ers without approval from management. With cloud computing, does he have the author-
ity to launch any number of instances that, over time, will equate to the same cost?

VIRTUALIZATION AND CLOUD COMPUTING 111

 F I G U R E A - 4 . Amazon EC2 menu: choices and prices

The point here is that tracking costs is another reason to pay attention and measure your
cloud usage in the same way you would your own resources.

Measuring the clouds

When you operate your own infrastructure, you have many options for measuring your
current capacity. Metric collection tools for historical data and trending, event monitoring
for critical threshold altering, and of course OS-level tools for ad-hoc troubleshooting;
these are among the tools we’ve talked about in this book.

Because cloud infrastructure is in many ways a “black box,” and because most cloud ser-
vice providers offer a relatively limited menu of resource choices, the importance of effec-
tive measurement becomes even clearer. Even though deployment timelines can shrink
significantly with cloud services, you should still be paying attention to the rest of the
capacity planning process. Finding your ceilings is still a necessity, and deploying new
instances ahead of growth still follows that.

In the case of storage consumption, you can offload the forecasting to your cloud provider;
it’s their job to stay ahead of that curve. However, in the case of compute instances, the
amount of actual “work” you’ll get out of them is going to depend largely on the size/class
of the instance, and your application’s resource usage (taking into account, of course, the
size/class of the instance).

112 APPENDIX A
Outside of the normal reasons for measuring your cloud’s capacity, there are other reasons
for setting metric collection and event notification in place:

• The performance of any of your instances and your storage may vary as the cloud pro-
vider shuffles capacity around to rebalance their architecture for increasing load.
• Even if your cloud provider has SLAs in place, knowing when your instances or storage
has failed is obviously paramount. A good rule of thumb: trust, but verify on your own,
that the infrastructure is available.
• The rapid deployment of new capacity can happen quickly, but not instantaneously.
It’s worth measuring how long it can take to launch a new instance.
• While cloud storage may be auto-scaling for disk consumption, there may not be any
guarantees regarding how fast you can retrieve data from that storage. Just as in the
disk I/O-bound database in Chapter 3, storage throughput can be a factor in your
capacity, and should be measured. The same goes for “local” storage I/O that any of
your compute nodes experiences.

With most cloud providers, you may not have any clarity on how your allocated instances
run (or sometimes where). They may be running by themselves on brand new hardware.
They may be sharing hardware resources alongside other virtualized instances that have
their own fluctuating capacity needs. They may be located on a network switch whose
usage is close to its limits. Your instances may or may not have consistent latency in stor-
ing or retrieving data from its persistent storage. You simply don’t know any of the physi-
cal details under the covers of your cloud provider’s interface.

This lack of transparency brings some advantages. It allows for the cloud provider to make
unfettered changes to their operation (which can enable the rapid provisioning of new
instances) and also relieves the customer of concerns about the nitty-gritty details
involved in scaling the capacity. But it also means you don’t have all the details you may
wish, especially when you’re accustomed to having every detail at your fingertips.

Cloud Case Studies

Should you use cloud services for your growing website? The short answer: “it depends.”
The longer answer is, cloud infrastructure, like every technology, has its pros and cons. On
the one hand, ridding yourself of deployment and provisioning hassles may be benefit
enough for you to switch to clouds. On the other hand, you give up some degree of con-
trol that you may be used to.

It’s too early in the history of cloud computing, and the variety of offerings is too great to
set up heuristics and formal guidelines for when to use it. Anecdotes and case studies,
however, can help you decide whether you’re in the right class of user and what factors to
consider. As I mentioned earlier, some businesses have been able to migrate some of their
mission-critical services to clouds with varying success. To illustrate this, I’ve included
some cases below. In preparation for this book, I interviewed a number of organizations

VIRTUALIZATION AND CLOUD COMPUTING 113

 that have either direct experience using cloud infrastructure or have evaluated them as
part of the process of building out new capacity.

Each one of these use cases highlights how drastically different the needs, concerns, and
benefits can be for each organization when using cloud infrastructure.

Cloud Use Case: Anonymous Desktop Software Company

A well-established desktop software company investigated using cloud infrastructure to
store and host digital media its users would process and upload from its product. Being a
desktop software company, it didn’t have a large online presence until this product was
launched. Nor did it have the necessary operational staff to manage the clusters of servers
needed to provide the hosting. The company considered using cloud infrastructure, hop-
ing to offload some of the deployment and management overhead that comes with a 24/7
online product. It evaluated the idea at the technical and business levels, looking at the
pros and cons.

In the end, the company decided to run their own infrastructure, for several reasons.

Lack of suitable SLAs

At the time, SLAs were only available in very limited terms with most of the established
cloud service providers. Being relatively new to the online world, the company didn’t
have high confidence in trusting a third-party (even a well-known cloud provider) with
the reliability and performance of its customer’s data, so the lack of a comprehensive SLA
played a significant role in their decision.

Legal concerns of user data

A concern was raised about who actually owns the user’s data while it’s being hosted on
the cloud provider’s servers. Did the cloud provider have any rights to the data since it was
physically located on its servers? Were there restrictions put in place that respected
national or international borders with regard to hosting? Legal issues involving user data,
privacy, and reliability are always fraught with what-if scenarios, and also played a role in
this particular company’s decision.

Cost
As it was an already established business, it had the capital to host the data itself, so the
economics of using cloud services were different from that of a startup. Once the company
factored in what it would cost to operate its own data center with the forecasts of the
online feature’s adoption, it decided the investment was going to be worth it in the long-
term, based on the cost histories of cloud providers at that point in time, as well as the
company’s own Total Cost of Ownership (TCO) calculations.

114 APPENDIX A
Control and confidence
Another main reason for the company not to use cloud infrastructure was a non-technical
one. Developing it themselves allowed the company to have familiarity with what it was
building. If the system failed, the company felt empowered to fix it in whatever manner
necessary. It also didn’t want to redesign its application in order to utilize the cloud
resources via the APIs. Finally, the company felt more comfortable that it could move to
the cloud sometime in the future, once the feature’s usage patterns emerged.

Cloud Use Case: [Link]

[Link] hosts over 2 million blogs (as of this writing), and gets upward of 30 mil-
lion pageviews a day. The company’s servers are located in three different data centers
which all replicate their data. For some time, this made uploading media (video, audio,
photos) difficult for the user simply because the company didn’t have a good handle on
deploying new storage, so it looked into using Amazon’s Simple Storage Service (S3) as a
backup/data recovery solution.

As WordPress became familiar with the service, it started to use S3 for its primary storage.
The reasoning behind using cloud storage was not economics; at the time of this writing,
S3 costs are actually three to four times more than if it bought and managed its own stor-
age. The advantage it was looking for was the ease of deployment and management. Not
having to keep ahead of storage usage meant they could focus on the other parts of the
infrastructure and site features. WordPress intended to use S3 essentially as a near-infinite
storage bin.

Because Amazon Web Services (AWS) charges for transferring data in and out of its S3
cloud, WordPress reverse-proxy caches the content it takes out of S3 to its own servers
(see Figure A-5).

F I G U R E A - 5 . [Link] reverse-proxy caching of Amazon of S3 storage

By caching frequently requested objects (or objects deemed to be “hot” enough to cache),
WordPress uses S3 in the most efficient way possible, avoiding unnecessary transfer costs
and serving content quickly via its own servers.

VIRTUALIZATION AND CLOUD COMPUTING 115

 Caching in this manner allows it to use S3 as a near-infinite storage resource. It doesn’t
need to worry about the limits and overhead involved with incrementally deploying new
storage itself.

Does this mean WordPress has ceased to do capacity planning? Far from it. It now has
caching systems (and databases and web servers) that need to be scaled, but this is accept-
able, as storage was its biggest pain point. WordPress was happy to redesign how it uses
storage in order to avoid worrying about it again.

Cloud Use Case: Anonymous News Aggregation Site

An early-stage startup decided to use cloud infrastructure to run its development environ-
ment as well as its limited beta website. The site is designed to crawl the Web and index
well-known news sources. The company would then process those documents with some
natural-language algorithms in order to uncover articles in which its users would be inter-
ested. It used compute instances to do the crawling and would store both the raw data and
the processed data in cloud storage.

While developing the site, the company noticed performance in its cloud-hosted applica-
tion would vary, with no emerging pattern. Sometimes, it would bring up a new compute
instance and notice the processing speed would vary from instance to instance, or would
sometimes pause inexplicably in the middle of a computation.

In addition, since the company’s processing was CPU intensive, it would launch many
smaller instances from the cloud provider’s menu of choices. As the application grew and
the company started using more and more of the smaller instances, they considered
upgrading to the next tier of the compute service for more CPU horsepower. While it was
more attractive to launch fewer “large” CPU instances to have more processing power on
hand, the pricing menu in place at the time made it prohibitively expensive when com-
pared to the smaller instances. This meant it would be forced to continue running many
small instances and accept the degraded memory and local I/O subsystems performance.
This wasn’t an ideal situation.

This gap in the cloud provider’s pricing menu caused the startup to conclude it should run
its own systems, and the organization began investigating managed hosting and co-location
facilities. The cloud provider then introduced changes to its pricing menu, offering a
“medium” level that better suited the startup’s budget and capacity needs.

This case underscores that cloud infrastructure services are still evolving their businesses
and still adjusting to their customer’s needs with regard to configurations and manage-
ment tools.

It also reinforces the importance of monitoring. When the startup began using Nagios to
monitor its instances, it was able to see and record how the instances were performing,
and make much more educated decisions on whether to launch more compute instances.

116 APPENDIX A
Cloud Use Case: [Link]
[Link] is a photo-sharing website, not unlike Flickr. It manages its own web serv-
ers and databases, but it uses Amazon’s cloud infrastructure extensively for both storage
(S3) and compute processing (EC2).

As of this writing, [Link] used over 600 TB of space with Amazon’s S3, and off-
loading this storage capacity to Amazon’s cloud allows it to focus on developing new fea-
tures. It started by using S3 as backup storage for its site. Once SmugMug felt comfortable
with performance and reliability, it moved all primary storage to S3, even when Amazon
Web Services did not have an SLA for this service (it does now). Moving storage was an
easy decision for SmugMug because it saw economic benefits and also did not want to
increase the size of its operations staff to manage in-house storage.

When photos are uploaded to SmugMug, they are placed in a queue that ships them off to
EC2 for processing into their various sizes for use on its site as well as other photo process-
ing actions. The same is done for uploaded video. Once the media has been processed, it’s
then stored directly to S3. This is an oversimplification of course, but during this process,
SmugMug manages their cloud use in an interesting way.

Capacity feedback loops

Since SmugMug automates the transfer and processing of the videos and photos to the
cloud via queues, it can control the rate at which processing jobs are sent, and how many
compute instances it might need at any given time to handle the load. Every minute,
SmugMug’s system looks at a number of different metrics to decide whether to turn on or
tear down compute instances, based on need. Some of the metrics it looks at are:

• Number of currently pending jobs

• Priority of pending jobs
• Type of jobs (photos, video, and so on)
• Complexity of the jobs (HD video versus 1 megapixel photo)
• Time sensitivity of pending jobs
• Current load on EC2 instances
• Average time per job
• Historical load and job performance
• Length of time it’s currently taking to launch a new compute instance

By taking these metrics into account, SmugMug’s system can appropriately scale up or
down the number of EC2 instances it’s using at any given time, thus making processing
much more efficient than without cloud infrastructure.

SmugMug’s use case follows the tenor of Chapters 3 and 4: marrying application and
system metrics to plan for capacity, and making forecasts based on that historical data.

VIRTUALIZATION AND CLOUD COMPUTING 117

 In SmugMug’s case, it is aware of its ceilings, monitoring how close it is to reaching them,
and adjusting its capacity on a tight, sliding window of time.

For SmugMug, taking into account its Total Cost of Ownership calculations, its desire to
keep its team small, and its scale, it’s economically advantageous for it to use cloud infra-
structure for storage and processing.

Summary
Deploying your site to cloud infrastructure can change how you view deploying capacity,
and largely depends on how you intend to make efficient use of it. In the use cases above,
we see both non-technical, and technical considerations, as outlined in the lists that follow.

Non-technical considerations:

• Legal concerns surrounding privacy, security, and ownership of data by a third-party.

• Confidence in the availability and performance of cloud infrastructure.
• The effect of SLAs (or lack thereof) in the context of pieces of your infrastructure.
• Levels of comfort with a still-emerging technology platform.

Technical considerations:

• Redesigning their own application to make the most efficient use of cloud resources.
Architectures that avoid transfer costs when possible and deploying compute instances
only when you need them are very common practices.
• Not knowing where your data physically resides. This forces developers to think about
their application (and the management of their application) at a higher-level. Expecting
that compute instances can stall, disappear, or migrate requires redundancy to be built in.

Regardless of how organizations decide to use cloud infrastructure, its effect on capacity
planning can be significant. [Link] is paying more for their storage than it did
prior to migrating its data storage, but is comfortable with that. [Link] is paying
less for Amazon S3 than it would if it were managing its own storage. Ultimately, there is
no one-case-fits-all situation with respect to cloud infrastructure; each decision is depen-
dent on the application and organization involved, just as with so many other technologies.

Clouds can shrink deployment timelines and provide more granular control over how
you’re using your capacity. These are facets of capacity management that we’ve discussed
in the previous chapters, and should be applied to cloud infrastructure as well:

• Put capacity measurement into place—both metric collection and event notification
systems—to collect and record systems and application statistics.
• Discover the current limits of your resources (utilization on compute nodes, for exam-
ple) and determine how close you are to those limits.
• Use historical data not only to predict what you’ll need, but to compare against what
you will actually use.

118 APPENDIX A
Planning for growth using cloud infrastructure is an evolving area. Cloud providers have
different restrictions, features, benefits, and drawbacks when compared to running your
own infrastructure. But once you have a good capacity planning process, you’ll be able to
adapt your planning methods to take those into account.

VIRTUALIZATION AND CLOUD COMPUTING 119