Download to read offline
![2.3 Implementing the Stable Matching Algorithm Using Lists and Arrays 43
an algorithm expressed in a high-level fashion—as we expressed the Gale-
Shapley Stable Matching algorithm in Chapter 1, for example—one doesn’t
have to actually program, compile, and execute it, but one does have to think
about how the data will be represented and manipulated in an implementation
of the algorithm, so as to bound the number of computational steps it takes.
The implementation of basic algorithms using data structures is something
that you probably have had some experience with. In this book, data structures
will be covered in the context of implementing specific algorithms, and so we
will encounter different data structures based on the needs of the algorithms
we are developing. To get this process started, we consider an implementation
of the Gale-Shapley Stable Matching algorithm; we showed earlier that the
algorithm terminates in at most n2 iterations, and our implementation here
provides a corresponding worst-case running time of O(n2), counting actual
computational steps rather than simply the total number of iterations. To get
such a bound for the Stable Matching algorithm, we will only need to use two
of the simplest data structures: lists and arrays. Thus, our implementation also
provides a good chance to review the use of these basic data structures as well.
In the Stable Matching Problem, each man and each woman has a ranking
of all members of the opposite gender. The very first question we need to
discuss is how such a ranking will be represented. Further, the algorithm
maintains a matching and will need to know at each step which men and
women are free, and who is matched with whom. In order to implement the
algorithm, we need to decide which data structures we will use for all these
things.
An important issue to note here is that the choice of data structure is up
to the algorithm designer; for each algorithm we will choose data structures
that make it efficient and easy to implement. In some cases, this may involve
preprocessing the input to convert it from its given input representation into a
data structure that is more appropriate for the problem being solved.
Arrays and Lists
To start our discussion we will focus on a single list, such as the list of women
in order of preference by a single man. Maybe the simplest way to keep a list
of n elements is to use an array A of length n, and have A[i] be the ith element
of the list. Such an array is simple to implement in essentially all standard
programming languages, and it has the following properties.
. We can answer a query of the form “What is the ith element on the list?”
in O(1) time, by a direct access to the value A[i].
. If we want to determine whether a particular element e belongs to the
list (i.e., whether it is equal to A[i] for some i), we need to check the](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-69-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![46 Chapter 2 Basics of Algorithm Analysis
assumption (or notation) allows us to define an array indexed by all men
or all women. We need to have a preference list for each man and for each
woman. To do this we will have two arrays, one for women’s preference lists
and one for the men’s preference lists; we will use ManPref[m, i] to denote
the ith woman on man m’s preference list, and similarly WomanPref[w, i] to
be the ith man on the preference list of woman w. Note that the amount of
space needed to give the preferences for all 2n individuals is O(n2), as each
person has a list of length n.
We need to consider each step of the algorithm and understand what data
structure allows us to implement it efficiently. Essentially, we need to be able
to do each of four things in constant time.
1. We need to be able to identify a free man.
2. We need, for a man m, to be able to identify the highest-ranked woman
to whom he has not yet proposed.
3. For a woman w, we need to decide if w is currently engaged, and if she
is, we need to identify her current partner.
4. For a woman w and two men m and m′, we need to be able to decide,
again in constant time, which of m or m′ is preferred by w.
First, consider selecting a free man. We will do this by maintaining the set
of free men as a linked list. When we need to select a free man, we take the
first man m on this list. We delete m from the list if he becomes engaged, and
possibly insert a different man m′, if some other man m′ becomes free. In this
case, m′ can be inserted at the front of the list, again in constant time.
Next, consider a man m. We need to identify the highest-ranked woman
to whom he has not yet proposed. To do this we will need to maintain an extra
array Next that indicates for each man m the position of the next woman he
will propose to on his list. We initialize Next[m]= 1 for all men m. If a man m
needs to propose to a woman, he’ll propose to w = ManPref[m,Next[m]], and
once he proposes to w, we increment the value of Next[m] by one, regardless
of whether or not w accepts the proposal.
Now assume man m proposes to woman w; we need to be able to identify
the man m′ that w is engaged to (if there is such a man). We can do this by
maintaining an array Current of length n, where Current[w] is the woman
w’s current partner m′. We set Current[w] to a special null symbol when we
need to indicate that woman w is not currently engaged; at the start of the
algorithm, Current[w] is initialized to this null symbol for all women w.
To sum up, the data structures we have set up thus far can implement the
operations (1)–(3) in O(1) time each.](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-72-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![2.4 A Survey of Common Running Times 47
Maybe the trickiest question is how to maintain women’s preferences to
keep step (4) efficient. Consider a step of the algorithm, when man m proposes
to a woman w. Assume w is already engaged, and her current partner is
m′ =Current[w]. We would like to decide in O(1) time if woman w prefers m
or m′. Keeping the women’s preferences in an array WomanPref, analogous to
the one we used for men, does not work, as we would need to walk through
w’s list one by one, taking O(n) time to find m and m′ on the list. While O(n)
is still polynomial, we can do a lot better if we build an auxiliary data structure
at the beginning.
At the start of the algorithm, we create an n × n array Ranking, where
Ranking[w, m] contains the rank of man m in the sorted order of w’s prefer-
ences. By a single pass through w’s preference list, we can create this array in
linear time for each woman, for a total initial time investment proportional to
n2. Then, to decide which of m or m′ is preferred by w, we simply compare
the values Ranking[w, m] and Ranking[w, m′].
This allows us to execute step (4) in constant time, and hence we have
everything we need to obtain the desired running time.
(2.10) The data structures described above allow us to implement the G-S
algorithm in O(n2) time.
2.4 A Survey of Common Running Times
When trying to analyze a new algorithm, it helps to have a rough sense of
the “landscape” of different running times. Indeed, there are styles of analysis
that recur frequently, and so when one sees running-time bounds like O(n),
O(n log n), and O(n2) appearing over and over, it’s often for one of a very
small number of distinct reasons. Learning to recognize these common styles
of analysis is a long-term goal. To get things under way, we offer the following
survey of common running-time bounds and some of the typical approaches
that lead to them.
Earlier we discussed the notion that most problems have a natural “search
space”—the set of all possible solutions—and we noted that a unifying theme
in algorithm design is the search for algorithms whose performance is more
efficient than a brute-force enumeration of this search space. In approaching a
new problem, then, it often helps to think about two kinds of bounds: one on
the running time you hope to achieve, and the other on the size of the problem’s
natural search space (and hence on the running time of a brute-force algorithm
for the problem). The discussion of running times in this section will begin in
many cases with an analysis of the brute-force algorithm, since it is a useful](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-73-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![2.5 A More Complex Data Structure: Priority Queues 59
of a priority queue. We could just have the elements in a list, and separately
have a pointer labeled Min to the one with minimum key. This makes adding
new elements easy, but extraction of the minimum hard. Specifically, finding
the minimum is quick—we just consult the Min pointer—but after removing
this minimum element, we need to update the Min pointer to be ready for the
next operation, and this would require a scan of all elements in O(n) time to
find the new minimum.
This complication suggests that we should perhaps maintain the elements
in the sorted order of the keys. This makes it easy to extract the element with
smallest key, but now how do we add a new element to our set? Should we
have the elements in an array, or a linked list? Suppose we want to add s
with key value key(s). If the set S is maintained as a sorted array, we can use
binary search to find the array position where s should be inserted in O(log n)
time, but to insert s in the array, we would have to move all later elements
one position to the right. This would take O(n) time. On the other hand, if we
maintain the set as a sorted doubly linked list, we could insert it in O(1) time
into any position, but the doubly linked list would not support binary search,
and hence we may need up to O(n) time to find the position where s should
be inserted.
The Definition of a Heap So in all these simple approaches, at least one of
the operations can take up to O(n) time—much more than the O(log n) per
operation that we’re hoping for. This is where heaps come in. The heap data
structure combines the benefits of a sorted array and list for purposes of this
application. Conceptually, we think of a heap as a balanced binary tree as
shown on the left of Figure 2.3. The tree will have a root, and each node can
have up to two children, a left and a right child. The keys in such a binary tree
are said to be in heap order if the key of any element is at least as large as the
key of the element at its parent node in the tree. In other words,
Heap order: For every element v, at a node i, the element w at i’s parent
satisfies key(w) ≤ key(v).
In Figure 2.3 the numbers in the nodes are the keys of the corresponding
elements.
Before we discuss how to work with a heap, we need to consider what data
structure should be used to represent it. We can use pointers: each node at the
heap could keep the element it stores, its key, and three pointers pointing to
the two children and the parent of the heap node. We can avoid using pointers,
however, if a bound N is known in advance on the total number of elements
that will ever be in the heap at any one time. Such heaps can be maintained
in an array H indexed by i = 1, . . . , N. We will think of the heap nodes as
corresponding to the positions in this array. H[1] is the root, and for any node](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-85-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![60 Chapter 2 Basics of Algorithm Analysis
1
2 5
10 3 11
7
15 17 20 9 15 8 16
1 2 5 10 3 11
7 15 17 20 9 15 8 16 X
Each node’s key is at least
as large as its parent’s.
Figure 2.3 Values in a heap shown as a binary tree on the left, and represented as an
array on the right. The arrows show the children for the top three nodes in the tree.
at position i, the children are the nodes at positions leftChild(i) = 2i and
rightChild(i) = 2i + 1. So the two children of the root are at positions 2 and
3, and the parent of a node at position i is at position parent(i) = ⌊i/2⌋. If
the heap has n N elements at some time, we will use the first n positions
of the array to store the n heap elements, and use length(H) to denote the
number of elements in H. This representation keeps the heap balanced at all
times. See the right-hand side of Figure 2.3 for the array representation of the
heap on the left-hand side.
Implementing the Heap Operations
The heap element with smallest key is at the root, so it takes O(1) time to
identify the minimal element. How do we add or delete heap elements? First
consider adding a new heap element v, and assume that our heap H has n N
elements so far. Now it will have n + 1 elements. To start with, we can add the
new element v to the final position i = n + 1, by setting H[i]= v. Unfortunately,
this does not maintain the heap property, as the key of element v may be
smaller than the key of its parent. So we now have something that is almost a
heap, except for a small “damaged” part where v was pasted on at the end.
We will use the procedure Heapify-up to fix our heap. Let j = parent(i) =
⌊i/2⌋ be the parent of the node i, and assume H[j]= w. If key[v] key[w],
then we will simply swap the positions of v and w. This will fix the heap
property at position i, but the resulting structure will possibly fail to satisfy
the heap property at position j—in other words, the site of the “damage” has
moved upward from i to j. We thus call the process recursively from position](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-86-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![2.5 A More Complex Data Structure: Priority Queues 61
2
4 5
10 9 11
7
15 17 20 17 15 8 16 3
w
v
v
w
2
4 5
10 9 3
7
15 17 20 17 15 8 16 11
The Heapify-up process is moving
element v toward the root.
Figure 2.4 The Heapify-up process. Key 3 (at position 16) is too small (on the left).
After swapping keys 3 and 11, the heap violation moves one step closer to the root of
the tree (on the right).
j = parent(i) to continue fixing the heap by pushing the damaged part upward.
Figure 2.4 shows the first two steps of the process after an insertion.
Heapify-up(H,i):
If i 1 then
let j = parent(i) = ⌊i/2⌋
If key[H[i]]key[H[j]] then
swap the array entries H[i] and H[j]
Heapify-up(H,j)
Endif
Endif
To see why Heapify-up works, eventually restoring the heap order, it
helps to understand more fully the structure of our slightly damaged heap in
the middle of this process. Assume that H is an array, and v is the element in
position i. We say that H is almost a heap with the key of H[i]too small, if there
is a value α ≥ key(v) such that raising the value of key(v) to α would make
the resulting array satisfy the heap property. (In other words, element v in H[i]
is too small, but raising it to α would fix the problem.) One important point
to note is that if H is almost a heap with the key of the root (i.e., H[1]) too
small, then in fact it is a heap. To see why this is true, consider that if raising
the value of H[1] to α would make H a heap, then the value of H[1] must
also be smaller than both its children, and hence it already has the heap-order
property.](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-87-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![62 Chapter 2 Basics of Algorithm Analysis
(2.12) The procedure Heapify-up(H, i) fixes the heap property in O(log i)
time, assuming that the array H is almost a heap with the key of H[i]too small.
Using Heapify-up we can insert a new element in a heap of n elements in
O(log n) time.
Proof. We prove the statement by induction on i. If i = 1 there is nothing to
prove, since we have already argued that in this case H is actually a heap.
Now consider the case in which i 1: Let v = H[i], j = parent(i), w = H[j],
and β = key(w). Swapping elements v and w takes O(1) time. We claim that
after the swap, the array H is either a heap or almost a heap with the key of
H[j] (which now holds v) too small. This is true, as setting the key value at
node j to β would make H a heap.
So by the induction hypothesis, applying Heapify-up(j) recursively will
produce a heap as required. The process follows the tree-path from position i
to the root, so it takes O(log i) time.
To insert a new element in a heap, we first add it as the last element. If the
new element has a very large key value, then the array is a heap. Otherwise,
it is almost a heap with the key value of the new element too small. We use
Heapify-up to fix the heap property.
Now consider deleting an element. Many applications of priority queues
don’t require the deletion of arbitrary elements, but only the extraction of
the minimum. In a heap, this corresponds to identifying the key at the root
(which will be the minimum) and then deleting it; we will refer to this oper-
ation as ExtractMin(H). Here we will implement a more general operation
Delete(H, i), which will delete the element in position i. Assume the heap
currently has n elements. After deleting the element H[i], the heap will have
only n − 1 elements; and not only is the heap-order property violated, there
is actually a “hole” at position i, since H[i] is now empty. So as a first step,
to patch the hole in H, we move the element w in position n to position i.
After doing this, H at least has the property that its n − 1 elements are in the
first n − 1 positions, as required, but we may well still not have the heap-order
property.
However, the only place in the heap where the order might be violated is
position i, as the key of element w may be either too small or too big for the
position i. If the key is too small (that is, the violation of the heap property is
between node i and its parent), then we can use Heapify-up(i) to reestablish
the heap order. On the other hand, if key[w] is too big, the heap property
may be violated between i and one or both of its children. In this case, we will
use a procedure called Heapify-down, closely analogous to Heapify-up, that](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-88-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![2.5 A More Complex Data Structure: Priority Queues 63
4
7 21
10 16 11
7
15 17 20 17 15 8 16
The Heapify-down process
is moving element w down,
toward the leaves.
w
4
7 7
10 16 11
21
15 17 20 17 15 8 16
w
Figure 2.5 The Heapify-down process:. Key 21 (at position 3) is too big (on the left).
After swapping keys 21 and 7, the heap violation moves one step closer to the bottom
of the tree (on the right).
swaps the element at position i with one of its children and proceeds down
the tree recursively. Figure 2.5 shows the first steps of this process.
Heapify-down(H,i):
Let n = length(H)
If 2i n then
Terminate with H unchanged
Else if 2i n then
Let left = 2i, and right = 2i + 1
Let j be the index that minimizes key[H[left]] and key[H[right]]
Else if 2i = n then
Let j = 2i
Endif
If key[H[j]] key[H[i]] then
swap the array entries H[i] and H[j]
Heapify-down(H, j)
Endif
Assume that H is an array and w is the element in position i. We say that
H is almost a heap with the key of H[i] too big, if there is a value α ≤ key(w)
such that lowering the value of key(w) to α would make the resulting array
satisfy the heap property. Note that if H[i] corresponds to a leaf in the heap
(i.e., it has no children), and H is almost a heap with H[i] too big, then in fact
H is a heap. Indeed, if lowering the value in H[i] would make H a heap, then](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-89-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![64 Chapter 2 Basics of Algorithm Analysis
H[i] is already larger than its parent and hence it already has the heap-order
property.
(2.13) The procedure Heapify-down(H, i) fixes the heap property in O(log n)
time, assuming that H is almost a heap with the key value of H[i]too big. Using
Heapify-up or Heapify-down we can delete a new element in a heap of n
elements in O(log n) time.
Proof. We prove that the process fixes the heap by reverse induction on the
value i. Let n be the number of elements in the heap. If 2i n, then, as we
just argued above, H is a heap and hence there is nothing to prove. Otherwise,
let j be the child of i with smaller key value, and let w = H[j]. Swapping the
array elements w and v takes O(1) time. We claim that the resulting array is
either a heap or almost a heap with H[j]= v too big. This is true as setting
key(v) = key(w) would make H a heap. Now j ≥ 2i, so by the induction
hypothesis, the recursive call to Heapify-down fixes the heap property.
The algorithm repeatedly swaps the element originally at position i down,
following a tree-path, so in O(log n) iterations the process results in a heap.
To use the process to remove an element v = H[i]from the heap, we replace
H[i] with the last element in the array, H[n]= w. If the resulting array is not a
heap, it is almost a heap with the key value of H[i] either too small or too big.
We use Heapify-down or Heapify-down to fix the heap property in O(log n)
time.
Implementing Priority Queues with Heaps
The heap data structure with the Heapify-down and Heapify-up operations
can efficiently implement a priority queue that is constrained to hold at most
N elements at any point in time. Here we summarize the operations we will
use.
. StartHeap(N) returns an empty heap H that is set up to store at most N
elements. This operation takes O(N) time, as it involves initializing the
array that will hold the heap.
. Insert(H, v) inserts the item v into heap H. If the heap currently has n
elements, this takes O(log n) time.
. FindMin(H) identifies the minimum element in the heap H but does not
remove it. This takes O(1) time.
. Delete(H, i) deletes the element in heap position i. This is implemented
in O(log n) time for heaps that have n elements.
. ExtractMin(H) identifies and deletes an element with minimum key
value from a heap. This is a combination of the preceding two operations,
and so it takes O(log n) time.](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-90-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![Solved Exercises 65
There is a second class of operations in which we want to operate on
elements by name, rather than by their position in the heap. For example, in
a number of graph algorithms that use heaps, the heap elements are nodes of
the graph with key values that are computed during the algorithm. At various
points in these algorithms, we want to operate on a particular node, regardless
of where it happens to be in the heap.
To be able to access given elements of the priority queue efficiently, we
simply maintain an additional array Position that stores the current position
of each element (each node) in the heap. We can now implement the following
further operations.
. To delete the element v, we apply Delete(H,Position[v]). Maintaining
this array does not increase the overall running time, and so we can
delete an element v from a heap with n nodes in O(log n) time.
. An additional operation that is used by some algorithms is ChangeKey
(H, v, α), which changes the key value of element v to key(v) = α. To
implement this operation in O(log n) time, we first need to be able to
identify the position of element v in the array, which we do by using
the array Position. Once we have identified the position of element v,
we change the key and then apply Heapify-up or Heapify-down as
appropriate.
Solved Exercises
Solved Exercise 1
Take the following list of functions and arrange them in ascending order of
growth rate. That is, if function g(n) immediately follows function f(n) in
your list, then it should be the case that f(n) is O(g(n)).
f1(n) = 10n
f2(n) = n1/3
f3(n) = nn
f4(n) = log2 n
f5(n) = 2
√
log2 n
Solution We can deal with functions f1, f2, and f4 very easily, since they
belong to the basic families of exponentials, polynomials, and logarithms.
In particular, by (2.8), we have f4(n) = O(f2(n)); and by (2.9), we have
f2(n) = O(f1(n)).](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-91-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
![68 Chapter 2 Basics of Algorithm Analysis
g1(n) = 2
√
log n
g2(n) = 2n
g4(n) = n4/3
g3(n) = n(log n)3
g5(n) = nlog n
g6(n) = 22n
g7(n) = 2n2
5. Assume you have functions f and g such that f(n) is O(g(n)). For each of
the following statements, decide whether you think it is true or false and
give a proof or counterexample.
(a) log2 f(n) is O(log2 g(n)).
(b) 2f(n) is O(2g(n)).
(c) f(n)2 is O(g(n)2).
6. Consider the following basic problem. You’re given an array A consisting
of n integers A[1], A[2], . . . , A[n]. You’d like to output a two-dimensional
n-by-n array B in which B[i, j] (for i j) contains the sum of array entries
A[i] through A[j]—that is, the sum A[i]+ A[i + 1]+ . . . + A[j]. (The value of
array entry B[i, j] is left unspecified whenever i ≥ j, so it doesn’t matter
what is output for these values.)
Here’s a simple algorithm to solve this problem.
For i = 1, 2, . . . , n
For j = i + 1, i + 2, . . . , n
Add up array entries A[i] through A[j]
Store the result in B[i, j]
Endfor
Endfor
(a) For some function f that you should choose, give a bound of the
form O(f(n)) on the running time of this algorithm on an input of
size n (i.e., a bound on the number of operations performed by the
algorithm).
(b) For this same function f, show that the running time of the algorithm
on an input of size n is also (f(n)). (This shows an asymptotically
tight bound of (f(n)) on the running time.)
(c) Although the algorithm you analyzed in parts (a) and (b) is the most
natural way to solve the problem—after all, it just iterates through](/p?url=https%3A%2F%2Fimage.slidesharecdn.com%2Falgorithmdesign-230806181921-eb1eb3a0%2F85%2FAlgorithm-Design-94-320.jpg&__src=https%3A%2F%2Fwww.slideshare.net%2Fslideshow%2Falgorithm-design-259674790%2F259674790&__type=image)
Uploaded byLaurie Smith
Algorithm Design
This document provides biographical information about the authors Jon Kleinberg and Éva Tardos, who authored the book "Algorithm Design". It also includes a brief overview of the book's contents and topics covered, such as algorithms, computational complexity, graph algorithms, greedy algorithms, divide-and-conquer, dynamic programming, and more. The preface outlines the book's goal of conveying the algorithm design process through problems from various computing applications.
