Introduction to Algorithms

Lecture 1
 Text Books

• Coreman, Rivest, Lisserson, : “Introduction to

Algorithms", PHI.
• Horowitz & Sahani, "Fundamental of
Computer Algorithm", Galgotia.
Introduction to Algorithms
An algorithm is a step-by-step procedure for solving a problem or
performing a task.

• Types:
⚬ Recursive Algorithms- Using Divide and Conquer method
Ex- Mergesort
⚬ Iterative Algorithms- Incremental approach
Ex-Insertion sort

• Importance: Efficient problem solving in computer science.

 Analysis of Algorithms

• WhyAnalyze?: To evaluate the performance (time and space complexity) of

an algorithm.

• Types ofAnalysis:
⚬ Worst Case
⚬ Best Case
⚬ Average Case
 Asymptotic Performance of an
Algorithm
• In this course, we care most about asymptotic
performance
 How does the algorithm behave as the problem
size gets very large?
o Running time
o Memory/storage requirements
o Bandwidth/power requirements/logic gates/etc.
 Asymptotic Notation
• By now you should have an intuitive feel for
asymptotic (big-O) notation:
 What does O(n) running time mean? O(n2)?
O(n lg n)?
 How does asymptotic running time relate to
asymptotic memory usage?
• Our first task is to define this notation more
formally and completely (which we will do in
next class)
 Analysis of Algorithms
• Analysis is performed with respect to a computational
model
• We will usually use a generic uniprocessor random-
access machine (RAM)
 All memory equally expensive to access
 No concurrent operations
 All reasonable instructions take unit time
o Except, of course, function calls
 Constant word size
o Unless we are explicitly manipulating bits
 Input Size

• Time and space complexity

 This is generally a function of the input size
o E.g., sorting, multiplication
 How we characterize input size depends:
o Sorting: number of input items
o Multiplication: total number of bits
o Graph algorithms: number of nodes & edges
o Etc
 Running Time

• Number of primitive steps that are executed

 Except for time of executing a function call most
statements roughly require the same amount of
time
oy=m*x+b
o c = 5 / 9 * (t - 32 )
o z = f(x) + g(y)
 Analysis

• Worst case
 Provides an upper bound on running time
 An absolute guarantee
• Average case
 Provides the expected running time
 Very useful, but treat with care: what is “average”?
o Random (equally likely) inputs
o Real-life inputs
 An Example: Insertion Sort
InsertionSort(A) {
for i = 2 to length[A] {
key = A[i]
j = i - 1
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
30 10 40 20 i =  j =  key = 
A[j] =  A[j+1] = 
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
30 10 40 20 i=2 j=1 key = 10
A[j] = 30 A[j+1] = 10
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
30 30 40 20 i=2 j=1 key = 10
A[j] = 30 A[j+1] = 30
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
30 30 40 20 i=2 j=1 key = 10
A[j] = 30 A[j+1] = 30
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
30 30 40 20 i=2 j=0 key = 10
A[j] =  A[j+1] = 30
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
30 30 40 20 i=2 j=0 key = 10
A[j] =  A[j+1] = 30
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=2 j=0 key = 10
A[j] =  A[j+1] = 10
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=3 j=0 key = 10
A[j] =  A[j+1] = 10
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=3 j=0 key = 40
A[j] =  A[j+1] = 10
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=3 j=0 key = 40
A[j] =  A[j+1] = 10
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=3 j=2 key = 40
A[j] = 30 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=3 j=2 key = 40
A[j] = 30 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=3 j=2 key = 40
A[j] = 30 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=4 j=2 key = 40
A[j] = 30 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 20
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 20 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 20
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 40 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 40 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 40 i=4 j=3 key = 20
A[j] = 40 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 40 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 40 40 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 40
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 30 40 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 30
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 30 40 i=4 j=2 key = 20
A[j] = 30 A[j+1] = 30
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 30 40 i=4 j=1 key = 20
A[j] = 10 A[j+1] = 30
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 30 30 40 i=4 j=1 key = 20
A[j] = 10 A[j+1] = 30
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 20 30 40 i=4 j=1 key = 20
A[j] = 10 A[j+1] = 20
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort
10 20 30 40 i=4 j=1 key = 20
A[j] = 10 A[j+1] = 20
1 2 3 4
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
} Done!
 Insertion Sort
InsertionSort(A, n) { What is the precondition
for i = 2 to n { for this loop?
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 Insertion Sort
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
} How many times will
} this loop execute?
 Insertion Sort
Statement Effort
InsertionSort(A) {
for i = 2 to length[A] { c1n
key = A[i] c2(n-1)
j = i - 1; c3(n-1)
while (j > 0) and (A[j] > key) { c4T
A[j+1] = A[j] c5(T-(n-1))
j = j - 1 c6(T-(n-1))
} 0
A[j+1] = key c7(n-1)
} 0
}
T = t2 + t3 + … + tn where ti is number of while expression evaluations for the ith
for loop iteration
 Analyzing Insertion Sort
• T(n) = c1n + c2(n-1) + c3(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)
= c8T + c9n + c10
• What can T be?
 Best case -- inner loop body never executed
o ti = 1  T(n) is a linear function
 Worst case -- inner loop body executed for all previous
elements
o ti = i  T(n) is a quadratic function
 Average case
o ti = i/2  T(n) is a quadratic function
 Order of Growth

• Ignore constants & lower order terms since it

is the high order term that dominates as input
size grows longer.

• We say running time of insertion sort is O(n2)

 Correct algorithms
• An algorithm is correct if:
– for any correct input data:
• it stops and
• it produces correct output.
– Correct input data: satisfies precondition (restrictions on input
data)
– Correct output data: satisfies postcondition (result)
• Example: Binary Search
– Input : a:array of integer; x:integer;
– Output : found:boolean;
– Precondition: a is sorted in ascending order
– Postcondition: found is true if x is in a, and found is false
otherwise
Proving correctness of an Algorithm

• This is easy to prove for simple sequential

algorithms
• This can be complicated to prove for repetitive
algorithms (containing loops)
 use techniques based on loop invariants and induction
 Example – a sequential algorithm
Swap1(x,y): Proof: the list of actions
aux := x applied to the
x := y precondition imply the
y := aux postcondition
1. Precondition:
x = a and y = b
Precondition:
2. aux := x => aux = a
x = a and y = b
3. x : = y => x = b
Postcondition:
4. y := aux => y = a
x = b and y = a
5. x = b and y = a is
the Postcondition
 Example – a repetitive algorithm
Algorithm Sum_of_N_numbers
Proof: the list of actions
applied to the
Input: a, an array of N numbers
precondition imply
Output: s, the sum of the N numbers
the postcondition
in a
BUT: we cannot
s:=0; enumerate all the
k:=0; actions in case of a
While (k<N) do repetitive algorithm !
k:=k+1; We use techniques
s:=s+a[k]; based on loop
end invariants and
induction
 Loop invariants
• A loop invariant is a logical predicate such that: if it is
satisfied before entering any single iteration of the
loop then it is also satisfied after the iteration
• In other words, a loop invariant is a property of a
program loop that is true before (and after) each
iteration
Example: Loop invariant for Sum of
n numbers
Algorithm Sum_of_N_numbers

Input: a, an array of N numbers

Output: s, the sum of the N numbers in a

s:=0; Loop invariant = induction

k:=0;
hypothesis: At step k, S holds the
While (k<N) do
k:=k+1; sum of the first k numbers
s:=s+a[k];
end
 Using loop invariants in proofs
• We must show the following 3 things about a loop
invariant:
1. Initialization: It is true prior to the first iteration of
the loop.
2. Maintenance: If it is true before an iteration of the
loop, it remains true before the next iteration.
3. Termination: When the loop terminates, the
invariant gives us a useful property that helps show
that the algorithm is correct.
 Example: Proving the correctness
of the Sum algorithm (1)
• Induction hypothesis: S= sum of the first k
numbers
1. Initialization: The hypothesis is true at the
beginning of the loop:
Before the first iteration: k=0, S=0. The first 0 numbers have
sum zero (there are no numbers) => hypothesis true
before entering the loop
 Example: Proving the correctness
of the Sum algorithm (2)
• Induction hypothesis: S= sum of the first k
numbers
2. Maintenance: If hypothesis is true before step k, then it
will be true before step k+1 (immediately after step k is
finished)
We assume that it is true at beginning of step k: “S is the sum of
the first k numbers”
We have to prove that after executing step k, at the beginning of
step k+1: “S is the sum of the first k+1 numbers”
We calculate the value of S at the end of this step
K:=k+1, s:=s+a[k+1] => s is the sum of the first k+1 numbers
 Example: Proving the correctness
of the Sum algorithm (3)
• Induction hypothesis: S= sum of the first k
numbers
3. Termination: When the loop terminates, the
hypothesis implies the correctness of the algorithm
The loop terminates when k=n=> s= sum of first k=n
numbers => postcondition of algorithm, DONE
 Loop invariants and induction

• Proving loop invariants is similar to

mathematical induction:
 showing that the invariant holds before the first
iteration corresponds to the base case, and
 showing that the invariant holds from iteration to
iteration corresponds to the inductive step.
 Correctness of algorithms

• Induction can be used for proving the

correctness of repetitive algorithms:
 Iterative algorithms:
o Loop invariants
 Induction hypothesis = loop invariant = relationships between
the variables during loop execution
 Recursive algorithms
o Direct induction
 Hypothesis = a recursive call itself ; often a case for applying
strong induction
 Up Next

• Asymptotic Notations