2011 PMOC Annual Meeting
FEDERAL TRANSIT ADMINISTRATION
Risk Review Process Basics
Michael P. Wetherell, PE - Urban Engineers
David N. Sillars, PE Sillars Consulting
FEDERAL TRANSIT ADMINISTRATION
2011 PMOC Annual Meeting
Background
As with many large, public projects, FTA
experienced a significant amount of project cost
overrun
In this environment of overrun risk, planning a cost
budget has been difficult
The risk review process is intended to:
Inform the FTA about the projects risk
Provide the grantee with recommendations to
strengthen the project
Uncertainty
Uncertainty is the only certainty there is, and
knowing how to live with insecurity is the only
security.
- John Allen Paulos
The fundamental assumption is the grantee may
decrease the level of uncertainty by studying the
problem, exposing the risks, and taking actions to
mitigate known risks
Certainty and Uncertainty
No Information Partial Information Complete Information
Unknown unknowns Known unknowns Knowns
Scope of project risk assessment
General Specific
Total Uncertainty Total Certainty
Uncertainty Uncertainty
Risk Review vs. Risk Assessment
Risk Review
FTA provides a review of the risk and contingency plan
within a Grantees project management plan
This review evaluates the 4 fundamental parts of risk
management, i.e., Identification, Assessment, Response,
Management
Risk Assessment
1 of the 4 parts that quantifies the impact of all risks
FTA treatment of risk assessment depends on the projects
size
Risk Management Process
Identifying risk situations for which outcomes
are likely to vary in amounts that may
significantly affect the projects goals
Assessing the likelihood and magnitude of
such variance
Responding by developing plans for
minimizing hazard and maximizing gain
Managing the execution of those plans
Risk Review Benefits
Reviews an agencys technical capacity and
capability to perform the project
Identifies and develops action plans for risky
events
Creates informed assessment of likely cost
and schedule outcomes
Assures plans are in place to protect the
project
FTA Risk Review
Oversight Procedures (OPs) prescribe fundamentals of the risk
review process
Grantee
Technical
Capacity
(OP-21)
Schedule
Grantees Review Grantees
Project Project Scope (OP-34) Project
PMP Review Risk Review
Management and Capacity Management
(OP-20) (OP-40)
Plan (OP-32) Plan
(PMP) (PMP)
Estimate
Review
(OP-33)
Monitoring Reports (OP-25)
Presentation Outline
Project Status Evaluation and Risk Assessment
Project Status Evaluation Procedures
Project Cost and Schedule Reviews
Risk Identification
Risk Categories
Risk Register
Assessment Evaluation and Risk Management
Overview of Beta Model Development/ Interpretation
Overview of Cumulative Cost Risk Curve/ Schedule Risk
Curve
Risk Mitigation
Risk and Contingency Management Plan (RCMP) Basics
Project Status Evaluation
Technical capacity and capability
Project capacity and scope
Cost estimate
Project schedule
Grantee
Technical
Capacity
(OP-21)
Schedule
Grantees Review Grantees
Project Project Scope (OP-34) Project
PMP Review Risk Review
Management and Capacity Management
(OP-20) (OP-40)
Plan (OP-32) Plan
(PMP) (PMP)
Estimate
Review
(OP-33)
Monitoring Reports (OP-25)
The project status evaluation is a precursor to the detailed Risk Review
Risk Review Meetings
The Risk Review process involves 3 primary
review meetings between the grantee, the
PMOC, and FTA before each evaluation
Supplemented by working groups designed to
develop the details of the risk review
Its important each meeting is attended by those
most able to provide seasoned, professional input
Risk Review Meeting 1
Introduce PMOC team and grantee team
Grantee presents the project to PMOC team
Agency organization and project team plan for staffing
Description of work and reviews over the previous year
Review of the project by discipline
Review of schedule, cost estimate, and risk register, if
available
PMOC presents the risk process to grantee
Tour of alignment, station, and support facility
locations
Project Status Evaluation
Fundamental project documents must be provided to the
PMOC to complete the project status evaluation
Program documents
Project agreements
Design documentation
Cost estimates, with contingencies exposed
Schedule, with contingencies exposed
If these elements are flawed or incomplete, risk
assessment may be undermined before it even begins
Technical Capacity and Capability
Evaluates grantee management, organization, and project
definition data
Requires:
Project program information
Interviews of grantees organization
Grantee Project Management Plan (PMP)
Assesses technical capacity and capability to implement
the project in conformance with the Development Strategy
Provides recommendations that should be addressed in
the PMP
Project Capacity and Scope
Evaluates the projects operating capacity, in terms of line capacity
and ridership
Includes current and projected ridership
Considers phasing of facility
Requires:
Project program, alignment, and ridership information
Capacity calculations (like TCRP-100 ridership capacity worksheets
and station stop worksheets, if used by Grantee, etc.)
Identifies gaps (or requirements risks) in the forecasted
balance between potential ridership and system capacity
Project Capacity and Scope
Assesses the physical components of the project scope and
how those components relate to their surroundings
Requirements for assessment depend on the projects phase
Compares against project description and initial goals
Requires:
All program and design documents and investigative reports
Site evaluation visits
Scope checklist
Cost Estimate Review
Review basis for Escalation Factor(s)
Standard Indices
Basis for Assumptions
Review grantees project cost estimate and general
conditions in base year and escalated dollars
Requires grantees detailed cost estimate and back-up,
Standard Cost Category (SCC) workbook, and project
scope review results
Provides a mathematical check of the estimate
May include recommendations for adjustments to the
grantees budget estimate
Project Schedule Review
Reviews grantees project schedule
Requires grantees schedule, SCC workbook,
and project scope review results
Provides evaluation of schedule
Continues discovery of items that are
uncertain and constitute risks
Project Status Evaluation
Upon completion of the Project Status
Evaluation, FTA (through reports from the
PMOC) has a thorough understanding of the
project, grantee, cost and time required to
complete the project, and uncertainties to be
resolved
Information revealed will be used by FTA for
programmatic decisions and will form the basis
for identification of risk events
Risk Identification
Accumulate issues Categorize by SCC code
discovered through the Categorize by risk type:
various reviews and Requirements
identify potential risk Design
events Market
Develop a Risk Register Construction
listing of risk events
Risk Identification
Risk identification is a collaborative process
Purpose is to objectively discover potential
threats to the projects goals
Starting points for risk event identification:
1. PMOCs listing of issues discovered through the
project status reviews
2. Grantees own risk analysis (if undertaken)
Risk Review Meeting 2
PMOC summary reviews of technical capacity and
capability, scope, cost, and schedule
Discuss risks identified by PMOC and record
additional risks discovered during the workshop
Enter into discussions with grantees project team
to resolve open questions
Discuss actions required to facilitate the next
stage of quantitative risk assessment
Inform grantee of next steps
Key Individuals
Project owner
Staff of project owner
Management staff, preferably those who participated in
any earlier risk reviews
Appropriate technical staff (e.g., operations staff)
Senior members of the project team
Including cost, schedule, design experts, etc.
Participating or affected agencies
If risk mitigation measures have been allocated to
contractors, their management and technical staff
should be involved
Risk facilitator
Risk Factoring
Probability (P) chance of occurrence
Magnitude (M) value
These 2 factors determine the expected value
(E) of a risk event
E = P*M
Risk Factoring
It is helpful to provide guidance regarding
what constitutes each category of probability
or magnitude assessment
One possible risk factoring:
Assessment Low (1) Medium (2) High (3)
Probability < 10% 10% 50% > 50%
Cost ($) < $1.0 million $1.0 $5.0 million > $10.0 million
Time (months)
< 1 month 1 12 months > 12 months
potential delay
Risk Register
An initial evaluation of the expected value of each risk is made
Risk Register
Identification Assessment
Cost Schedule
Probability Expected Expected
Magnitude Magnitude
(A) Value Value
(B) (C)
SCC Code
Risk event Risk
description Type 3 = high 3 = high 3 = high
Schedule
2 = medium 2 = medium 2 = medium Cost Score
Score
1 = low 1 = low 1 = low
4
Risk Review
Risk assessment by PMOC
after risk identification
workshop with Grantee
Grantee
Technical
Capacity
(OP-21)
Schedule
Grantees Review Grantees
Project Project Scope (OP-34) Project
PMP Review Risk Review
Management and Capacity Management
(OP-20) (OP-40)
Plan (OP-32) Plan
(PMP) (PMP)
Estimate
Review
(OP-33)
Monitoring Reports (OP-25)
FTA Standard Cost Categories (SCC
Codes)
10 Guideway and Track Elements
20 Stations, Stops, Terminals, Intermodals
30 Support Facilities: Yards, Shops, Administration Buildings
40 Sitework and Special Conditions
50 Systems
60 Right-of-Way, Land, Existing Improvements
70 Vehicles
80 Professional Services
90 Unallocated Contingency
100 Finance Charges
Risk Assessment Steps
1. Strip the estimate and schedules of all contingencies
2. Summarize the costs and activities
3. Apply ranges of cost and durations to the summaries
4. Calculate the range of cost and schedule for the
project
Cost and Schedule Risk Assessment
1. The schedule and cost estimate are
scrubbed of contingencies and adjusted by
the PMOC
Identify and remove hidden and exposed
contingencies
Remaining schedule and cost estimate are used to
forecast risk on the project
After the first step, the cost and schedule assessment processes differ
Cost Risk Assessment
2. Summarize the grantees scrubbed and
adjusted estimate into the SCC
SCC Category 10%
SCC 10 Guideway
10.01 Guideway: At-grade exclusive right-of-way 42,501
10.04 Guideway: Aerial Structure 51,900
10.11 Track: Ballasted 45,176
SCC 20 Stations, Stops, Terminals, Intermodals
20.01 At-grade station, stop, shelter, mall, terminal, platform 45,122
20.02 Aerial station, stop, shelter, mall, terminal, platform 46,551
20.06 Automobile parking multi-story structure 215,380
20.07 Elevators, escalators 9,248
Cost Risk Assessment
3. A range multiplier (Beta) is applied to the
scrubbed and adjusted estimate, on a
category-by-category basis, to yield the range
from highly optimistic (10%) to highly
pessimistic (90%)
The range multiplier varies depending on the
phase of project development
FTA provides recommended values for the Beta
factor
Cost Risk Assessment
How is the Beta chosen?
Depends on what phase the elements of the project are in: the earlier in
the process, the more unknowns and the higher the risk
4 stages of project completion are considered important; completion of
each phase successively reduces risk
Stage of Project Completion Risk Example
Unknown stakeholder requirements for
Requirements definition Requirements risk
station
Project design Design risk Vertical alignment issues
Construction contracts procurement Market risk Significant spike in cost of steel
Unforeseen conditions, project
Project construction Construction risk
interference
Cost Risk Assessment
Requirements Risk,
=?
Note: Beta () is a factor
applied to the base cost
2.50
estimate indicating amount
of risk.
Design Risk, =0.50
2.25
Cumulative Risk Beta
2.00
Market Risk, =0.25
1.75
Early, =0.40
1.50
Constr. Risk, =0.70
Mid, =0.15
1.25
Late, =0.15
1.00 Early Mid Late
Design
Concept Bid
Project Phase Construction
Cost Risk Assessment
Sample Beta assessment
Post
Requirements Design Marketing Construction
SCC Category 10% Construction Total Beta
Beta Beta Beta Beta
Beta
SCC 10 Guideway
Guideway: At-grade exclusive
10.01 42,501 0.00 0.50 0.25 0.70 0.05 2.50
right-of-way
10.04 Guideway: Aerial Structure 51,900 0.00 0.50 0.25 0.70 0.05 2.50
10.11 Track: Ballasted 45,176 0.00 0.50 0.25 0.70 0.05 2.50
Stations, Stops, Terminals,
SCC 20
Intermodals
At-grade station, stop,
20.01 shelter, mall, terminal, 45,122 0.00 0.50 0.25 0.70 0.05 2.50
platform
Aerial Station, stop, shelter,
20.02 46,551 0.00 0.50 0.25 0.70 0.05 2.50
mall, terminal, platform
Automobile parking multi-
20.06 215,380 0.00 0.50 0.25 0.70 0.05 2.50
story structure
20.07 Elevators, escalators 9,248 0.00 0.50 0.25 0.70 0.05 2.50
Note: Total Beta includes starting base of 1.0
Cost Risk Assessment
4. Categories of the project are then aggregated
probabilistically to yield likely project
outcomes
PMOC will use the Cost Risk Assessment
Workbook to simplify the process of project risk
assessment range modeling
Cost Risk Assessment
PMOC uses the OP-40 Cost Risk Assessment Workbook
Consists of several parts:
Conditioned
base estimate, PMOC
in standard SCC adjustments to Adjusted YOE
categories raw YOE estimate Betas
Betasassigned
assigned
estimate without totoBetas
Betasassigned
SCC
SCC assigned
contingency to SCC
Categories
to SCC by
Categories
Categories byby
phase Categories by
phase Risk reduction
phase
phase graph
Cost Risk Assessment
Cost Risk Assessment
Project is forecasted out to each stage of development to identify
potential risk reduction; the results are plotted across the project life-
cycle, as if all mitigation is successful
Cost Risks
Completion, phase by phase >>>>>>>>
Project Project
Conception Completion
Schedule Risk Assessment
2. A summary schedule is developed with a
reasonable number of activities
Activity One
Start
10 12 16
Criticality = 63% Finish
Criticality = 37%
Activity Two
8 10 8
Schedule Risk Assessment
3. The duration range is established based on the risk register
and applied to the summarized schedule
MOST
Orig Dur Rem Dur % Comp Early Start Early Finish OPTIMISTIC PESSIMISTIC
LIKELY
Pico/Aliso Station
Segment at grade works
9815 Segment 2 Utility diversions and duct banks 225 71 68 30NOV05A 10DEC07 65 75 93
9835 Segment 2 1st Street bridge track bed/track 129 129 0 08JAN07A 28FEB08 100 129 149
9819 Segment 2a/b Complete systems installation 221 221 0 11DEC07 14OCT08 199 221 243
9820 Segment 2 Sequencing/resource constraints 20 20 0 5OCT08 11NOV08 5 20 40
Station
980 Pico/Aliso Final design and MTA approvals 118 0 100 30NOV05A 11MAY06A
981 Traffic plan approvals and street closures 61 0 100 15MAY06A 31JUL06A
9810 Peco/Aliso station demo and structure 95 47 51 01FEB07A 06NOV07 40 52 69
9830 Peco/Aloso MEP and Str restore and punch list 190 190 0 07NOV07* 29JUL08 180 200 212
Schedule Risk Assessment
4. Range-loaded activities are then simulated
(using Monte Carlo methods) to yield likely
project outcomes
The results are reviewed and rerun as necessary
for schedule integrity
The results may affect the cost analysis
FTA Schedule Risk Assessment
The results indicate the likelihood of meeting the target completion date
*See back of module for larger graphic
Risk Review Meeting 3
Describe process used to develop quantitative risk
assessment
Summarize key findings of the assessment
Provide recommendations regarding risk
Review which risks strongly influence overall project
risk
Review specific recommended mitigation measures
and solicit completion dates
Discuss action items and next steps
Risk Response
Risk response
recommendations
and planning
Grantee
Technical
Capacity
(OP-21)
Schedule
Grantees Review Grantees
Project Project Scope (OP-34) Project
PMP Review Risk Review
Management and Capacity Management
(OP-20) (OP-40)
Plan (OP-32) Plan
(PMP) (PMP)
Estimate
Review
(OP-33)
Monitoring Reports (OP-25)
Risk response
monitoring
4 Risk Responses (Risk Aversion Order)
1. Avoidance
Pursue a different project or project element
Alternatives Analysis is used here
2. Transfer
By contract or agreement with the performing parties
Major concepts to be considered:
Efficient allocation of risk
Equitable allocation of risk
4 Risk Responses (Risk Aversion Order)
3. Reduce
Modify either the probability (P) or magnitude
(M), or both, reducing the expected value (E) of
the risk
E = P*M
Modify the design or PMP
Protect against theft or damage
Influence the projects environment
4 Risk Responses (Risk Aversion Order)
4. Retain/absorb
Recognize a risk event, but take no action
If it appears to be of minimal consequence
Retain it and protect against it
Where risks are unavoidable
Establish contingencies that can absorb realized risks
Develop a Plan B in case risks become realized
Risk Response Recommendations
Two types of information are used:
1. Qualitative information is used to develop
mitigation action plans for the grantee to
undertake, through the use of Primary
Mitigation
2. Quantitative information is used to establish Risk
Protection against the inherent project risk
through the use of Contingencies and
Secondary Mitigation
Primary Risk Mitigation
Executed by the grantee through completing
efforts described in the grantees most current
Risk Contingency Management Plan in the PMP
Grantees plans should be consistent with the risk
review and mitigation recommendations
Work is monitored on a continual basis during the
PMOCs monthly monitoring and oversight
Failure to adequately complete work translates to
greater uncertainty as the project moves forward
Risk Protection
Risk protection is approached from 2 directions:
Contingency budgeted amounts to overcome
normal overruns
Secondary Mitigation development of plans to
recover cost or schedule overruns using redesign
techniques where contingency is insufficient
Cost Contingency
FTA risk review provides quantitative modeling to assess
the sufficiency of the grantees budget at various project
phases
Specific target values are identified at primary phases:
Entry to Preliminary Engineering
Entry to Final Design
FFGA
Construction
The target values vary, depending on the project phase
current at the time of calculation
Cost Contingency Targets
Cost Contingency Targets
35%
30%
25%
20%
15%
10%
5%
0%
PE Phase FD Phase FFGA Award 100% Bid 50% Construction
Cost Contingency
An additional analysis of specific, potential
project risks is conducted to determine how
contingency must be maintained and at what
amounts throughout the project completion
This is called the backward pass method
This analysis identifies points during the
construction period at which certain
minimums of contingency should maintained
to protect the project;
Schedule Contingency
FTA also established criteria for suggested
minimum schedule contingency levels
The targeted amount of schedule contingency,
for the project as a whole, is 20% of the
stripped duration from Final Design through
Construction completion
Schedule Contingency
In an analysis similar to that for cost
contingency, a separate, detailed analysis of
minimum schedule contingency levels occur
This analysis identifies points during the project
phases at which certain minimums of schedule
contingency must be maintained
Cost and Schedule Contingency
Contingency needs are analyzed on an individual
project basis by looking at specific risks
Contingency is not released until these risks are
no longer of concern
Project Milestones indicate the times and
amounts at which the contingency minimum may
be reduced
Final recommendation is developed by PMOC in
consideration of 3 forecast methods
Secondary Mitigation
To determine the amount of capacity a grantee
should maintain, the FTA establishes targets for
specific phases of the project
Secondary Mitigation does not reduce the
functionality of the initial project
At a given stage of a project, the grantees
estimate (including contingency) is compared
against a target point within the potential range
of cost risk
The difference is called the gap
Secondary Mitigation
Target @ Entry
into FD=
30% likely value
Mitigation in
amount of gap
must be available
Grantees
estimate
including
contingency
Risk Management Planning
General process for risk management begins with a written
Risk and Contingency Management Plan (RCMP):
Documentation of the various analyses
Listing of major project risks, their description and the
nature of their consequences
How they are to be handled (by whom, when)
Watch List
A list of secondary and/or residual minor risks that need to be
kept an eye on
Timing for re-examination of the Risk Management portion
of the PMP
RCMP Requirements
Section of the Grantees Project Management Plan (PMP)
Fully updated and active PMP
Highlight specific areas of management focus as identified
through the risk review process
Means for monitoring Grantees progress
Actions to strengthen technical capacity and capability, project
performance, cost and schedule analyses, mitigations of identified
project risks, and others.
Information contained within the RCMP should complement
and not be in conflict with information contained elsewhere
within the PMP or in other FTA guidance documents
RCMP Overview
Topics:
Primary Mitigation, organized by significant
project activities, such as:
Technical Capacity
Project Scoping and Design;
Delivery Methods and Contracting;
Construction Process;
Project Tracking, including:
Cost Estimating, Financing and Financial Management;
and
Project Schedule Management.
RCMP Overview
Insurance:
Professional services, construction phase, wrap-up, or other specialized
insurances purchased for reduction of risk exposure
Contingency Management:
Cost Contingency Management Plan; and
Schedule Contingency Management Plan
Secondary Mitigation:
Establishment of Secondary Mitigation actions and cost targets which
may trigger the implementation of Secondary Mitigation
Risk Management:
Risks management and mitigation monitoring, change identification,
and management controls
Risk Management Planning
Risk Management Plan is a combination of elements within the
grantees PMP and its sub-parts
Primary Mitigation occurs through professional planning, design,
and execution
Risk Protection involves planning for the what-ifs (e.g., What if
poor soils unexpectedly arise?, What if the market tightens?)
Written plans to handle project overages result in a well-
managed response to any crisis and should be documented
within the PMP
Plans are updated as the project progresses
Risk Management Process
The process of:
Identifying risk situations, for which outcomes are likely to
vary in amounts that may significantly affect the projects
goals
Assessing the likelihood and magnitude of such variance
Establishing plans for minimizing hazard and maximizing gain
Managing the execution of those plans
A rational means for working in an uncertain world
Risk workshops and meetings
Meeting Purpose When Attendance Note
Introduction Introduce team, Before formal risk Project May be reduced
get to know the workshops management, during risk
project, discuss risk technical experts, refreshes
process lead designers
Risk workshop #1 Establish current After scope, cost, Project If Grantee has a risk
risk register, schedule etc. reviews. management, process, focus on
evaluate risks, PMOC list of risks technical experts, suggested
identify potential required. lead designers amendments to
mitigations Grantees risk
register
Risk workshop #2 Discuss findings After PMOC risk Project
from PMOC risk modeling and after FTA management
modeling; next concurrence with
steps recommendations;
discuss key mitigations
Risk workshop #3 Discuss mitigations After workshop #2, Project May not be
(optional) if not discussed before finalizing OP40 management, required if covered
earlier report technical experts, in workshops #1&2
lead designers
