Model Risk Management

Driving the value in modelling

May 2018, Risk Advisory
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Agenda

CONTEXT 1

APPENDIX 6

2018 Deloitte 2
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Part 1
Context

2018 Deloitte 3
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

How important is model risk ?

Model risk may be particularly high, especially under stressed conditions
or combined with other interrelated trigger events.
LTCM – Arbitrage investment strategies
Impacts: the hedge fund lost $4.4bn in 1998,
JP Morgan – The London Whale depleting almost its entire capital CDO / MBS – 2007 subprime mortgage
Impacts: the bank made losses of £6bn and was fined What happened ? The hedge fund was established by crisis
£1bn renowned bond traders and the main shareholders
Impacts: one of the main cause and source of losses in
What happened ? The bank’s Chief Investment included Nobel prize-winning economists (Myron
the 2007 financial crisis. As-of Sept. 2008, bank write-
Officer was responsible for investing excess bank Scholes and Robert Merton). Investors consisted in
downs and losses totaled $523bn.
deposits in a low-risk manner. To hedge against high net worth individuals and in financial institutions.
possible downturns in the economy, the CIO bought The fund had followed an arbitrage investment strategy What happened ? Rating agencies had provided a AAA
synthetic CDS derivatives. Initially intended as an on bonds, involving hedging against a range of rating to a significant portion of securities backed by
hedging strategy, this portfolio became a speculative volatility in foreign currencies and bonds, based on pools of loans including a significant proportion of loans
source of profit and increased from $4bn in 2010 to complex models. to homebuyers with bad credit and undocumented
$157bn in early 2012. However, the internal risk How is model risk involved? Arbitrage margins are incomes (subprime mortgage loans)
controllers duly reported those trades as being too small and the fund took on leveraged positions to
risky. maintain or increase profits. At one point, the notional How is model risk involved? Between 2002 and 2007,
the mortgage underwriting standards had significantly
How is model risk involved? Instead of scaling back value of the derivative position was $1.25tn. When the
deteriorated. However those loans bundled into MBS and
the risk, the bank changed its VaR metric in early Russian crisis kicked off in 1998, European and US
CDO with high ratings which were believed justified by
2012. But there was an error in the spreadsheet used markets fell drastically and LTCM was badly hit through
credit enhancement techniques. Investors relied on rating
for that purpose and the risk was understated by 50%. market losses and fire sales.
agencies, blindly in many cases. However, a significant
This error enabled the portfolio to continue growing,
portion of AAA CDO and MBS tranches were finally
but the bank was then hit by the European sovereign
downgraded to junk in 2007 and early 2008, once the
debt crisis.
housing bubble burst in the 2006 H2.
The US Financial Crisis Inquiry Commission found that
agencies' credit ratings were influenced by "flawed
Market risk regulatory pre-crisis models computer models, the pressure from financial firms that
Impacts: the VaR metrics used before the outburst of the financial crisis did not adequately capture paid for the ratings, the relentless drive for market share,
tail-risk events, credit risk events as well as market illiquidity. the lack of resources to do the job despite record profits,
and the absence of meaningful public oversight”.
What happened ? When the financial crisis arose, essentially driven by credit risk events, a large
number of banks posted daily trading losses many times greater than their VaR estimates and quite
frequently during that period, in a context where some financial markets became largely illiquid.
How is model risk involved? The market risk model was build upon assumptions that were not
reflective of the real world in stressed financial markets (assuming market liquidity and large
diversification effects across asset classes, etc.). In addition, tail credit risk events were not
adequately modelled, hence underestimating possible losses in stressed conditions.

2018 Deloitte 4
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Main regulatory references on MRM

CRD IV/CRR Bank IT Circular

OCC-Fed 2013 – 36 285/2013
OCC 2000-16 2011-12 BCBS IRRBB
Technical criteria Management Body
First Definition 2006 CEBS GL 10 concerning the must understand all TRIM Guide Interest rate risk
SR - 11 - 7
of models and New Validation organisation and of the business risks, in the banking
First Supervisory Feb-2017
model risk Requirements treatment of risks including model risk… book
Guidance on MRM

What about
the future
regulatory
framework?

BCBS 2004-06 BCBS 2010-11 EBA RTS 2013 EBA SREP TRIM PRA ECB Guide to
Valuation Introduction of a on Prudent CP/2014/14 Stress Test Model Internal
adjustments […] Leverage Ratio as Valuation Integration of RTS/2016/03
Management Models
where appropriate, a safeguard Valuation Model Risk as Structure of 3
Principles Implementation
model Risk. against Model Risk adjustments on part of Pillar II lines of defence
of a model risk
MR Quantification management
framework

In Europe, regulatory bodies confirmed that banks need to implement a Model

Risk Management comprehensive framework.

2018 Deloitte 5
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Regulatory references in the EU

CRD IV / CRR Defines Model Risk (Art. 3.1.11) and the process by which the Competent Authorities should
assess how the institutions manage and implement policies and processes to evaluate
the exposure to Model Risk as part of the Operational Risk (Art. 85).

The ‘Guidelines on common procedures and methodologies for the supervisory review and
Guidelines on evaluation process’ define the main activities that the Competent Authorities should
SREP assess in the institution’s exposure to model risk arising from the use of internal
models in its main business areas and operations. In particular, the Competent Authorities
should consider to what extent, and for which purposes, the institution uses models to make
decisions and its level of awareness (Management Body and Senior Management) of and
how it manages model risk.

According to SREP Guidelines, the model risk can be split into two distinct
forms of risk with two different impacts risk profiles.
Form of risks Risk profile
“Competent authorities should consider the model
"Risk relating to the underestimation of own funds
risk as part of the assessment of specific risks to
1 requirements by regulatory approved models
(e.g. internal ratings-based (IRB) models for credit
risk)”
capital (e.g. IRB model deficiency is considered as
part of the credit risk assessment) and for the
capital adequacy assessment”
“Risk of losses relating to the development,
“Competent authorities should consider the risk as
implementation or improper use of any other
2 models by the institution for decision-making
(e.g. product pricing, evaluation of financial
part of the assessment of operational risk” and it
should be evaluated within this perimeter
instruments, monitoring of risk limits, etc.)”

2018 Deloitte 6
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

ECB Guide to internal models

The guide was drafted in close cooperation with the national competent authorities (NCAs) and draws on
the experience gained in the context of the targeted review of internal models (TRIM) project. The guide
mainly focuses on: overarching principles, internal model governance, internal validation, internal
audit, model use, model change management and third-party involvement.

Effective model risk management allows institutions to reduce the risk of potential losses and underestimation of own
funds requirements as a result of flaws in the development, implementation or use of the models. To mitigate these
risks, institutions should have a model risk management framework in place that allows them to identify, understand and
manage their model risk for internal models across the group.

Elements of an MRM framework Extract from ECB Guide in Internal models

“This policy should include a definition of a model, provide the
A written model risk management
(a) policy
institution’s interpretation of model risk and describe the model risk
framework with reference to its different components.”
“This register should facilitate a holistic understanding of the application
A register of the institution’s internal
(b) models
and use of the models and provide the institution’s management body
and SM with a comprehensive overview of the models in place.”
“Identifying and mitigating any areas where measurement uncertainty
(c) Guidelines on mitigation actions and model deficiencies are known to exist, according to their materiality,
applied consistently across the Group.”

Guidelines on model risk “Methodologies for the qualitative and/or quantitative assessment and
(d) quantification measurement of the institution’s model risk.”

“The model life cycle includes the following steps: requirements analysis,
Guidelines with respect to the model
(e) life cycle
development, implementation, testing, use, validation, maintenance and
changes.”

Model risk communication and “Procedures for model risk internal and external communication and
(f) reporting reporting.”

Definition of roles and “Definition of roles and responsibilities within the model risk
(g) responsibilities management framework.”
7
2018 Deloitte
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Impact of the New Regulations and Standards

• Impact FRTB
The FRTB includes updates to both the advanced and
standardized models as well as stricter disclosure
requirements and validation standards. More More More
Data Models Interpretations
• Impact IRB
EBA Guidelines on PD, LGD estimation and treatment of
defaulted asset as well as new default definition, conservatism
margins, NPL assessment, rating process. More
Auditability
• Impact of Stress Testing Governance
More More Internal
New stress testing methodology and principles defined by the Often and Controls Cooperation
PRA and EBA.

• Impact of IFRS9
The introduction of the IFRS 9 Impairments standard is
demanding that banks use a new set of credit risk models;
these models must be developed, deployed and maintained,
which will literally double the number of Risk parameters More More Financial More Complex
models to manage.
Impairments Impact Calculations

2018 Deloitte 8
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Model Risk: the next risk type

STRENGHTHEN INCREASING
MODEL RISK MRM
GOVERNANCE
AWARENESS
Regulators want to have a core understanding
MODEL LIFECYCLE of the way banks develop, document, use,
EFFICIENCY monitor, set up and maintain inventories,
validate and control models for credit, finance
and marketing activities.

• Faster model deployment

MODEL RISK
SCOPE EXTENSION • Streamlined processes
• Centralised modelling infrastructure

• Banks need to develop more models in order to comply with parallel regulations, typically :
• IFRS9 Framework Implementation and Forward Looking Integration
• FRTB, IRB Models and TRIM
• ECB Regular Stress Tests
• Banks should consider Model Risk limits within Risk Appetite Framework.
2018 Deloitte 9
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

MRM for better business decisions

 Banks are increasingly using decision models in their credit processes such as origination, limit
management, collections and recoveries. In the commercial area, customers are able to select a product’s
characteristics and the system makes a real time decision on viability and price.
 Customer on boarding, engagement and marketing campaign models have become more prevalent for
establishing customer loyalty and engagement actions in all stages of the relationship with the institution
and at any time in the customer life cycle.
 Another area is fraud and money laundering detection.

Therefore, a clearly defined MRM framework with a strong management insight on monitoring models and
their risks will allow institutions to strengthen their decision making processes and improve their
profitability.

Access to trusted, quality models is essential to effectively using enterprise data − now considered a
1 strategic asset − to drive better decision making and business results.
• Banks are heavily dependent on models to help them make the best decisions and navigate an increasingly
competitive landscape. Banking executives, for example, are expected to rely on analytical models − not
just gut instinct and experience − when making decisions about deploying capital in support of lending and
customer management strategies.
As limited expert resources are often an issue in financial institutions, it is important to handle existing
2 resources in the most cost-efficient way.
• In order to achieve cost efficiency, model risk activities are prioritized and conducted for portfolios that are
of higher importance, i.e. that contain strategically relevant positions with substantial position size,
significant risk contribution or complex risk profiles.
3 Model risk management should add value to the enterprise as well as reduce risk.
• Visibility into the source of data, confidence in the reliability and applicability of the model, and ongoing
model improvements all support more effective decision-making for the organization, ultimately protecting
its financial position and reputation.
2018 Deloitte 10
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Part 2
MRM
Contents

2018 Deloitte 11
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Definitions

Model
A quantitative method* or system that applies theories to process input data into quantitative estimates for
decision making (used repeatedly).
• Data • Statistical • Forecasts
• Inputs • Financial • Estimates
• Assumptions • Mathematical • Management
Inputs • Scenarios Model • Economic Outputs decision support

*Including also the complex manipulations of expert judgements.

Model Risk
Model Risk can be defined as the potential loss an institution may incur, as a consequence of decisions that
could be principally based on the output of (internal) models, due to errors in the development,
implementation or use of such models. (CRD IV, Article 3.1.11)

Model Risk Management Framework

Regardless of the organization’s size and structure, regulators require that enterprise MRM frameworks
encompass all relevant aspects of the MRM life cycle with clearly assigned roles and responsibilities:
• Model Risk Identification and Assessment,
• Model Risk Measurement and Mitigation,
• Model Risk Monitoring and Reporting.

2018 Deloitte 12
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Types of Models in the Scope

Comprehensive Model Coverage
A large global bank has a wide range of model types that are subject to governance and model risk
management.

Market and Liquidity Credit & Counterparty Operational Risk Compliance

Risk Models Risk Models Models Models
• VaR (inc. Stressed • PD, LGD and EAD • Loss Distribution • Anti-Money
MODELS USED VaR, IRC) • Risk rating models Approach Model Laundering (AML)
• ALM & Liquidity Risk • Exposure and CVA • Integration Model • Anti Fraud
FOR
• Expected Shortfall • IFRS 9 Impairment • Trader surveillance
« REGULATORY,
MANAGERIAL Portfolio & Financial Decision Support Valuation & Pricing Finance Models
AND Risk Models Models Models
ACCOUNTING »
• Capital forecasting • LOB models for • Derivatives • P&L Attribution
PURPOSES • Stress testing customer targeting- • Structured products • Cash flow /NPV/Ratio
• Econometric models marketing • Risk based pricing Analysis
• Credit underwriting tools/models
• Risk based collection
models

Marketing Models Insurance Models Investment Other Models

Management
MODELS USED
FOR OTHER • Marketing models • Actuarial models • Trading • Corporate Finance
PURPOSES • Client Targeting • Loss Forecasting • Security / Asset Models (e.g. M&A,
• Reserving models Pricing LBO, MBO)
• Portfolio Allocation

2018 Deloitte 13
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Elements of an objective MRM framework

Organisation and Governance Model Lifecycle Management
• Existence of a Model Risk Management • Includes the model development,
that has been approved by the Board, who documentation, classification,
receive periodic reports regarding inventory and follow-up:
compliance.
o Comprehensive inventory
• Existence of a Model Risk function that covering all existing models and
reports directly to the CRO and is framework scope.
responsible for the MRM framework and the
o Models are classified according to
governance.
the level of risk.
• Existence of a model validation function
o The documentation should include
responsible for the independent
description, key variables,
validation of models.
assumptions and algorithms.

Model Risk Quantification Model Control Framework

• Quantitative techniques for model risk • Models assigned the highest level of risk
mitigation (beyond regulatory are subject to continuous assessment.
requirements) regarding:
• In addition to the above, all models should
o Data, sensitivity to errors or be re-evaluated by Validation:
absence of variables;
o Annually.
o Estimates, sensitivity of estimates
o If they undergo material changes.
(maximum impact, alternative
models); Model Risk Management • Before they are deployed to production,
Processes & Technology they should have been approved.
o Uses, predictive power evolution,
impact of erroneous use, etc. The MRM framework should be
supported by optimized
processes and a technological
2018 Deloitte platform. 14
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Strategic MRM benefits

An effective and robust MRM framework improves earnings
through cost reduction, loss avoidance, and capital improvement

Increased operational and Reduction of undue capital

process efficiency Risk Capital buffers
reduction improvement
Optimized and automated key Effective management of
modelling processes excessive conservatism

Model risk appetite alignment Control over fragmented

Model risk Process model ownership
Investments based on risk- appetite efficiencies
appetite limits Increasing model
Model risk transparency
management
framework Institutional risk culture

Resources alignment
Improved Control &
resource quantify-
Optimized resource
allocation cation
management
Improving insights based on
different factors

Common understanding Transparent Alignment

across all lines of defence communication of goals
Facilitates alignment of
Standardized reporting, common goals and priorities
continuous monitoring, and throughout the model life
transparent reports cycle

15

2018 Deloitte
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Building Model Risk Management

Given the risk appetite and ambition level for model risk management,
building the framework consists of three maturity stages

Stage 1 Stage 2 Stage 3

Foundation Implementation Integrated MRM

Lay the foundation of MRM framework Implement the key functions of MRM MRM value creation via integrated
platform

Governance, • Strategic integration with Model

• Basic MRM policy • Updated MRM control
Policies risk appetite
• Roles and responsibilities framework and rationalization
and Controls • Continuous MRM monitoring
• MRM risk awareness & mind-set

Development • Model definition • Basic model categorization • Continuous model risk

Implementation • Periodical model discovery and model risk quantification quantification enhances MRM
and Use • Basic model inventory analytics and reporting and
capabilities

Model • Documented validation • Periodical coordination • Clear model prioritization

Validation procedures between MRM, development throughout organization
process • Separate model development and validation processes • Optimal use of process
and validation team automation / robotics

• Scattered platforms • Basic, but centralised • Integrated MRM platform

MRM
• Start with tooling
platform
requirements and vendor
selection

16

2018 Deloitte
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

The Future of MRM: Approaching a Steady State

Line of Defence Line of Defence Line of Defence

1st Model Development / Use
2nd Model Validation / Control
3rd Internal Audit

o Complete ownership of Model o Expanding the Coverage of o More focused on processes

Risk as an exposure class Models and controls
o More robust and automated o More focused on model o Less focus on model-level
controls around model validation rather than content (e.g. mathematics,
development and use development (i.e. no co- theory)
o Performing more vigorous development) o More focus on the first line –
model testing during the o Enhanced focus on efficiency development, documentation
implementation phase (core teams supplemented by implementation & use of
o Ongoing monitoring of seasonal pools, and/or models
models performance offshore resources) o Continuous / BAU instead of
o Post implementation and o Efficiency through the use of event-based
testing technology platforms o Internal Audit Findings should
o Introducing an IT o Introducing stricter controls be clearly documented and
infrastructure allowing for and documentation standards reported to Senior
model user feedback Management and the Board
o Assessment of the process for
establishing and monitoring
limits on model use

2018 Deloitte 17
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Part 3
Deloitte
MRM Offer

2018 Deloitte 18
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Deloitte MRM Offer

Deloitte proposed services
MRM Organisational Design : Role profiles according to the 3 lines of Defence, Internal Structuration,
ORGANISATION Outsourcing of some activities (model validation), Change Management, Creation and facilitation of MRM
live training / e-learning modules, Peer benchmark assessment, Model Risk Attestation
AND MRM Policy: Creation of policies, procedures, practices & controls according to regulatory guidance,
GOVERNANCE MRM Program: Set-up effective MRM program, PMO activities
MRM Governance: Define Model Risk Appetite, MR reporting, Guidelines for model usage and limitations

Model Inventory: Migration of multiple model systems into a single enterprise platform
Model Classification: Assistance in classifying models (complexity, sophistication, materiality)
MODEL LIFECYCLE MR Documentation: Standardized documentation templates including MR evaluation, Documentation
MANAGEMENT review, Development of MRM life-cycle process flow charts with narratives
Model Follow-Up: Design of reporting routines & ad-hoc reporting capabilities, Model Monitoring
Model Assessment: Assessment of model implementation

Data Quality Review: Model input data review

MODEL Validation Approach: Design of appropriate model validation methodology
Model Independent Review: Perform independent review and deliver model review report
CONTROL Model Control Process Optimization: Analyze and streamline the existing control processes, Design
FRAMEWORK and adjust processes
Model Risk monitoring: Model performance monitoring, KRI’s and KPI’s, Model Risk Appetite

MRA Development: MR Source identification, Qualitative Assessment: Model Risk Scorecard,

MODEL RISK Quantitative Assessment: Model Uncertainty measurement, Model Risk incident database
ASSESSMENT Model Risk Analytics: Sensitivity Analysis, Measuring MR impact on P&L and Capital
Model Risk Mitigation: Identification of techniques for MR Mitigation

Platform Design: Set up & implementation of the MRM platform core functionalities & technology
infrastructure, workflow process management
MODEL RISK Platform Selection: Assistance for choosing a MRM technological solution
TECHNOLOGY Data Quality Management: MRM data quality management for accuracy, consistency & completeness
initiatives
Automation: Automation of repeatable areas processes and activities (RPA)

2018 Deloitte 19
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Structure Organisation and Governance

Deloitte can assist institutions with defining and implementing the MR functions and the associated
governance within the organisations.

Cross-cutting nature Roles MRM Function

• Business Lines, • Ownership : use of the model • Maintain an updated inventory
• Risk, Finance, • Control : measure, limits and • Validate classified models
• Model Validation, monitoring • Approve the use and limitations
• Internal Audit, • Compliance : policy compliance • MR policy preparation
• Technology

Model Risk Function should be independent of the other Validation, Audit and Model Development functions
in the company.

• The day-to-day management of

inherent model risk is delegated to the
first, second and the third lines of
defense.
• The Board of Directors is responsible for
the approval of the MRM framework,
receiving regular reports on the
implementation of the MRM Policy.
• External Resources may be
commissioned by banks to supplement
internal capabilities for model validation
and review, compliance functions, or other
activities in support of internal audit or
other lines of defence.

2018 Deloitte 20
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Structure Organisation and Governance

 If the establishment of a MRM function plays a central role in the implementation of a robust MRM framework, there might be
various organisational options to consider with each having its pros and cons: the MRM function may be (i) a separate function
within the 2nd LoD, (ii) grouped with / reporting to the model validation function.

Model builders / model users Model validation

• Perform model validation tests and
• Distinction to be made between performance review for models
model builders and users whose model risk is deemed
• Clearly establish their respective significant or high
duties and responsibilities in MRM
• If model health is ‘poor’ or ‘fair’, is
• Clear model ownership framework to
empowered to propose model risk
be established (especially for models
mitigants and quantification of
used in a number of entities / BLs)
model risk, in liaison with model
• Both model builders and users are owners and with the MRM function
subject to model construction and
• As an outcome of model validations
MRM policies (incl. model risk
and performance reviews, confirm
assessment)
or amend model risk ratings
Model Risk Management function
• Strong integration of model
• Create and maintain the MRM validation into a firm’s risk culture
framework
• Maintain and update the inventory of
models
• Design and promote implementation of
model risk management policies
• Evaluate model risk to verify that it
remains in the risk appetite boundaries
• Provide model risk reports to Senior
Management and Board

Three lines of defence Operational risk managers

• Implement a model risk control framework • Capture model risk events in the OR
database events
• Report to the MRM function on the related
control KPIs feeding the key MR metrics • Report model risk events to the MRM
(model materiality, model health, etc.) function
• Verification that model risk mitigation
requirements are in place 21

2018 Deloitte
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Model Lifecycle Management

Our Model Lifecycle approach includes the following activities :
• Model inventory covering all of the organization’s models by type and goal;
• Model classification and prioritisation according to the risk posed to the bank, which will be required
in the monitoring, validation and documentation of models;
• Model documentation requires a comprehensive documentation that provides evidence of the diligence
used to create the model, captures the findings of the validation, and clarifies the intended use and
limitations of the model;
• Model follow-up scheme for the early detection of both deviations from target performance and model
misuse, in order to act accordingly.

MODEL DEVELOPMENT MODEL MODEL

MODEL INVENTORY MODEL FOLLOW-UP
AND USE CLASSIFICATION DOCUMENTATION

• Models with clear • Record uses and • Depends on : • Data sources • Statistical model
objective usage changes of the model (performance of the
o Materiality as • Model Methodology algorithms)
• Data Sources • Record approval economic
• Test plan • Decision strategies
status consequences
• Model Calibration (the decision rules)
• User’s manual
• Be supported by a o Sophistication
technological tool • Technological • Expert adjustments
o Impact on
Environment
• Keep track of all decisions
versions
• Link with the register
of rating systems
requested by EBA

The MRM should be assured by a suitable tool that keeps track of all changes and versions of the
models.
2018 Deloitte 22
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Model Lifecycle Management

Inventory of the models supposes the detection of the perimeter of the models potentially impacted by
the risk, through the identification of:
• The type of model (i.e. regulatory vs. model used for “managerial” or decision-making purposes)
• The goal of the model (e.g. risk management, budgeting, planning, product pricing, etc.)
Classification of the models according to the risk they bear, typically including tiers corresponding to
high, medium, and low levels of model risk , based on certain criteria, or a combination of them :

MODEL CLASSIFICATION (TIERING) MODEL RISK PRIORITISATION

• Model Materiality
• Model Complexity
Definition of a ranking for the interventions on
• Model Impact the models based on priority criteria on the main
• Model Significance for enterprise decisions metrics.
• Availability of model Decisions
• Model Quality

Models inventory Models assessment Models prioritisation

Model Risk Identification
Potential loss impact Quality Vintage
Model type Model
Material Non material Performance Years
Credit
Regulatory Operational
…
Used for Credit
managerial Planning
purposes …

2018 Deloitte 23
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Model Lifecycle Management

Enhance the workflow efficiency through streamlined lifecycle management
Decommissioning: Revoke permission of use
for non-performing or unused/ outdated/ Specification: Define model requirements
replaced models

(Re)development: Model design, choice of

Reporting: Performance monitoring;
methods; Identification of model weaknesses
Consolidated model risk
and limitations; Documentation

Validation: Classifications and individual risk

Model Planning: Propose (re)development of
new and existing models Model Lifecycle assessment (categorisation, quality/ rating,
materiality; Initial and recurrent validations

Performance Monitoring: Periodic review

Approval: Permission to use the model for the
which may trigger (re)developments and
intended purpose
(re)validations

Use: Ensure that model is only used for the Implementation: Implementation on delivery
intended purpose; Control of post model platform; Functional and user acceptance
adjustments testing; Defined change control process

Workflow

• Role-based responsibilities • Workflow oriented deadlines for each task

at each control point in the workflow process
• Status reporting to the model inventory
• Issue management and resolution
• Planning of resources

2018 Deloitte 24
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Model Control Framework

The key part of a MRM framework is establishing a strong and independent monitoring and validation
function. This function should be able to address the quantitative and qualitative review of models across
the areas of data, methodology, documentation, processes and governance.

Independence
Process Design
Validation function should be an
Analysis of the existing independent unit in the institution.
process.

Thoroughness Internal Criteria

All models that involve risk for the Each institution needs to set standards
institution should undergo the validation using it’s own criteria, which should be
process. commensurate with model risk.

Scope Frequency
Validation should cover : It is an iterative process performed with a
• Methodology • Quantitative aspects specific frequency.
• Documentation • Governance
• Quality of the data used • Technological
Environment

Governance Organization
Set mechanisms for model annual review. The validation function roles, responsibilities
and work scheme should be documented and
Establish model validation committees. approved at the corresponding level.

Staff Audit
Sufficient number of qualified professionals. The validation function itself must be reviewed
by the Internal Audit, which needs to analyze
Consider Outsourcing the Validation Function. its work and implemented controls.
2018 Deloitte 25
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Model Control Framework

“Banks should have a more holistic approach to Model Validation. Not only the mathematics, but also the
process and the data quality”.

INITIAL MODEL VALIDATION AND REVIEW

Validation KPIs and activities : • Sensitivity tests
• Actual vs. Estimation analysis • Benchmark tests
• Stability Tests • Stress tests
• Discriminatory Power • Convergence Tests
• Concentration analysis • Counter-checking with expert judgment
• Source Code Tests

MODEL APPROVAL
Internal Model Approval Process:
• Model Approval requires the model validation to be completed
• Further approval from the senior risk and/or board risk committees

ONGOING REVIEW AND VALIDATION

Ongoing model validation and review are essential to assessing whether models are continuing to perform as expected new model
limitations. Typically, model review and revalidation include :
• Material model changes • Backtesting
• Significant market changes • Model performance deterioration detected by ongoing performance
• Significant product or portfolio changes monitoring
• Change in the model risk ranking • Regulatory and audit concerns

2018 Deloitte 26
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Model Risk Assessment

Key questions: What is your appetite regarding model risk and how to quantify it?
Ability to provide a comprehensive and consistent view on model risk at a defined level of aggregation is an important goal of a
MRM framework.

Model risk appetite Model risk policy

• The expression of the Board’s appetite for model • An overarching Model Risk Policy sets out the roles
risk is one of the crucial steps in robust model risk and responsibilities of the various stakeholders in the
management. MRM framework, including those of the 3 lines of
defence and of model owners, accompanied with the
• As for any other risks, model risk appetite is
group-wide modelling and MRM standards:
articulated in the form of appetite statements and
of risk tolerance limits applied to effectively – model risk definition and identification tailored to the
monitored model risk metrics. bank
– monitoring of MRM: model risk KPIs and metrics
Organizing – specific requirements for the development, validation
MRM and use of model

Model risk reporting Model risk mitigation

• The Board has ultimate responsibility for managing • To reduce exposure to model risk and to ensure the
the firm’s model risk. It is therefore important that bank constantly operates within the boundaries of its
information provided to the Board and BRC enables risk appetite, model risk mitigants are to be
effective oversight of that risk: prescribed when model health is weak or
– Model risk profile against model risk appetite
deteriorating:
boundaries – A broad range of model risk mitigants are available
depending on model types concerned, its purpose and
– Qualitative information (outcomes of model
the modelling issues met.
validations)
– Appropriate checks should be performed by the 3 lines
– Model risk assessment
of defence

2018 Deloitte 27
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Integrated Approach for Model Risk Assessment

 Assessment of model materiality helps in prioritising actions and in rolling-out the MRM framework. There might not be a unique view of what are the
most significant drivers of model materiality, although the regulatory context and possible operational / compliance risks should be seen as key
drivers.
 The health assessment framework in place should enable to highlight the most important causes or issues met (“root cause” approach instead of a
mere assessment). It should capture the feedback received from Supervisors, Auditors and other lines of defence.
 Model risk inherent to models used for multiple purposes may vary depending on each particular context of use, which need to be inventoried.

Regulatory models Risk management models Financial reporting First line models

 Frequency and number of

Volume  Gross exposure amounts  Gross exposure amounts  Gross exposure amounts
decisions taken
Model materiality

 Compliance with regulation  Supervision / regulation  Financial statements

 Dependence of other
 Dependence of other  Dependence of other  Compliance with standards
models or activities
Context of models or activities models or activities  Dependence of other
use  Automated decisions vs.
 Automated decisions vs.  Automated decisions vs. models or activities
overrides able to challenge
overrides able to challenge overrides able to challenge  Automated decisions vs.
model outputs
model outputs model outputs overrides / challenge Model materiality
 Significance / weight in  Operational & non- Model risk
regulatory ratios  Operational & non-  Operational & non- compliance risks rating Low Moderate Significant High
 Volatility of gains / losses compliance risks compliance risks  Proportion of P/L
Financial
of relevant activities  Volatility of gains / losses  Volatility of gains / losses dependent on model for
impacts
 Impacts if supervisory of activities concerned of activities concerned decision-making
model approval is  Reputation risk  Reputation risk  Opportunity costs

Model health assessment

Very
removed  Reputation risk good

 Data quality and integrity, wrong data inputted in models

Data Good
 Other data limitations: availability / scarcity, use of external data, changes in definitions over time, etc.

 Theory: non-standard & emerging model theory, un-tested limitations in underpinning theories/assumptions, proxies
Model health assessment

Modelling
 Obsolescence, non-approved material changes
choices Fair
 Modelling complexities, missing variables, misspecifications
 Scenario analyses: model behaviour in stressed or extreme conditions
 Sensitivity analyses: sensitivity to variations in estimations of input parameters, to available data and to changes in
Model Poor
assumptions
uncertainties
 Benchmarking analyses: comparison of outputs and theories to alternative modelling choices
 Stability & robustness: population stability, input and output stability, etc.

 Backtesting: actual performance vs. past experience

Model
 Discrimination power (if applicable)
accuracy
 Limited range of validation techniques used (incl. expert-judgments, reasonableness checks, etc.)

 Model infrastructure: (resources, systems / processes)

Model
 Model governance: model challenges, permanent controls, on-going accuracy checks
environment
 Model not used as intended
and use
 Model interconnections: upstream and downstream dependencies to other models

2018 Deloitte 28
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Illustration – Model Risk Quantification

SCENARIO OVERVIEW
Product/Client Model Risks
• Scenario is aimed at • Client counterparties (Market best practice) usually measure FwStart • Risks arise from model design -
analyzing and options market values with Heston model key measurement phase manually
measuring the • Client Murex release, do not include Heston model, thus option managed - and is related to
"Pricing Model Risk" market price is obtained as a sum of two separate deals: manual input of parameters
relate to Forward o One priced with Black&Scholes model, which generates a • The loss could occur if there is a
Start Options for UE price different from the market one lack of the parameters’ upgrade
CIB o A dummy deal manually booked and calculated "out of the and in the same time, the
box" just to measure the difference between the B&S price counterparties ask the deal
and the Heston one closing
• The Heston value is obtained with external model (xls based) with
manual input of main parameters

QUANTIFICATION
"Pricing Model Risk“ is measured with a Statistical/Actuarial Approach (LDA) used for operational risk
measurement, based on separate modeling of:
Frequency Severity
Count of the lack of recalibration, in the past 5 years, of the model’s Sample of potential losses: difference between the FWstart option
parameters and the early termination of deals, then the joint price calculated with client model with no parameter recalibration
probability of occurrence and the FWstart market value
Investigation about the occurrences in the past, with the aim to Investigation about the most frequent impact and the worst one in
make a prediction for the next year analyzing budget volumes and the past, with the aim to make a prediction for the coming year,
expert opinions asking for confirmation to the experts

As result of the analysis the costumer has decided to:

• Evaluate the integration of Heston model directly, or of other
models, currently not included in Murex
• Extend the existing controls for deals measured with Heston
models

2018 Deloitte 29
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Integrated MRM Platform (1/3)

Four coherent dimensions that structure the model life cycle
management in a MRM platform
• A successful model risk management platform integrates model lifecycle workflow, a model inventory, a document
repository, and analytical/reporting capabilities
• While most organizations have some form of the below components, the industry is moving towards a centralized
system that integrates and connects all components into a single robust Model Risk Management platform.

Model Inventory Model Lifecycle and

Database Workflow Capabilities

 Store and maintain  Role-based model

standardized and non- validation workflow
standardized model-  Issue management and
Model Workflow & resolution
specific information Inventory Process  Role-based responsibilities
 Complex querying

Centralized Document
Storage

Reporting Capabilities  Upload and download

documentation
 Board and regulatory  Documents linked to
Document model inventory
reporting
 Operations/workflow
Repository  Document version control
Analytics &
reports
Reporting
 Administrative reports

2018 Deloitte 30
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Integrated MRM Platform (2/3)

Stage 3: MRM best practices of the four coherent dimensions

Model Inventory Workflow & Analytics & Document

Process Reporting Repository

 Enhanced  Enhanced workflow  Enhanced reporting  Enhanced and

inventory/model data processes to stream-line capabilities through standardized existing
through automation MRM activities (e.g. templates and model documentation
capabilities model eligibility) automated reporting
 Automated components
 Integrated inventory  Defined governance  Ongoing monitoring of validation report
data for centralization responsibilities at each capabilities by generation process
on a single platform control point in the connecting testing
workflow process results to recurring  Enhanced document
 Standardized data monitoring processes repository (e.g.
dictionary across entire  Specific interfaces for organization, versioning)
inventory (e.g. model each control point across  Model risk aggregation
types, assumption the workflow process, capabilities using
types, input types) enabling governance with standardized metrics
clearly defined handoffs and scoring
 Enhanced model methodologies
inventory controls to
minimize model data
issues

31

2018 Deloitte
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Integrated MRM Platform (3/3)

Stage 3: Model risk quantification enhances analytics and reporting
and capabilities with a means to identify the level of model risk
Model Inventory Workflow & Analytics & Document
Process Reporting Repository

Model Materiality
Low Medium High Critical

• Model risk quantification provides insight

into model risk for analytics and reporting Model risk

quality
Top
purposes appetite frontier

• Each model is scored individually on three

criteria using a measurable scoring system

Good
Model Health
• The aggregation framework transforms
individual model risk scores into model area
risk scores

factory
• The resulting model risk scores are Satis-

visualized in a model risk matrix as seen on models with high

the right priority
Poor

Size = Model size

Color = Model type

32

2018 Deloitte
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Off-shore support
Driving “around the clock” progress and meaningful cost savings
Operating Strategy Benefits
Degree of
involvement is
Degrees of tailored based on
Involvement complexity, type of
models, and
preferences.

Time Validation activities

Efficiency performed around
the clock – 24-hour
validation cycle

Savings

 Deloitte professionals manage workload, communication and Component of

handoffs with off-shore resources, with no requirement for Cost work in India leads
the client to interact with the off-shore team. US to meaningful cost
Efficiency
India

savings
 The degree of direct communication and interaction between
the client and off-shore resources can be customized to the
client’s preference.

Experience Level
Our Off-Shore Team At a Glance
 150+ validation, MRM, data analytics, and quantitative modeling professionals
 Extensive experience with: Stress tests, Basel, Credit / Market / Operational / Liquidity Risk
 Programming skills include: SAS, R, Matlab, Python, SQL, C++
 A large, structured and leveraged team that includes a combination of senior resources with 5-
10+ years of US experience leading staff teams with strong industry experience and
US India
educational credentials (economics / econometrics, math / statistics, and quantitative finance)
2018 Deloitte 33
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Part 4
Credentials

2018 Deloitte 34
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

US Credentials
• Assistance to top 5 Banks with the enhancement of its MRM framework
• More than 100 professionals are helping or have helped the Bank comply with U.S. regulatory guidance
related to models
• Project activities include MRM planning, model validation, technology enhancement, & process
improvement

MRM Gap Assessment: Comparison of governance, practices, & controls to regulatory guidance
Benchmarking: Analysis & presentation of the differences in industry practices for MRM
PLANNING
MRM Program Design: Development of a multi-year plan for enhancing MRM

Staff Augmentation: 100+ professionals serving as an extension of the Bank’s validation team
MODEL
Academic Research: Assistance with the creation of modeling “white-papers”
VALIDATION Peer Review: Secondary review of internally performed model validations

Model Inventory: Migration of multiple model systems into a single enterprise platform
TECHNOLOGY Platform Development: Enhancement of the MRM platform & workflow, prototype
ENHANCEMENT development
Data Quality Management: MRM data for accuracy, consistency, & completeness initiatives

PROCESS Practice Development: Creation of MRM practices (e.g., Risk Classification Methodology)
Reporting & Analytics: Design of reporting routines & ad-hoc reporting capabilities
IMPROVEMENT Documentation: Creation of policies, procedures, & standardized documentation templates

2018 Deloitte 35
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

EU Credentials
Mission name / perimeter Project type Details Period Country

Internal rating system authorized

Updating of the Validation Framework for Credit Risk 4 months Italy
by Local Authority

LTS (project with an

Internal rating system authorized Support to Basel II Project PMO: management of the workstreams;
elapsed time higher than Italy
by Local Authority coordination and communication to Directors and Corporate Bodies
2 years)

Internal Rating System validation (using SAS), with focus on:

Credit Risk PD models for Corporate and Retail portfolios
LGD models for Corporate and Retail portfolios and LGD for
Defaulted Asset LTS (project with an
Internal rating system authorized
Structured Finance models elapsed time higher than Italy
by Local Authority
Models for Banks and Public Sector Entities portfolios 2 years)
Processes
IT Systems
II Pillar Risks

Development (using SAS) of the rating system:

Support in development of PD, LGD and EAD centralized models for LTS (project with an
Credit Risk Internal rating system authorized Corporate portfolio elapsed time higher than Italy
Personal loans, Mortgages, Salary Loans PD and LGD models 2 years)
development

Internal Rating System validation (using SPSS), with focus on:

LTS (project with an
Validation Framework definition
Credit Risk Basel II roadmap in place elapsed time higher than Italy
PD models for Corporate and Retail portfolios
2 years)
LGD models for Corporate and Retail portfolios

Rating system review:

Internal Auditing Framework definition LTS (project with an
Basel II roadmap in place Quantitative aspects – Corporate and Retail models elapsed time higher than Italy
Organizational aspects/governance 2 years)
Credit Risk IT systems and Data Quality

Rating system development: LTS (project with an

Basel II roadmap in place Defaulted Assets LGD model development elapsed time higher than Italy
EAD model development 2 years)

Review of Risk Classes for AMA capital calculation

Support to validation of AMA calculation model LTS (project with an
Operational Risk AMA Development and validation Support to validation of AMA capital allocation methodology elapsed time higher than Italy
Support to Audit Unit on AMA process and validation calculation 2 years)
Support to AMA framework validation

2018 Deloitte 36
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

EU Credentials
Mission name
Project type Details Period Country
/ perimeter

AMA framework designed by Deloitte for EIB resulted won the “European
Operational Operational Risk Awards 2006” for the category “Best AMA Framework of
AMA Design and Implementation 4 months Italy
Risk the year” during the OpRisk Europe Conference in London in 2006

Operational
ORM framework development Definition of methodology and validation instruments for ORM framework 7 months Italy
Risk

(i) stress-testing of the model under a variety of conditions,

Operational (ii) in-depth technical analysis of the code used by the client to compute
AMA model validation 2 months Belgium
risk the capital requirement and (iii) benchmark survey of the market practise
of AMA models across 10 Deloitte member firms

Counterparty CVA model review for collateralized In the context of the Asset Quality Review, the National Bank of Belgium
1 month Belgium
Credit Risk exposures asked Deloitte to review the

Development of a credit scoring model that is used to give a probability of

default to consumer loans. The model uses logistic regression and is
Development a credit scoring model for
Credit Risk developed in R. The model can be used by the bank to assess the health of 6 months Belgium
acceptation of private loans
the current portfolio or the probability of default of new "through-the-
door" customer.

In the context of audit mandates within the Fund Industry, Deloitte

Review of the valuation of complex equity reviews the valuation of complex equity swaps and interest rate
Market risk 6 months Belgium
swaps and interest rate derivatives derivatives by full revaluation using in-house built models (Black, Heston,
Variance Gamma, SABR, Hull White).

Development of the shifted SABR model. The model consists of a calibrator

Development of a valuation model for
that generates the SABR parameters and a pricer that uses the SABR
Market risk interest rate derivatives under the negative 6 months Belgium
parameters in order to price. The pricing covers most of the vanilla
rate environment
products, such as caps, floors, swaptions, CMS caps.

Statistical analysis of historical prepayment rates across various clusters of

Model validation of a prepayment risk
Market risk clients and timeperiods. The institution is using this analysis as a back up 6 months Belgium
model for mortgage loans
to their expert-judgement

Review of the collateral haircut

Market risk 1 month Belgium
methodology Review of the collateral haircut methodology

validation of a number of models used to

Validation of a number of models used to value vanilla instruments (swap,
Market risk value vanilla instruments (swap, FRA, 3 months Belgium
FRA, swaptions, ZC inflation swaps)
swaptions, ZC inflation swaps)

2018 Deloitte 37
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

EU Credentials
Mission name / perimeter Project type Details Period Country

Validation of the valuation model for a synthetic CDO,

Validation of the valuation model for including derivation of the default intensities from bond
Market risk 2 months Belgium
a synthetic CDO spreads; loss distribution and simulation based on Gaussian
copula to generate the time to default.

LTS (project with an

Internal rating system authorized by
Credit Risk Review of measures to solve regulatory findings elapsed time higher than 2 Germany
Local Authority
years)

Internal rating system authorized by

Validation of Corporate Rating Model (PD) (using SAS) 2 months Germany
Local Authority

LTS (project with an

Internal rating system authorized by Re-Development of Ratingmodel for Factoring including initial
elapsed time higher than 2 Germany
Local Authority valdidation (PD, LGD, CCF, dilution) (using SAS)
years)

LTS (project with an

Credit Risk
Support of rating model development for overdraft portfolio elapsed time higher than 2
Internal rating system awaiting
(PD, LGD, CCF) including initial validation. Additionally years) Germany
approval by Local Authority
validation after first year (PD) (using SAS)
.

LTS (project with an

Development of IFRS 9 impairment methodology including
IFRS 9 Impairment model elapsed time higher than 2 Germany
implementation of validation/Re-calibration process/methods
years).

Qualitative and quantitative validation of the stochastic

Internal Model Method (IMM)
processes, risk factors and dependencies including 4 months Germany
authorized by Local Authority
benchmarkings with historical data
Counterparty Credit Risk
Qualitative and quantitative validation of the market implied
Internal Model Method (IMM)
exposure simulation including independent implementation 1,5 years Germany
awaiting approval by Local Authority
and exposure benchmarking

Project framework (Factoring & Leasing)

Management and monitoring of Basel II project (governance
project structuring, PMO, quality assurance, progressive
validation of the approval file, skills transfer)
Support for the definition and the
Methodologies definition and risk parameters estimation :
establishment of the Basel II IRBA
Credit risk o Framework – Review of the existing model and 3 years France
project on factoring and leasing
development plan definition
activities
o Model design
Model development within information systems
Operational integration of model within process and
organization

2018 Deloitte 38
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

EU Credentials
Mission name
Project type Details Period Country
/ perimeter

Monte Carlo VAR - Assistance in the approval process by the French

supervisor
Cartography of the business Iines / desks (type of instrument, P&L, risk
limits)
Writing of the model documentation provided to auditors and to the French
supervisor
Governance & Organization of the Market Risk department
Market Risk monitoring : limits framework, model validation, reserves
Assistance in the approval process of the
Market Risk calculation 6 months France
internal model
Perimeter covered by internal model
Monte Carlo VAR methodology : instruments re-pricing, scope of the risk
factors, design of the scenarii, calibration of the correlation matrix
IT architecture, workflow, definition of the controls to ensure the quality of
the data
Back testing framework
Reportings (including stress tests)
Follow up of the recommendations issued by the supervisor after its review

Organisational structure : charts, sizing and roles of the involved

departments (FO, Risk, Product Control, Finance)
Governance around model changes & validation
Internal Model Approval - on site
Market Risk Model use & outputs : internal use of risk measurement system (including 2 months France
inspection
market risk management & limits setting), internal reportings, calculation of
own funds requirements, capital allocation, stress testing
Back testing process & results

MRM maturity stage target definition

Identification of the main gaps between the target and the existing and the
MRM Gap analysis and road map definition formulation of generic recommendations that are deduced directly from the 2 months France
gap analysis.
Recommendations for model validation practices

Model Identity card definition

Model Risk Scorecard definition with regard to three main axis : materiality, health and
Credit Risk Model risk quantification framework design costs
3 months France
Model Aggregation methodology definition

Definition of a groupwide (bank and investment fund manager) model risk framework
with focus on validation of internal models.
Credit Risk and ascertaining the different regulatory requirements (between bank and ifm per risk
MRM gap analysis and best practice
Investment category) and comparison with their internal processes. 3 months Germany
recommendations Definition f the framework document containing the regulatory validation requirements
Fund manager
(roles, governance, reporting, tasks, …) with a general section and risk model specific
parts.

2018 Deloitte 39
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

EU Credentials
Mission name
Project type Details Period Country
/ perimeter

- Review of the governance supporting Model Management,

including maintenance, monitoring, model change
Independent audit of Credit Risk Model
management, back testing, use test and stress tests.
Credit risk Management and Quality Control processes 3 months Luxembourg
- Review of the quality control in place that cover data quality
(Retail models)
management, internal credit risk reporting, calibration of risk
parameters.

Critical analysis of the methodology

Independent validation of the CVA internal
Credit risk Review of the process for calculating CVA 3 months Luxembourg
model
Review of compliance to best banking practices

Critical analysis of the methodology

Independent validation of the LGD Retail
Credit risk Review of the process for calculating LGD 2 months Luxembourg
model
Review of compliance to CRD IV requirements

Audit outsourcing for Credit Risk Model Review of the Model Validation life cycle, including Coverage,
Credit risk (PD, LGD, CCF) and Market Risk Model Governance, Documentation, Methodology and Maitenance of 3 months Luxembourg
Validation Processes the Model Validation process.

Independent review of IRB models for Four-year audit program covering all IRB models of the Bank
Credit risk Corporate, PSE and Institutions (model and methodology, governance, use test) 4 months Luxembourg
counterparts (PD + LGD) Benchmark against best banking practices

External, independent review of EIF’s rating model with

Review and validation of the securitisation emphasis on the operating structure of the model and the
Market risk 4 months Luxembourg
internal rating model model’s underpinning assumptions and rating assignment
mechanism.

2018 Deloitte 40
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

EU Credentials
Mission name
Project type Details Period Country
/ perimeter

Development (using SAS) of the IFRS9 risk models:

IFRS9 IFRS9 Risk Models Support in definition and implementation of PD, LGD and EAD models for credit 1 year Spain
portfolios.

Credit Risk
Analysis of Business Processes supported in IT systems, according to RDA
RDA
Data and processes review Framework, reviewing Source Data and Derived Data (calculations in loads, ETLs, 1 year Spain
Framework
transformations, etc.) for Credit Risk information.
review

Data issues Development of a methodology and an integrated tool to manage, from their
Data and processes review 6 months Spain
Management identification to their solution, Data Quality issues for Credit Risk information.

Definition and development of a Testing Plan and Controls over Data Models to
Data Models
Data and processes review ensure the correct functioning of consolidation and reporting engines (reconciling 6 months Spain
review
with general ledgers, error reviews, etc.) using IT solutions.

Support (using IT tools) of the analysis of Capital requirements under the AQR
AQR review Capital requirements 3 months Spain
stress test methodology

Development of Pricing Models to validate and audit the fair value of financial
Market Risk Pricing Validation & Fair Value Audit instruments (Level 1, Level2 and Level 3) using Montecarlo techniques, Gaussian 6 months Spain
Copulas, Black-Sholes among others..

Development of CVA models to use them as a proxy for the validation of the credit
Market Risk Pricing Validation & Fair Value Audit 6 months Spain
risk adjustment reasonability

Risk Sensitivity Analysis regarding risk model such as: liquidity analysis (inputs
variations more than 95% of the standard deviation, quoted prices, bid/ask
Market Risk Pricing Validation & Fair Value Audit spread..), analysis of fair value impact considering changes in the key inputs 6 months Spain
(correlations, beta parameter, volatility due to market price uncertainty), close-out
cost, CVA-FVA analysis.

Validation of the whole internal control environment related to fair value process
Market Risk Pricing Validation & Fair Value Audit 6 months Spain
from the inputs capture to the accounting of the fair value

Credit risk, Evaluation of the adequacy of the governance and control level established for the
market risk, Entity
Assessment on governance and control
operational All Risks (credit risk, market risk, operational risk, etc.) were under the scope, 4 months Spain
environment associated to all risks
risk, model including model risk
risk 'Work methodology was based on the revision of written policies and procedures.

2018 Deloitte 41
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

EU Credentials
Mission name
Project type Details Period Country
/ perimeter

Assessment of the internal control framework and its compliance with EBA
guidelines
Internal - Apropiate organisational framework and structure (independence and
Control Risk Control Function (RCF) proporcionality) 2 months Spain
Framework - Composition, Responsibilities
- Ensure there is a clear, transparent and documented decision-making process
(reporting).

Support in the definition of internal control

Internal Assistance in the definition of the internal control model in a systemic financial
model and the SAP GRC system structure to 1 year Spain
Control organization, based in the international standard COSO 2013:
manage it

2018 Deloitte 42
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Part 5
Contacts

2018 Deloitte 43
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Contacts
Please do not Germany Italy Spain

Francesco Zeigner Cesar Gurrea

hesitate to Thomas Moosbrucker
Director Partner Partner
contact your Florian Wilbert Vincenzo Cosenza Oriol Arnau
relevant experts. Senior Manager Manager Partner

MRM EMEA lead Netherlands Belgium Portugal

Marc Van Caeneghem Koen Dessens Nicolas Castelein Miguel Morais

Partner Partner Director Partner
mvancaeneghem@[Link]
Sjoerd Kampen Roeland Baeten Vitor Lopes
Senior Manager Director Partner

France Austria United Kingdom Central Europe

Nadège Grennepois Kurt Blecha Thomas Clifford Adam Kolaczyk

Director Partner Director Partner
ngrennepois@[Link]
Ute Romana Schoeggl Henry Jupe Marcin Gadomski
Michel Guidoux Manager Associate Director Director
Senior Manager
mguidoux@[Link]
Luxembourg
Anca Alvirescu Jean-Philippe Peters
Senior Consultant Partner
aalvirescu@[Link]
Arnaud Duchesne
Director

2018 Deloitte 44
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Part 6
Appendix

2018 Deloitte 45
 MRM DELOITTE
CONTEXT CREDENTIALS CONTACTS APPENDIX
CONTENTS MRM OFFER

Tools & Methodologies

Validation Playbooks Validation Tools Validation Templates

Project Management Tools Training Materials Reporting Templates

Team Summary
Project Status (as of May 18, 2011)

Deliverable Budget
 International, Def Annuities have been delayed due to data not being sent by the
Key Status client
Status Items
 SOP and Auto & Home continues to be highly over budget

Annual Budget Deliverable Budget

Teams Comments
(Hours) Status status
Delayed Work Streams: Reasons and Action plans
The team needed to perform unanticipated cash flow testing and loss
Oversight 1250
recognition testing this year
Work streams delayed
Def Annuities 2475 Deadlines have been missed due to delayed inputs from client
Teams/
Teamover due to additional
Work Stream Risk Owner Reason for delay Action plan deliverables
Payout Annuities 1100 Budget is slightly detail testing selections
affected

Individual Life 1460 Deliverables are on track andInterim

1 Reinsurance
Detail
the team is testing
on budget Timothy Need to confirm new
Send questions to client in the w/o May 23
US Ops -> Assumption
work papers Gasaatura assumptions Heat map

SOP/DAC 1740 Budget is significantly

2 Payout
Payout Annuity 4Q
Annuity over due to additional detail testing selections
William Bang
More time than expected on Memo will be completed by the deadline of
US Ops -> ROMM
Memo the memo May 25

Group L&H 1200 Due to resource restrains deliverables are delayed and budget is over
FAS157 Detail
Waiting on the capital market
3 FAS157 Maria Itteilag team to confirm account Follow up with the capital market team US Ops -> ROMM
Testing Memo
balance
International 2570 Delayed data requested impacted the completion of deliverables
Deferred annuity
Tom Waiting to receive data back
4 Deferred Annuity Cash Flow Testing Follow up with the client on requested data US Ops -> ROMM
Chamberlain from the client
Reinsurance 880 Budget is slightly over due toMemo
additional detail testing selections

Scope of service has been increased and additional resources need to

Auto & Home 325
be staffed onto the team Work Streams affected due to delay by other Teams

At Risk Dependent Dependent Deliverable

Actuarial team Impacted deliverable Interdependent team Status Owner
deliverable Action Plan

Custody Rule Control Testing Dobrina

1 FAS 157 4Q Work papers US Ops -> Retirement products New requirements from client
work paper Naydenova

More time than expected on the

2 Payout Annuity Interim Detail Testing Corporate Tampa Account roll forwards Jamil Jones
memo

Deferred Roll forward testing work Jennifer Communicate with the client on
3 Cash Flow Testing Investments
Annuity paper Welsh data request

Legend

At Risk

Off Track
Data For Illustrative Purpose only

Risk Classification
Scenario Generators Ongoing Monitoring Templates
Methodology
Actual Completion
Static View: Actuarial Planned Completion
Budget Used
% at last report (3/31)
180%

160%

Percent Completion
140%

120%

100%

80%

60%

40%

20%

0%
1250 2475 1100 1460 1740 1200 2570 880 325

2018 Deloitte 46
This communication contains general information only not suitable for addressing the particular circumstances of any individual case and is not intended to be
used as a basis for commercial decisions or decisions of any other kind. None of Deloitte GmbH Wirtschaftsprüfungsgesellschaft or Deloitte Touche Tohmatsu
Limited, its member firms, or their related entities (collectively, the “Deloitte network”) is, by means of this communication, rendering professional advice or
services. No entity in the Deloitte network shall be responsible for any loss whatsoever sustained by any person who relies on this communication.
Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee (“DTTL”), its network of member firms, and their
related entities. DTTL and each of its member firms are legally separate and independent entities. DTTL (also referred to as “Deloitte Global”) does not provide
services to clients. Please see [Link]/de/UeberUns for a more detailed description of DTTL and its member firms.
Deloitte provides audit, risk advisory, tax, financial advisory and consulting services to public and private clients spanning multiple industries; legal advisory
services in Germany are provided by Deloitte Legal. With a globally connected network of member firms in more than 150 countries, Deloitte brings world-class
capabilities and high-quality service to clients, delivering the insights they need to address their most complex business challenges. Deloitte’s approximately
263,900 professionals are committed to making an impact that matters.