Frankfurt, Germany
A Comparative
Analysis of
CMMI and Automotive
SPICE
June 14th, 2006
Hans Sassenburg
David Kitson
Sponsored by European Industry
©2006 Carnegie Mellon University
Contents
1. Why this Analysis?
2. Overview of Automotive SPICE (ASPICE)
3. Comparative Analysis
4. Analysis Results
5. Generating ASPICE profiles from SCAMPI A/B results
6. Conclusions
7. Proposed Next Steps (HIS)
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 2
1. Why this Analysis?
Some of the key European automotive manufacturers
(OEMs) have created an industry sector specific process
reference model and process assessment model (collectively
known as Automotive SPICE or ASPICE) using the
framework established by ISO/IEC 15504; these artifacts
were published in August 2005. Because some of the
organizations in this industry sector already have CMMI-
based process improvement programs in place there is a
need to have an understanding of how the two approaches
compare as well as insights into how ASPICE process
profiles might be derived from the results of CMMI appraisals.
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 3
2. Overview of Automotive SPICE
ISO/IEC 15504-2 was developed, in part, to provide a basis
for relating assessment results from process improvement
models and assessment methods; it specifies the
conditions under which the results of appraisals based on
different process assessment models are comparable.
ISO/IEC 15504-2 specifies requirements for process
reference models (PRM), process assessment models and
process assessment methods (PAM) as well as defining a
measurement framework for process capability.
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 4
Role of ISO/IEC 15504-2
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 5
Automotive SPICE (ASPICE)
Process Reference Model (August 2005, v4.2):
- defines the processes composing ASPICE
- claims complies with the ISO/IEC 15504-2 requirements
- developed by selecting and modifying processes defined
in ISO/IEC 12207 (as amended)
- composed of 31 processes (ISO/IEC 12207: 50-60)
Process Assessment Model (August 2005, v2.2):
- related to the ASPICE PRM in satisfying the ISO/IEC
15504-2 conformity requirements
- an example of a process assessment model satisfying
the relevant ISO/IEC 15504-2 requirements relative to
the ASPICE PRM
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 6
History (ISO/IEC) 12207
ISO/IEC 14598 sub processes for
ISO/IEC 13407 (product evaluation) ISO/IEC 15939 Supply Process
(usability) (measurement) added
ISO/IEC 12207:2002 ISO/IEC 12207:2004
ISO/IEC 12207:1995
Amendment 1 Amendment 2
IEEE 1517
ISO/IEC 15502-2:1998 (asset management Change Request
(process reference) reuse program management Management
domain management) Process added
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 7
Correspondence of 12207 and ASPICE
The relationship is described in the ASPICE PRM by the
following extract from clause 1.1 –
“The Automotive SPICE PRM defined in this document is
derived from Annex F and H of ISO/IEC 12207 AMD1:
2002 and ISO/IEC 12207 AMD2: 2004. It contains a sub
set of the total processes with minor editorial changes
together with a number of other changes to reflect
consistency in use of terminology and application in the
automotive sector.”
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 8
Legend
3 signifies that the wording is either identical or considered
to be semantically equivalent,
? signifies that the wording is different and there is a
possibility of being interpreted differently from the 12207
process,
2 signifies that the wording is different and there is a
likelihood of being interpreted differently from the 12207
process,
NO means no,
NA means not applicable (e.g. there were no notes in either
process definition).
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 9
Correspondence of 12207 and ASPICE - 1
Component Comparisons with 12207
ASPICE Process Name ASPICE Name Purpose Outcomes Notes Revisit Reference
Process ID 12207 Mapping?
Acquisition process group (ACQ)
Contract agreement ACQ.3 3 ? ? NA NO
Supplier monitoring ACQ.4 3 3 3 3 NO
Technical requirements ACQ.11 3 3 3 3 NO
Legal and administrative requirements ACQ.12 3 3 3 3 NO
Project requirements ACQ.13 3 3 ? NA NO
Request for proposals ACQ.14 3 3 3 NA NO
Supplier qualification ACQ.15 3 3 3 NA NO
Supply Process Group (SPL)
Supplier tendering SPL.1 3 3 ? NA NO
Product release SPL.2 3 3 3 NA NO
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 10
Correspondence of 12207 and ASPICE - 2
Engineering Process Group (ENG)
Requirement elicitation ENG.1
3 ? ? ? YES
System requirements analysis ENG.2
3 3 ? ? YES
System architectural design ENG.3
3 3 2 ? YES
Software requirements analysis ENG.4
3 3 2 ? YES
Software design ENG.5
3 3 2 ? YES
Software construction ENG.6
3 3 2 ? YES
Software integration test ENG.7
2 2 2 ? YES
Software testing ENG.8
3 3 2 ? YES
System integration test ENG.9
2 3 2 ? YES
System testing ENG.10
3 3 2 ? YES
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 11
Correspondence of 12207 and ASPICE - 3
Supporting Process Group (SUP)
Quality assurance SUP.1 3 2 2 2 YES
Verification SUP.2 3 3 ? NA NO
Joint review SUP.4 3 3 3 ? NO
Documentation management SUP.7 3 3 3 ? NO
Configuration management SUP.8 3 3 3 3 NO
Problem resolution management SUP.9 3 ? 3 3 NO
Change request management SUP.10 3 3 ? NA NO
Management process group (MAN)
Project management MAN.3 3 3 ? ? NO
Risk management MAN.5 3 3 3 3 NO
Measurement MAN.6 3 3 3 3 NO
Process Improvement process group (PIM)
Process improvement PIM.3 3 3 ? 3 NO
Reuse process group (REU)
Reuse program management REU.2 3 3 3 ? NO
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 12
Questions …
• ISO/IEC 15504-2 requirements
- Why are they not met?
• Rephrasing process outcomes
- Why not issue change requests for ISO/IEC 12207?
• Deleting/adding process outcomes
- Why not issue change requests for ISO/IEC 12207?
• Claim that it is an “automotive specific” model
- A few notes are “embedded software” specific
• Rationale
- Why use this model when CMMI is used as the
internal reference model for PI, at least by suppliers,
but also by OEMs?
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 13
3. Comparative Analysis
Issues addressed:
• How do the models compare?
• Frame of reference
- Process Appraisals
- Training Support
- Infrastructure
- Adoption Status
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 14
How do the two models compare?
Both cover the 4 categories of Process Areas associated with the development of
software-intensive products:
- Process Management
- Project Management
- Engineering
- Support
However:
- coverage of specific practices vs. process outcomes are not at the same
depth
- CMMI covers some process areas more thoroughly than ASPICE
- ASPICE covers some process areas more thoroughly than CMMI
- CMMI satisfies ISO/IEC 15504-2 requirements, whereas ASPICE not as
PRM issues are addressed in PAM (base practices)
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 15
Process Appraisals
For CMMI:
- The CMMI Product Suite contains a standard appraisal
method (SCAMPISM). Wide use of a standard method
significantly reduces variation in appraisal results
- The SEI has expanded on SCAMPI and now supports a
family of appraisal methods:
SCAMPI Class C – Approach to process improvement
SCAMPI Class B – Deployment of processes
SCAMPI Class A – Benchmark appraisals
For ASPICE:
- The users of ASPICE provide their own appraisal method
- Question: how can inter-rater reliability be assured?
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 16
Training Support
For CMMI:
- Introduction to CMMI
- Intermediate Concepts of CMMI
- Practical CMMI course being developed
- Introduction to CMMI Instructor Training
- SCAMPI Lead AppraiserSM Training
- Several other related/special topic courses
For ASPICE:
- 3 Assessor Training courses
- Instructor and assessor training being developed
- No PRM/PAM trainings
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 17
Infrastructure
CMMI Steward (SEI):
- CMMI Initiative and other related SEI Initiatives
- Licensing and authorization/certification programs
- Instructor and Lead Appraiser Quality Assurance
Programs
- SEPG and CMMI Conferences
ASPICE:
- Automotive Special Interest Group of the joint
Procurement Forum / SPICE User Group
- HIS “Assessment” Working Group
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 18
Adoption Status
For CMMI – 10/31/05 :
- Introduction to CMMI students 40,809
- Authorized Intro. Instructors 292
- Authorized Lead Appraisers 398
- Organizations Appraised 917
For ASPICE (information from HIS working group):
- Will be used at least by Audi, BMW, DaimlerChrysler, Fiat,
Porsche, Volkswagen, Volvo Car Group (representing Ford
Europe, Land Rover, Jaguar, Volvo Cars)
- Transition from SPICE to ASPICE will be in 2006 for most of
the OEMs
- HIS members have performed 160 supplier assessments
- 60 ASPICE assessments/year beyond 2006
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 19
4. Analysis Results
Work document:
- In-depth analysis regarding differences between
ISO/IEC 12207 and ASPICE process descriptions
(name, purpose, outcomes, notes)
To be released (soon):
- Technical Note
(status: draft; under review)
- Mapping rules between CMMI and ASPICE
(status: draft; under review)
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 20
5. Generating a Mapping
CMMI ASPICE
CMMI ASPICE
overlap
exclusive exclusive
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 21
Assumptions
1. Mapping is one-directional: CMMI -> ASPICE
Motivation: CMMI is used as the internal reference model
2. Mapping is done of the following level:
- CMMI: specific practices
- ASPICE: process outcomes (PRM)
Motivation: a. Conform ISO/IEC 5504-2 requirements
b. No more detail needed
3. Available:
a. Mapping rules between CMMI and ISO/IEC 12207
b. Automated tool support
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 22
Generating ASPICE profiles from
SCAMPI A/B results - 1
Approach 1:
- Generate ISO/IEC 12207 process profile from SCAMPI A/B
results
- Use additional evidence for non-covered issues
- Disregard processes not addressed in ASPICE PRM
- Perform manual inspection for differences between ISO/IEC
12207 and ASPICE process outcomes
SCAMPI results ISO/IEC 12207
profile ASPICE profile
- Process filtering
- Mapping rules
- Manual inspection
Additional evidence
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 23
Generating ASPICE profiles from
SCAMPI A/B results - 2
Approach 2:
- Develop direct mapping rules between CMMI and ASPICE
- Using additional evidence for non-covered issues
SCAMPI results
ASPICE profile
- Mapping rules
Additional evidence
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 24
6. Conclusion
• CMMI has a broader scope of application than ASPICE, it has
a significant supporting infrastructure capable of providing
sustaining support over the long term, it has a significant
community of users around the world and it is built on proven
model-based process improvement technology
• There are some likely to be minor adjustments in the objective
evidence required which might need to be made by
organizations engaged in CMMI-based appraisal in order to
support the derivation of an ASPICE process outcome profile
as an outcome of a SCAMPI (A or B) appraisal
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 25
7. Proposed Next Steps (HIS)
1. Revise ASPICE PRM/PAM to satisfy ISO/IEC 15504-2
requirements and to reduce unwanted differences with
ISO/IEC 12207
2. Determine ASPICE exclusive issues, not covered by
CMMI
3. Execute pilot project(s) to investigate the feasibility of
re-using SCAMP A/B results to generate ASPICE
process profile
4. If feasible, provide automated tool support
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 26
Contact information
David Kitson
dhk@[Link]
Hans Sassenburg
hanss@[Link]
Cell: +41 79 231 6600
An der Welle 4
D-60322 Frankfurt
© 2006 by Carnegie Mellon University European SEPG June 14th 2006 - page 27