IBM AI Applications

AI maturity
framework for
enterprise
applications
March 2021
Authors
Rishi Vaish is the Chief Technology Officer (CTO) for IBM AI
Applications, a market leading portfolio of enterprise applications
spanning Asset Management, Facilities Management, Supply
Chain Management, Engineering Lifecycle Management and Weather
Business solutions. He is responsible for driving innovation through
Machine Learning, Data Science, Site Reliability Engineering and Hybrid
Cloud architecture for the portfolio. Through his career Rishi has over
15 years of executive leadership experience in technology, product
strategy, product development, product operations and product
management from startups to large scale organizations across a variety
of industries and technologies. He has deep expertise in AI-based
applications, hybrid cloud technologies, cloud computing, software-as-
a-service and application middleware. He is passionate about driving
innovation, modernization and transforming product and technology
organizations to meet scale and growth demands.
vaish@[Link] | [Link]/in/rishivaish/

Ashish Agrawal leads the Strategy and Portfolio Planning team for
IBM AI Applications. He is responsible for collaborating with business
and technology leaders to establish long-term growth strategy for
the business unit as well as is greatly involved in the execution of the
strategy. He works closely with the CTO office as well as the product
teams in identifying new technologies like Artificial Intelligence (AI)
that should be leveraged to innovate the product portfolio. His 20+
years of Strategy and Operations experience at IBM, BCG, Deloitte,
and Shell spans across multiple industries like High Tech, Energy, Oil &
Gas, Retail, Healthcare and Life Sciences with a focus on helping clients
grow and overcome their complex business challenges.
aagrawa@[Link] | [Link]/in/ashish-agrawal-972283

Shalini Kapoor is IBM Fellow for Artificial Intelligence and drives

research, strategy, and development to maximise the infusion of AI. She
is leading IBM’s AI market-leading capabilities, driving consistent, re-
usable implementation patterns, business transformation and value, and
making AI accessible and useable to business users, and not just data
scientists. She is a torchbearer of Good tech projects in India focusing on
STEM and AI for women and underprivileged while working with schools,
universities, government and industry leaders. She has 14 patents and is
a prolific speaker in academia and various industry forums.
kshalini@[Link] | [Link]/in/kshalini

Rob Parkin is the Principal Scientist for IBM’s AI Applications

business unit. He has worked for over 20 years in applying large scale
machine learning and optimization technologies to solve core business
problems. In his current role, he leads the development of new AI
technologies to support IBM’s Supply Chain, Weather, IoT, Facilities
Management and Engineering Lifecycle product portfolios. Prior to his
work at IBM, he held research positions at Stanford University and at
the National Center for Atmospheric Research.
rparkin@[Link] | [Link]/in/rob-parkin-b751b61/

IBM AI Applications AI maturity framework for enterprise applications 2

Introduction The journey of AI infusion in
Artificial Intelligence (AI) is transforming how we live, work and do enterprise applications
business. From smartphones to smart factories, AI is ubiquitous – and
whether we realize it or not – it is influencing our everyday interactions Maturity curve is always a journey and maximizing AI’s value for any
with businesses. AI has disrupted our lives by entering the entire organization starts with assessing the current state of AI adoption,
gamut of areas like recommendations during ecommerce purchases, gaining consensus on short- and long-term goals, and understanding
personalized video streaming options, predictive texts while typing, and the execution capabilities and gaps. There are many maturity
prioritizing social media feeds. frameworks available – from Gartner1, IDC2, Ovum3, and others – that
can help assess an organization’s AI maturity and related skills. We
As organizations move on their AI journey, they have to infuse AI in recommend The AI Ladder: Accelerate Your Journey to AI4 as a starting
their enterprise applications and sometimes build new AI offerings point for leaders interested in understanding enterprise AI’s value and
from scratch. The use cases for AI infusion span industries and product the steps necessary to implement it at scale. The AI ladder explains
categories, and cover a wide spectrum ranging from providing insights, how a data strategy is an essential foundation before the AI infusion
recommending the next best actions, predicting and forecasting journey commences. After understanding the AI ladder, assess the
business outcomes to fully automating business processes. Most maturity of your organization by using IBM AI Maturity Assessment5
organizations start their AI journey and soon want to scale AI but tool. If you are keen to assess the maturity of the machine learning
get stuck during the process. Identification of all AI capabilities, process, then we recommend the paper on Characterizing Machine
technologies, client adoption levels, and value derived from AI are Learning Process.6
some salient metrics that govern the journey’s success.
This white paper assumes that you already have an AI strategy in place
At IBM AI Applications, we are infusing AI into enterprise products and have identified the applications where AI needs to be infused and
ranging from asset management, facilities management, supply understand the impact that AI will have for the end-users. Once this is
chain management, engineering lifecycle management to weather achieved, organizations wanting to scale AI in applications ask, “How
business solutions. Our experience of infusing AI and delivering value good is the AI included in my applications?”, “What is the level of AI
to clients has been collated in this white paper and has led us to create sophistication that I need to include”? “Is there a trade-off between
the AI Maturity Framework. This framework, now a standardized the cost of AI functionalities to be included and the revenues the AI
methodology in our organization, is a measure of how mature AI is in application is impacting?”
any enterprise application. We deliberate on the factors that contribute
to AI’s maturity and how this is a journey and not an end state. This This white paper covers these critical questions for business and
framework prescribes a methodology for assessing business and technology leaders in-depth and explains how to move from one stage
technical aspects of AI. to the other while advancing our AI capabilities at each level.

This white paper is a ready reckoner for enterprises across industries

to measure the readiness and completeness of AI applications that
they have implemented. These enterprises are the ones who have
either infused their existing legacy applications with AI or have
bought new bespoke AI applications or even those who are evaluating
independent software vendors (ISV) selling specific AI applications.
In the same light, this whitepaper would be used as a tool by ISVs and
system integrators (SI) to measure the maturity of the AI applications
that they build and implement for clients. The factors contributing to
the maturity are both business- and technology-centric, hence the
measurement will be a combination of both. The value of the maturity
framework can be derived by a spectrum of business-oriented roles like
CXOs, product managers and business leaders who are correlating the
extent of the impact of AI to their revenues. The maturity framework is
also relevant for technical roles like enterprise architects, application
developers and data scientists who can measure the extent of advance
AI features that they have included to advance the solution.

IBM AI Applications AI maturity framework for enterprise applications 3

How good is the AI in my For better readability, factors mentioned in A and B are purely intended
for business leaders measuring the maturity of AI applications. If you
enterprise applications? are part of an enterprise product company such as an independent
software vendor or even if you are a systems integrator, then you are
What constitutes “good” AI is a challenging question to answer. The likely to find all of A and B applicable. If you are adding AI to enterprise
field of AI is rapidly expanding and there are always better models and applications as part of an IT organization, you can modify the business
techniques available to improvise our usage of AI. As a gauge, there are impact dimensions to suit your specific business goals. Dimensions
tens of thousands of research papers and articles published each year – C, D, E, F and G are intended for the technical community who are
at an increasing rate. Thanks to a vast number of programming language building technical features in AI applications. For business leaders,
libraries, new research makes its way into products very quickly. To be understanding dimensions D and E is also essential as the factors
able to answer the question of goodness effectively, we studied various described here have a direct impact on usage and trust in AI.
parameters. These range from how it impacts the business users;
the direct value clients derive from AI; how sophisticated the chosen For each of the dimensions, a given AI capability in the enterprise
tools and frameworks are; the inclusion of bias detection and removal application is ranked on a scale of 0 to 3 and then aggregated into a
algorithms; the quality of data and cleansing mechanisms; and how single score. Based on the scoring of a given AI capability it will fall into
explainable the AI is to end-users. one of the three phases also shown in the ladder in Figure 1 below:

Applications differ in their maturity curve and we observed that the – Silver
inclusion of the above parameters impacted the application’s success.
For example, an application that is low in improving the quality of the – Gold
input data and has no bias detection would give lower value to end
clients. Where you run your AI in enterprise applications adds another – Platinum
dimension to the maturity model. The next generation of enterprise
applications will be based on a hybrid cloud software model. This For a given AI capability, the ranking by the seven dimensions is also
means that the application containers could be running in a public cloud plotted on a six radii radar plot for easier visualization. For the radar
or in some private cloud behind a firewall. There might be a need to plot, the “impact on your business” dimension (dimension A) is pulled
monitor the performance of AI models and this feature would need to out of the radar since the other six dimensions are specific to the AI
be factored in the AI design itself. capability vs. the business area. As a sample, one of the AI capabilities
in our IBM AI Applications is shown in Figure 2 below. As you can see,
In this paper, we share the IBM AI Maturity Framework for this capability is on its way from gold to platinum.
Enterprise Applications.

The maturity framework outline

The framework is comprised of seven dimensions which are a
combination of business and technical capabilities that determine the
success of an AI application. The seven dimensions of the maturity
framework are shown in Figure 1 below:

A. Impact on your business

B. Value to the end client

C. Technology sophistication

D. Trustworthiness

E. Ease of use

F. AI operating model

G. Data

IBM AI Applications AI maturity framework for enterprise applications 4

Figure 1: AI Maturity assessment framework
We created a model to assess the AI Maturity of our Offerings to measure progress towards AI Infusion.

Dimensions Criteria
Impact on our Business Business Impact, Portfolio Impact
Value to Client Business Process Outcome, Differentiators
Technology Sophistication Appropriateness of the technology to the business problem, Learning Techniques, Reuse of Models,
Use of Inner Source or Open Source
Trustworthiness Integrity, Quality, Bias (Fairness), Explainability, Security
Ease of Use Intuitive for use by the intended user
AI Operating Model Deployment (Manual, Automated), Update Frequency, Infrastructure/Architecture Scale, Monitoring
Data Data Acquisition & Instrumentation, Data Management

3. Platinum
2. Gold
1. Silver

Phases
I Maturity
Offering A

Figure 2: AI Feature/Offering Maturity: Visualization Value to client

3
Illustrative example
Impact on Business: 1 2

Data 1 Technological
Sophistication

AI Operating Model Trustworthiness

Ease of Use

Maturity framework:
What the phases mean
3. Platinum: AI capability that scores at this level is a sustainable
1. Silver: AI capability that scores at this level will include factors that
differentiator. It is part of a mission-critical workflow for the
have been more recently introduced into the product to become AI
enterprise users and they rely on it for automated decision making
ready. This is the first stage where you discover what AI is, how it
and only focus on exceptions. The AI capability is sophisticated and
impacts business, the tools and technologies required to implement
has mechanisms to adapt to incoming data and learn by feedback
AI, how to prepare data for usage in AI, etc. This is a capability level
provided by end-users. The decisions made by the AI are clearly
that enhances the experience for users but is not mission-critical to
explained and understood by business users. These users can adjust
the business outcome that enterprise users are seeking.
business level dials and levers to tune the outcome they desire.
There are very strong and automated data management and data
2. Gold: AI capability that scores at this level delivers a meaningful
governance measures in place.
business outcome to the users. It will deliver a competitive edge for
the AI offering or application in the market. The capability provides
recommendations based on optimization or offline training on data Not every AI capability needs to graduate from one phase to the next –
and provides basic explanations for why a particular decision or organizations will need to assess the cost and benefit of investing in the
recommendation was made. The AI features are usable by line- capability based on users’ business outcomes and the cost to graduate
of-business users without having to involve data scientists. In this them from silver to gold to platinum.
capability level, a good data hygiene and good automation of the
engineering processes producing the capability is demonstrated.

IBM AI Applications AI maturity framework for enterprise applications 5

Figure 3: Maturity Framework Output – the Phases

3. Platinum
2. Gold
1. Silver

work
ty frame
AI Maturi

Maturity framework:
Detailed criteria
A. Impact on business

Sub Criteria Definition Silver Gold Platinum

% of Revenue and/or % of Minimal Contribution Significant Contribution Exceptional Contribution

Deal Wins influenced by
this AI capability – Revenue Influenced: – Revenue Influenced: – Revenue Influenced: >
< 5% 5%-20% 20%
If Revenue or Deal Win
info is not available, then: – Deal Wins Influenced: – Deal Wins Influenced: – Deal Wins Influenced:
< 5% 5%-20% > 20%
A1. Business impact
– Key contributor in an
analyst ranking (e.g., Analyst Ranking, Brand Analyst Ranking, Brand Analyst Ranking, Brand
IDC, Gartner) impression: Not a factor Impression: A minor Impression: A major
factor factor
– Key contributor in
increased brand
impression

Drives adoption of the Low Medium High

offering and broader
portfolio. – Usage: Increase of 1-5% – Usage: Increase of – Usage: Increase of 20%+
5-20%
For example: – NPS: Increase of 1-3 – NPS: Increase of 7+
points – NPS: Increase of 4-6 points
– Usage of the offering that points
A2. Portfolio impact could be measured in #
of time users log in, #
of licensed users or # of
API hits

– Increase in customer
satisfaction measured by
NPS scores

IBM AI Applications AI maturity framework for enterprise applications 6

B. Value to client

Sub Criteria Definition Silver Gold Platinum

Defined and documented Business value or ROI for Business value/ROI is Business value/ROI is
business outcome the AI project is unclear documented but the clearly documented, and
delivered by the AI OR still being developed metrics do not always tie- metrics are tied to overall
capability. This is the back to overall BU KPIs BU KPIs. Both short term
measurable “business” and long-term metrics are
outcome that would not documented and tracked
B1. Business process have been delivered if AI
outcome was not infused. Every
AI project must have
clear metrics (short term,
long term) by which to
judge whether the project
was a success (Metrics
examples: NPS, Wins)

Clearly defined Clear and unique Differentiators are Differentiators are

differentiator as a result differentiators are still documented and give the documented and
of the AI capability. being assessed and offering competitive edge sustainable. They give
These differentiators documented at present. However, the the offering a clear
B2. Differentiators
allow clients to select competition is catching competitive edge today
our offerings over the up and working on a and near future. Enabling
competitors’ comparable capability client to select our offering
over the competition

C. Technology sophistication

Sub Criteria Definition Silver Gold Platinum

Are the tools and Simple analytics with Prescriptive analytics, Adaptive Learning
technologies used predictive capability making recommendations and decision-making:
appropriate based on the generated from ML based on optimization or Learning and/or decision-
C1. Appropriateness
business challenge to rules making processes that
of technology to the
be solved (e.g., ML, NLP, dynamically learn through
business problem
NLU, RPU, ChatBot)? feedback and adapt
their strategies to new
conditions

Maturity of learning Offline training for Supervised, offline or Reinforcement learning,

technique embedded and a specific task or a online training for multiple Adversarial learning,
C2. Learning Techniques
demonstrated by the AI statistical technique to tasks utilizing regular techniques that learn how
model generate an insight feedback for learning to learn with experience

How easy is it to re-use Models are packaged into Models are deployed as Models deployed as
the DS model to allow containers that must be stand-alone, RESTful services, composed from
C3. Re-use of Data for scaling and to gain manually modified and services that applications re-usable building blocks
Science models operational efficiencies deployed for each new can easily call that can be re-arranged to
use case form different workflows
for different use cases

How effectively does Inner/Open Source Most AI capabilities All AI capabilities were
the technology make are used for some were built on Inner/Open built on and shipped
use of Inner Source components of the AI Source frameworks that with Inner/Open Source
and Open Source code capability. Common have been packaged frameworks. Inner/
to gain development examples would and shipped with the Open Source libraries
efficiencies, build be data validation capability. Common are regularly updated to
C4. Use of Inner Source
communities, mitigate and transformation, examples would the latest versions. Key
or Open Source
ethical/risk concerns and feature engineering include model training, functionalities and tests
encourage reproducible and visualization and serving, monitoring of the developed AI have
experiments? reporting and explainability. been contributed back
Experiments are internally to the Inner/Open
reproducible and open to Source project
communities within IBM

IBM AI Applications AI maturity framework for enterprise applications 7

D. Trustworthiness

Sub Criteria Definition Silver Gold Platinum

How do we ensure data Data dictionary and Data definitions and Data lineage is
integrity throughout its lineage are known at time lineage for multiple documented, tracked
lifecycle? of ingest projects are documented across transformations
Data Provenance and and used consistently. and published through
Data Lineage are known Developing a canonical the application
and documented. data model
D1. Integrity Understanding the
history and origins of a
data set as well as what
happened to data after it
was collected and prior to
its use

How do we ensure data Basic type validation and More advanced outlier Automated detection
quality throughout its density-based outlier detection, normalization, and correction for a
lifecycle? detection are reported interpolation and wide range of possible
Data quality is measured standardized quality issues; Cross table and
to understand common reporting multivariate validations.
D2. Quality
issues and information
content, what kinds of
corrections are made and
how improvements are
measured.

How do we reject bias Basic bias reporting Bias assessment and Bias assessment and
towards groups, sets relying on manual remediation are done for remediation are done as
of individuals, or data intervention for key features of data. a standard practice with
attributes? remediation proactive improvements
D3. Bias (Fairness)
Ability to prove that in the approach and
For complete list on measuring the outcomes are fair implementation of the
Bias in an application/enterprise, and not skewed either techniques.
please see the link given due to the model or the
in the conclusion section.
data (like incomplete,
limited/insufficient,
missing, corrupt, biased,
ambiguous)

How can we explain, in Explainability is not Explanations for Explainability is built

business terms, the logic available and AI solution Inferencing and Decision into the solution and can
behind how the AI came is largely a "Black box". Management are be provided based on
to its conclusion? However, efforts are available on a limited stakeholders' needs
Outcomes can underway to make it scale.
D4. Explainability be explained and available
backtracked to the data
and model that generated
them. Explanations
provided are tailored to
stakeholders' needs.

How can we shield AI Risk and security Policies, processes, and Risk and security
and AI infused services management are minimal standards are defined management are
against cyber threats and reactive, mainly and institutionalized comprehensive across
or adversarial attacks? relying on key actors. for security and risk the enterprise and
D5. Security Applications and management at a among partners and
algorithms are resistant consistent level across customers, allowing for
to attacks from either the enterprise and continuous feedback and
data manipulation or partners. improvement.
direct security flaws

IBM AI Applications AI maturity framework for enterprise applications 8

E. Ease of use

Sub Criteria Definition Silver Gold Platinum

How well can the AI Basic AI features or More mature AI Advanced AI capabilities
capability be used by the capabilities that increase capabilities that provide that are tightly integrated
intended end-user? Does usability and deliver value significant value and into the experience
it require experts (like and insights to end users. insights to users and and leverages Watson
Data Scientists) to be AI provides little or no helps them accomplish Moments that not only
able to use and interpret Explainability, which can their goals more significantly improve the
the outcomes? lead to low confidence effectively. AI clearly ease of use and efficiency
E1. Intuitive for use by levels. Users may have conveys its value and of the product, but also
the intended user to rely on external reasoning, enabling users enable users to be more
assistance to interpret to build confidence in productive and make
outcomes, provide the insights delivered. smarter decisions faster.
additional insights and/or AI delivers insights Users have a high degree
create custom solutions. that increase user of confidence in the AI
effectiveness, efficiency, and the unique insights
and satisfaction. delivered through the
offering.

F. AI operating model

Sub Criteria Definition Silver Gold Platinum

Uses tools that automate Manual build, integration, Use of continuous Deployment is
model building, data containerization and integration tools to automated using CI/
cleaning, and other key testing tools and build and test artifacts CD and operational
F1. Deployment (manual,
processes. Use of CI/CD processes are used for and containers for process is structured
automated)
pipeline deployment deployment with widely and enables dark launch
available status reporting in production/test
environments

How frequently can No established plan for AI model is trained at a AI model update is
models be updated or ongoing training of the AI regular basis based on automatic and is based
F2. Update frequency retrained? Is the process model human understanding of on the automated
automated? model quality assessment of model and
data quality

Infrastructure is scoped, Consistent usage of AI workload deployments AI workload deployments

designed, and deployed containerization. Ad are based on an are based on flexible,
to allow it to adequately hoc usage of siloed architecture that is centrally governed
scale with the solution technologies (chips, reviewed and modified architecture with
NVMe, composable occasionally. Architecture containerization and
F3. Infrastructure/ computing, cloud supports centralized auto-scaling to adapt
architecture scaling computing etc.) critical to containerization and to ongoing user needs.
AI workloads; no defined workflow management. Dynamically balancing
architecture. No use of Limited use of use of accelerated
accelerated computing. accelerated computing. computing and general-
purpose computing for
AI.

Analyzing and evaluating Basic logging of predictive Reporting of how the Alerting and automatic
the ongoing efficacy of accuracy and results model accuracy changes triggering remodel
F4. Monitoring
the model (e.g. model over time and the typical based upon continuous
drift analysis) lifecycle monitoring

IBM AI Applications AI maturity framework for enterprise applications 9

G. Data

Sub Criteria Definition Silver Gold Platinum

How easy is it to acquire Data acquisition Data acquisition Data acquisition and data
data and get it into the and preparation are procedures and platforms preparation are provided
system for analysis? individuals' responsibility are in place; acquisition as a service that supports
G1. Data acquisition &
with no offering-wide and preparation are real-time provisioning
instrumentation
guidelines. documented for all data. of all needed resources
including data sets,
expertise, and tools.

How robust is Rigid undefined Agile processes are well Agile processes are
governance, provenance, processes with potentially defined, standardized and continuously improved
and standardization of unpredictable outcomes. accepted to support early by quantitative feedback
data throughout the and continuous delivery based on IT and business
G2. Data management lifecycle? of data to data scientists/ metrics to support early
engineers. and continuous delivery
of data to data scientists/
engineers.

Illustrative example

In thinking about how to apply a maturity model, examples are often would have been had no action been taken. The KPIs used are directly
very useful. AI Applications uses anomaly detection in a number of tied to the metrics that ecommerce companies use to evaluate their
different applications in its portfolio. Here we describe a fictitious business performance. In its first few implementations, the solution
ecommerce application called Anomaly Finder as an example. We has shown significant benefits to clients based on real-time tracking
assume that the application was recently launched and has had limited of these KPIs. Therefore, it is rated platinum for business process
marketing activity. At a high level, this application actively looks at outcome. There are several features that clients and analysts have
a stream of key performance indicators (KPIs) for an ecommerce told us are novel in the market, but we believe that those moats could
business over time, finds any anomalies relative to the history of the be challenged in the next few years. We rated Anomaly Finder gold in
KPI, alerts the user to the anomaly, and then suggests a potential terms of its differentiators.
cause of the issue and a potential action to take if one exists for this
type of issue. In this section, we will walk through how we might have Technological sophistication tries to understand the maturity of the
evaluated this application in terms of its AI maturity. AI in the application based on the relevance of the techniques used
relative to the business problem being solved, the sophistication of
The impact on the business criteria is meant to be an evaluation of the learning methods used, and the ability to easily re-use or extend
an application’s impact on both IBM revenue and on the brand of those methods used to enable continuous improvements in the AI in
the product portfolio. Anomaly Finder was scored as silver for both future releases. The appropriateness of technology criteria tries to
business impact and portfolio impact. Since it was just launched is understand the level of reasoning that was required for the application
does not yet contribute a significant amount of revenue to the portfolio to effectively help the human in solving the business problem. A
and while it has been mentioned by analysts in a couple of reports, silver level provides the human with insights that they would not
it was not listed as a significant factor in the analyst’s rating for the otherwise have had but relies on the person to make any decisions.
portfolio. It has also only had a limited impact in pulling through sales Gold describes methods that enable the machine to make decisions
or increasing interest in other parts of the product portfolio. Overall, based on learned insights and input from the human as to the goals
its impact on business area has not been significant enough thus far to and constraints of the business problem. Platinum applications apply
rate it above silver. learning to the decision process itself, modifying their reasoning in
solving the problem based on learnings from previous actions they
The value to client criteria tries to measure how the application has have taken. Anomaly Finder uses a proprietary set of algorithms to
created business value in terms of the client’s defined business understand the behavior of other metrics at the time of the anomaly
objectives for their AI journey for that specific business process. It also and uses its understanding of those behaviors to suggest potential
tries to estimate how distinct the value of the application for improving causes of the anomaly and to recommend potential actions based on
that business process is relative to competitive solutions from the previous scenarios. It was rated gold for appropriateness of technology.
client’s perspective. One of the key components of Anomaly Finder is
Risk Avoidance Reporting. This module directly estimates the benefit of Learning techniques is an evaluation of the sophistication of the
following the recommended actions relative to what the outcome likely machine learning capabilities embedded in the application. Silver

IBM AI Applications AI maturity framework for enterprise applications 10

typically implies offline, batch-based learning or a complex analytic the anomaly and why a given recommendation was made. Since the
that provides a single type of insight. Gold techniques are generally explanations are only available for a limited set of solutions near the
supervised, may utilize online training or train a model for multiple recommended one, we rated the application as gold. Anomaly Finder
tasks based on feedback from prior actions. Platinum implies methods doesn’t take special measures to secure its AI capabilities but relies on
that can learn about how to improve themselves as a part of the the broader application platform for security. This platform, however,
learning process. They effectively learn how to learn as they gain has extensive, well defined and institutionalized security measures. We
experience. Anomaly Finder utilizes a random forest model that uses rated Anomaly Finder gold for Security.
feedback from prior actions and thus was evaluated as gold.
Ease of use criteria focus on how well the AI demonstrates its value to
The predictive and recommendation models for Anomaly Finder are the user in an intuitive manner, how easy it is for the user to understand
containerized and are deployed as services. However, those services the reasoning behind why a particular recommendation was made,
cannot be easily called by other applications. Therefore, the models and the extent to which it enables a user to accomplish their task
have limited re-use and were evaluated as Silver for that criteria. They without involving experts in IT or data science. All of the technical
would need to be made widely available to become gold and would math and engineering have been abstracted away in Anomaly Finder
need to be broken up into composable services (e.g., data cleaning, so that a line of business user can use a simple set of sliders to define
feature engineering, model estimation, hyperparameter tuning, model the relative importance of different business objectives they wish to
scoring) to become platinum. have considered in the reasoning behind the recommendations. The
explainer module helps the user visualize the data points that were
The predictive models in Anomaly Finder utilize both Inner Source and critical in making the recommendation. They typically only need access
Open Source code. They use a set of time series prediction capabilities to a data scientist or engineer in very complicated scenarios that
from an Inner Source library from IBM Research and several common may at first seem counterintuitive. Conversations with our users have
Open Source packages like pandas, scikit-learn, PyOD and Seaborn revealed some areas where we believe that the AI still needs tighter
in different parts of the AI pipeline. Both the Inner and Open Source integration with the overall experience for the application. Therefore,
packages are integrated into the Anomaly Finder containers along we rated Anomaly Finder gold for ease of use.
with test data so that experiments used to develop Anomaly Finder
can be reproduced. The team has not yet contributed some of the key The AI operations model criteria attempt to evaluate the maturity of
functionality back to the Inner Source library from Research. Anomaly the processes required to successfully install and maintain the AI in a
Finder would be rated Gold for use of Inner and Open Source. production or client environment. Deployment evaluates the maturity
of the processes and level of automation for building the AI from
Trust is of paramount importance in AI applications. As maturity grows, source code, testing and deploying it. Update frequency describes the
users will generally need to cede more control over business processes process for updating the training of AI models. Infrastructure scaling
to the application. They must believe that the application is making evaluates the AI’s ability to scale up or down based on workloads
reasonable decisions that take into account situational context in order to ensure performance and minimize operational costs. Monitoring
to allow this transfer of responsibility. Integrity is a measure of how evaluates an application’s ability to give the user visibility into and
trusted the data and transformations of data are in the application. manage the quality of the AI models over time.
Quality looks at how sophisticated the data and model validations
are in ensuring that the application is learning appropriate, relevant For deployment, Anomaly Finder utilizes continuous integration tools
behaviors. Bias measures how well the AI handles cases where biased for building and deployment but does not currently have dark launch
data or algorithms might affect the recommendations that it makes. or A/B testing capabilities. We rated it as gold for deployment. The
Explainability evaluates how well the application helps the user to predictive models are updated on a regular basis based on a scheduling
understand why it made a particular decision. Security measures how algorithm rather than on an automated assessment of model quality,
resistant the application is to be being tampered with or attacks from so the application was rated as gold for update frequency. The AI
unauthorized parties. workloads for Anomaly Finder are based on an architecture designed
specifically for their use. They are containerized and additional
The data used in Anomaly Finder conforms to a canonical data model compute resources can be added to increase scale. It does not have
and the sources of the data are documented. However, the application the capacity to auto-scale based on current needs and thus was
does not yet consistently track transformations or report on lineage as rated as gold for infrastructure/architecture scaling. Anomaly Finder
a part of the user application. We rated Anomaly Finder gold in terms has logging in both the prediction and recommendation models that
of integrity. The Quality of the data is validated using machine learning captures both accuracy and results. The application does not currently
techniques, some normalization is done and there is a published data keep track of these over time or have alerting that could trigger a
quality report. Anomaly Finder, however, currently relies on manual remodel and thus was rated silver for monitoring.
intervention to correct several potential data and model issues. It was
rated gold for Quality. The regression algorithms used in the application Access and respect for data are essential for any AI application. The
are tuned to reduce some types of bias, but they do not exhaustively criterion around data are intended to capture the ease with which
evaluate bias in the data. Anomaly Finder was thus rated silver for new data can be utilized by the application and the robustness of
bias. The application has an Anomaly Explainer module that uses the processes and systems in place for managing the data once it
the underlying data to help users understand the potential cause of has been acquired. The processes for data acquisition and ingest for

IBM AI Applications AI maturity framework for enterprise applications 11

Anomaly Finder have been standardized, documented and to some
extent automated. We have not yet created a standalone service that
supports all of the major aspects of these processes. Therefore, we
assessed that the application was gold in terms of data acquisition
and instrumentation. Data management assesses the care taken with
data in terms of understanding its provenance, governing its access
and usage to respect data ownership and standardizing how data is
delivered to the application for production usage and to data science
for continued research and development. Anomaly Finder has a
relatively static, standard set of processes that are used by our DevOps
team to ensure that the data is treated respectfully and to enable
continuous delivery of the necessary data to both production and
research environments. Since the processes are static, we rated data
management gold.

Based on this evaluation and in collaboration with Offering

Management, we would have defined new development objectives
that focus on making further investments in model re-use, monitoring
and data quality as the next steps forward in improving the maturity of
Anomaly Finder as an AI application.

Conclusion
As you apply AI to enterprise applications, establishing goals around the maturity of the AI is critical to ensuring that you deliver value to your
enterprise clients, use the appropriate level of technological sophistication, ensure that the AI is trustworthy and easy to use by your targeted
line-of-business users, and that you have an AI operating model in place to manage the AI you’ve deployed in the field along with strong
data management and data governance practices. We hope that this maturity framework will help you measure and progress your enterprise
applications on their AI journey.

Below you will find links to enterprise-grade data and AI management stack:

[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]
[Link]

[Link]

Below you can learn more about IBM’s Enterprise Applications infused with AI:

Domain Enterprise Application AI Capability (example)

Enterprise Asset Management Maximo Asset Performance Management

Integrated Workplace Management System TRIRIGA Watson Works

Supply Chain Management Sterling IBM Sterling Fulfillment Optimizer with Watson

Systems Engineering Software Engineering Lifecycle Management Requirements Management

Weather Solutions Weather Business Solutions Vegetation Management and Outage Prediction

IBM AI Applications AI maturity framework for enterprise applications 12

References
1. Gartner, Artificial Intelligence Maturity Model, Published: 18 March
2020 ID: G00466009

2. IDC MaturityScape: Artificial Intelligence 1.0, #US44119919

3. Ovum, How to Achieve AI Maturity and Why It Matters

4. The AI Ladder: Accelerate Your Journey to AI,

[Link]
dp/1492073431/

5. AI Maturity Assessment, [Link]

architecture/assessments/ai-maturity-assessment

6. Characterizing machine Learning Process: A maturity Framework,

[Link]

© Copyright IBM Corporation 2021

IBM Corporation
New Orchard Road
Armonk, NY 10504
Produced in the
United States of America
2021

IBM, the IBM logo, [Link], and Watson are trademarks

of International Business Machines Corp., registered in many jurisdictions
worldwide. Other product and service names might be trademarks of IBM
or other companies. A current list of IBM trademarks is available on the web at
“Copyright and trademark information” at [Link]/legal/[Link].

This document is current as of the initial date of publication and may be

changed by IBM at any time. Not all offerings are available in every country
in which IBM operates.

All client examples cited or described are presented as illustration of the manner
in which some clients have used IBM products and the results they may have
achieved. Actual environmental costs and performance characteristics will vary
depending on individual client configurations and conditions. Contact IBM to see
what we can do for you.

THE INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT ANY

WARRANTY, EXPRESS OR IMPLIED, INCLUDING WITHOUT ANY WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND ANY WARRANTY
OR CONDITION OF NON-INFRINGEMENT. IBM products are warranted according
to the terms and conditions of the agreements under which they are provided. The
client is responsible for ensuring compliance with laws and regulations applicable
to it. IBM does not provide legal advice or represent or warrant that its services or
products will ensure that the client is in compliance with any law or regulation.

IBM AI Applications AI maturity framework for enterprise applications 13