abstract AI CAN DELIVER NUMEROUS BUSINESS
BENEFITS. BUT AUTOMATION WITH AI
COMES WITH A NEW SET OF CHALLENGES
AND RISKS. ORGANISATIONS WILL ONLY
MAKE A SUCCESS OF AI IF THEY CAN
EFFECTIVELY GOVERN WHAT SHOULD BE
AUTOMATED (NOT JUST WHAT COULD BE
AUTOMATED).
This article considers automation with AI and concludes that automation with AI must be:
• Ethical – to ensure that the automation meets ethical principles and policy;
• Controllable – to provide effective and timely control;
• Viable – to ensure that the desired outcomes can be delivered without unforeseen
circumstances;
• Desirable – to ensure that the business case for it includes thorough understanding, analysis
and mitigation of the AI-specific challenges and risks as well as the benefits.
2 of 9
What is Automation enables processes to be organised differently so that more responsibility is transferred to
technology. This can improve:
automation? • efficiency and productivity—by reducing error rates and improving process throughout and
pace;
• compliance with regulation and policy—by understanding the level of compliance, detecting
non-compliance and providing mitigations with respect to security, safety, ESG and other
organisational obligations;
• the experience of people involved in the process—by freeing them up from routine activities,
giving them better tools to conduct their work and providing better interaction with
technology;
• the range and quality of data and analytics to support further improvements.
Organisations have been automating their processes for many decades, so what is different about
AI? There are many different taxonomies for AI (for example, those provided by Russell and Norvig,
Poole and Mackworth or the different perspectives offered in the Creation of a Taxonomy for the
European AI Ecosystem). Figure 1 provides a simple summary.
Figure 1: AI and automation capabilities
The different taxonomies (and Figure 1) show that the scope of AI and the range of different
applications is very large. But it cannot just be treated just like another new technology. The key to
understanding the impact of AI on automation is that AI is a different type of information processing.
It's very different from conventional IT and (despite the name) it's very different from the information
processing that people and teams carry out. These differences introduce new and different types of
risks.
This is not just a theoretical issue because the pace of technology development is rapid. This is
creating a perfect storm:
1. Growth of AI. The use of AI in organisations is growing fast. It will become pervasive and
embedded in multiple forms in business processes.
2. Risks. Deliberate or inadvertent misuse of AI (for example, through “artificial stupidity or
other causes”) risks triggering reputational, compliance or financial damage. At a wider
scale, the proliferation of AI, and increased energy use, may inhibit sustainability.
3. Urgency. The problem cannot be deferred. An ethical approach to AI and automation needs
to be designed in and cannot be retrofitted.
Without a clear approach to AI and automation, AI will be in the hands of business users and
technical teams driven by financial and delivery pressures without a clear understanding of the risks
or the means of mitigating them.
3 of 9
So, what are the differences that organisations need to recognise and respond to? There are the
following five considerations:
• The commoditisation of AI and scale;
• The ethics of AI;
• Automating access to organisational knowledge;
• Maintaining control as the complexity and rate of change increase;
• Understanding how to change to incorporate the new realities of AI.
Commoditisation of AI and Scale
Automation technology is enabling AI to be used much more easily and the promise of intelligent
automation is becoming a reality. Many people will be able to automate activities using AI for
themselves, without involving professional teams.
There is a considerable focus on improving the ability of AI teams to create and deploy models
through technologies like MLOps (machine learning operations). But AI is increasingly present in
software products and services, partly because there are so many AI tools for developers to use
(through libraries for Python and other languages as well as ubiquitous cloud services).
More than that, AI is built into so-called no/low code tools like the Microsoft Power Platform and many
others. These tools reduce the level of skill required for development (by enabling users to develop
through a straightforward user interface rather than by writing code).
Implementing using AI is no longer the preserve of highly expensive data science and AI engineering
teams. Ordinary developers can use AI and even so-called citizen data scientists and citizen
developers, without any professional expertise, can use it.
This means that the governance of AI implementation has a challenge with scale. Any control
mechanisms need to apply outside the domain of specialists.
The ethics of AI
Many ethical questions are likely to arise when machine learning is implemented (machine learning
has driven the recent increase in AI). Machine learning uses data that, in itself, reflects the biases
and ethical issues of society. So, “bias is machine learning’s original sin” and comes in many forms.
In order to address ethical concerns, regulators are well aware of the need for regulation. For
example, the EU has published their proposed approach and the US Administration has published
their Blueprint for an AI Bill of Rights.
Organisational Knowledge
AI and automation also provide the possibility of automating access to organisational knowledge,
whether this is through chatbots, document analysis, providing AI explanations or through improved
human collaboration. This means that AI can both improve access to existing organisational
knowledge and also generate new knowledge.
Control
Because of the power of AI, increasing automation introduces a new concern: control. There are two
key questions:
• How should the use of AI be controlled so that it meets the ethical requirements?
• How can it be controlled fast enough to avoid unfavourable outcomes? Automation can
accelerate the pace of action and can introduce greater complexity that people will have
difficulty understanding and making decisions about.
Organisational Change
Many of the issues associated with AI and automation concern people. Automation carries a threat of
job losses affecting different roles differently even if there is also the potential for the creation of new
jobs. The key is trust—AI will not be successful if it is not trustworthy.
4 of 9
AI enables relationships between technology and people at a greater cognitive depth and can enable
very different types of business process. As a result, automation with AI at scale requires a rigorous
approach to the design of business processes and the relationship between people and automation
and a careful organisational change approach that recognises both the need for trust and the extent
of the change.
What should be automated?
These considerations mean that organisations need to be very clear about what should be
automated, not just what could be automated. AI needs to address the four topics shown in Figure 2.
AI must be:
• Ethical—to ensure that the automation meets ethical principles and policy;
• Controllable—to provide effective and timely control;
• Viable—to ensure that the desired outcomes can be delivered without unforeseen
consequences;
• Desirable—to ensure that the business case for it includes thorough understanding, analysis
and mitigation of the AI-specific challenges and risks as well as the benefits.
Figure 2: What should be automated?
5 of 9
An ethical approach to automation is essential. But implementing AI ethically is hard: !Organizations
Ethical can start with an admission: Getting serious about ethical AI is no easy task”. An incremental but
automation transformational approach is required that embeds the five components of ethical transformation:
1. AI governance: to define the direction of travel, the priorities and the changes required;
with ai 2. Analytics: to understand the impact of AI ethics challenges and how to resolve them;
3. AI implementation: to measure adherence to AI ethics principles in the development and use of
AI;
4. AI ethics enablers: to measure the maturity of the enablers (or inhibitors) of ethical AI;
5. Transformation: an incremental approach to learning and improvement.
Without these components, it will not be possible to understand the ethics of AI but also its viability
(see Figure 2).
Controllable automation with AI
How should control over the design and use of AI be exerted? To answer this question, it is important
to understand what control needs to achieve. As discussed above, there are some particular control
challenges.
The fundamental question is this: how should autonomy be balanced between humans and AI?
Implementation of the ethical principles implies the need for high levels of human control. But how
should this be achieved--what mechanisms should be used? Control can be applied at many different
levels including, for example:
• Society and the market: through regulation and accreditation;
• Organisation: through culture, AI governance (as above), policy, operating model design,
process design and business analysis;
• User: through user research and user experience;
• AI implementation: through model design, assurance and testing.
The need for high levels of human control has long been argued by Shneiderman (2022) in his book
"Human-Centred AI", amongst others, in line with well-established models for digital development.
But Shneideman's analysis is not about placing control just with either humans or AI. He argues for
processes and interfaces that exhibit "high levels of human control AND high levels of automation"
and suggests different types of control. This approach can result in increased human autonomy and
closer alignment with ethical principles.
The increasing automation of AI enables faster decision-making or action either through direct
automation or by switching decision-making closer to where it is needed. In either case, any
unfavourable outcome will be triggered faster. In the worst case, there is a risk that automation will
both contribute to, and accelerate, the type of event described by Taleb (2007) in his book “The Black
Swan: the impact of the Highly Improbable”. These are events that generate very unfavourable and
damaging outcomes (for example, the financial crash of 2007/8) but which are rare (and therefore not
present in data or organisational knowledge).
The commoditisation of AI means that non-professionals (for example, any business user) may have
access to AI through no/low code technology. Control mechanisms that applied to (relatively) small
teams of AI professionals or developers will not work at much greater scale. How can the right
balance between empowerment and control be established? What guardrails will be needed to
control at that scale?
A thorough analysis of these questions is needed to understand how to apply tradeoffs to automation
with AI. Tradeoffs are required throughout the design and implementation process. They concern the
balance between different types of requirement (for example, explainability vs model effectiveness),
architecture (for example, the use of edge AI vs centralised AI), prioritisation of expenditure (for
example, on data quality improvements vs model development) and many others. The risk in each
case is that tradeoffs are made implicitly or without sufficient scrutiny resulting in unexpected results.
Mitigating these risks requires iterative, cross-functional communication.
6 of 9
Viable automation with AI
People can be nervous about the implementation of AI because of its potential effect on jobs and
the changing relationship between technology and people. A human-centred approach (discussed
above) can mitigate some of the risks but, nevertheless, there are a number of uncertainties to
take into account.
Many organisations have found it difficult to scale AI implementations. Therefore, it is important to
understand if the capabilities required for implementation are sufficiently mature with respect to
AI. These include the following:
• Ethical assurance and governance;
• Data governance;
• AI implementation;
• Human-centred digital implementation capabilities (like process design, user research
and user experience);
• Organisational change.
Without these, is the proposed implementation viable?
Desirable automation with AI
The benefits that AI can deliver are summarised above but the implementation of AI needs a
business case that recognises the particular characteristics of AI. As well as the contents of
conventional business cases, there are a number of AI-specific questions to address:
• How well does the planned approach meet the ethical requirements?
• Is it sufficiently controllable?
• Is it viable?
• Have the tradeoffs been fully considered?
But more than this:
• Does it just automate current processes or consider how to improve them to take
advantage of the unique capabilities of AI?
• Will the implementation and the use of the AI be trusted:
o Is the proposed implementation fully understood? Are the risks and impacts, on
people and organisation, as well as technology, well understood and fully
mitigated?
o Have the organisational and people changes required been planned and can
they be implemented effectively?
o Will people trust the AI?
7 of 9
Conclusion AI is very different from the processing of conventional IT and (despite the name) it's
different from the processing that people carry out. The scale of these differences
means that AI cannot just be treated just like another new technology.
Automating with AI needs a thorough consideration of the four factors shown in Figure
2. AI must be:
• Ethical—to ensure that the automation meets ethical principles and policy;
• Controllable—to provide effective, human-centred control;
• Viable—to ensure that the desired outcomes can be delivered without
unforeseen consequences;
• Desirable—to ensure that the business case for it includes thorough
understanding, analysis and mitigation of the AI-specific challenges and risks.
8 of 9
About us
capgemini Capgemini is a global leader in consulting, digital transformation, technology
and engineering services. The Group is at the forefront of innovation to address
the entire breadth of clients’ opportunities in the evolving world of cloud, digital
and platforms.
The ucl school The UCL School of Management is the business school of University College
London, one of the world’s leading universities, consistently ranked in the
Of management global top 20 for its academic excellence and research. The School offers
innovative undergraduate, postgraduate, PhD and executive programmes in
Management, Entrepreneurship, Business Analytics, Business Information
Systems, and Finance, designed to prepare students for leadership roles in the
next generation of innovation-intensive organisations.
The analytics The Analytics Lab is an enrichment module for business students where they
are able to explore topical questions in the domain of business analytics and
lab digital economy via hands-on experience. Students are offered the opportunity
to conduct research and work on projects with leading technology service and
consulting companies.
It aspires to help UCL business students and alumni to be in the heart of
fundamental changes and digital transformations in the business environment.
Students enhance their practical abilities to manage and operate business
activities effectively in view of rapidly developing digital and technological
advancements in data analytics.
9 of 9