Why is AI

so hyped?
Risk, compliance, cyber
and audit in the AI Law

Prof. Hernan Huwyler MBA CPA

Director IE School Executive Education
Gartner has
identified AI risks
and security
management as the
#1 technology trend
for 2024
According to Gartner, applying
truth, risk, and security
controls to AI applications will
enhance decision-making
accuracy by eliminating 80% of
inaccurate and illegitimate
data by 2026
 Risks and controls
Prioritize AI
model governance,
trustworthiness,
fairness,
reliability,
robustness,
transparency, and
data protection
 Prioritize AI Control

Improve AI project Assess risks in AI- Continuously

success for driven applications monitor and test
increased business to prevent biases the AI model for
value and model control and promote accurate
precision and fairness explanations and
consistency interpretations
 Gartner Hype Cycle for AI
Prioritize AI Control
 AI impacts

Automate Fuel
work innovation

AI models can automate complex AI models can generate new ideas and
human tasks and decisions contents
• Self-driving cars and robots • Generate texts, code, images and
• Question answering
• Text translation videos
• Predictive modeling
• Content classification • Cause-and-effect relationships
• Capture data by computer vision • Data augmentation
AI-based software is not new
Profile clients,
Machine Provide insight for
candidates, and loan
learning decision-making
applicants

Natural Chat 24/7 with

Recommend actions
language clients to provide
on business flows
processing information

Detect faces, plates,

guns, protection Track and predict
Computer
equipment and other behaviors to warn
vision objects from video surveillance staff
footage
AI-based software is not new
Improve the efficiency of
Predictive Analyze large volume of
resources by predicting
analytics historical and real-time data
demand

Detect unusual user

Fraud Park transactions after a red
behaviors and other
analytics flag
abnormalities

Assist in verbal orders,

Speech Convert spoken language
transcript text and
recognition into written text
authenticate users
Massive use and investment are new
The techniques used by ChatGPT were
published in 2017; nevertheless, it
became the fastest-growing consumer
application in history after its late 2022
release, reflecting the significant impact
of interacting with this computer
program
EU Artificial
Intelligence
Act
_
 Regulation
vs
Innovation
In a context where Europe is trailing behind in the AI industrial
revolution, the burden of additional compliance costs,
restrictions, and liabilities will incentivize startups to relocate
to the United States and China
Rising demand for professionals
Need for highly skilled advisors
Talent shortage in a specialized expertise in AI, cybersecurity, risk
management, controls, and compliance within a close-knit expert
community

Impact on numerous organizations

Need to conduct new assessments and implement controls by
many AI developers, distributors, and all type of users

AI-specific risks and controls

New data-driven risk assessments and taxonomies, along with
highly specific control measures, need to be developed in
response to emerging threat vectors
 What is it?
Regulates the use and Global standard-setter on
marketing of AI the regulation of AI

Likely be adopted in early Fully enforced after 18

2024 months at least

Supported by the EU
Impacts AI software
Digital Services and
developers and sellers
Markets Acts

Excludes military, research Least 30 million Euro or

and open source uses 6% of global turnover
 Trilogue negotiations

EU Commission EU Council EU Parliament

Draft legislative proposals Approve or veto proposals Approve or veto proposals
Independent Co-legislative body Co-legislative body
27 Commissioners Represents governments Represents citizens
30k Public officers 27 Ministers 705 Members

Apr 21 Proposed the AI Act Nov 21 Changed social Mar 22 Amended the draft
Aug 21 Got 304 public scoring, biometric to ban social scoring and
consultations recognition systems and limiting the use of remote
high-risk applications biometric identification
Jun 23 Passed the AI Act
Dic 23? To debate a final version with 499 votes
May 24? To approve the AI Act
Artificial intelligence
• system designed to work autonomously
• software, data and hardware
• developed with machine learning, rule-based systems, statistical
approaches, and search and optimization methods (limits)
• processes data from both machines and humans to learn and make
decisions
• given set of human-defined objectives
• explicit or implicit objectives
• generates outputs like content, predictions, and recommendations
• affect its interactions with its environment
• Physical and electronic environment
Scope
A data quality management system is crucial for the proper functioning
of AI systems and the documentation of the controls

• Providers placing AI systems on the market or putting them into

service in the European Union, regardless of their physical presence or
location whether in the European Union or a third country
• Users of AI systems who are physically present or established within
the European Union
• Providers and users of AI systems physically present or established in
a third country, when the system's output is used in the European
Union
• Importers and distributors of AI systems
• Product manufacturers placing AI systems on the market or putting
them into service alongside their products, under their own name or
trademark
• Authorized representatives of providers established in the Union
 Main actors
Providers • developers of an AI system or components
• set the intended purpose of the model
• can be the end-user or not

• sellers of AI systems on the market under own name

• the course of a commercial activity, whether in return
Distributors for payment or free of charge
• targeting the European Union market
• inside or outside the European Union

Users • Person or company with the authority to use the AI

system
• Interact with the AI system following the instructions
for use
 Prohibited uses
Business cases
• Subliminal manipulation and exploitation
Significant • Social scoring of individuals based on social behaviors
potential for • Trace bank transactions, court orders and
government “discredit orders” to rank individuals,
manipulation
incentivize some behaviors and blacklist services in
a social credit system using AI models
unacceptable • Real-time remote biometric identification in public
risk spaces to recognize faces, bodies and movements in real
time
• Trace individuals in a online mass surveillance face
recognition in parks and malls by the police
 Prohibited uses
Added by the European Parliament

• Full ban on AI for biometric surveillance covering

Significant real-time and post-event remote biometric identification
potential for for police and all users, exempting the prosecution of
serious crimes after judicial authorization
manipulation • Biometric categorization systems using race, gender,
lifestyle and other sensitive characteristics
unacceptable • Prediction of potential criminal activities based on
risk profiling, locations, or past behaviors
• Scrape of pictures from the internet or footage to to
create facial recognition databases
• Emotion recognition in policing, schools, and
workplaces
Manipulation
Subliminal stimuli
• change behaviors or preferences without consciousness
• affect the voluntary and informed consents (Anthropomorphy)
• aimed to increase sales, spent time and attention or to
create harm
• using imperceptible messages, inaudible sounds, noise,
light waves, scents, colors, vibrations and sensations
Interactions with an AI system
• online games and add-ons
• AI-assisted toys, wearables, assistants and robots
• online advertising
Focus on vulnerable groups
• age, physical and mental disability
• vulnerable to addictions
Social scoring
Increase credits or points if
• donate money, blood and charity work
• praise the authoritarian president
Decrease credits or points if
• not visit your parents and official religion event
• demonstrate against corporations or authorities
• criticize the government in social media
AI model to integrate data and deny access to
• loans, insurance and internet
• admission and scholarships to education
• government jobs and permits
• international and domestic travel
• children to be assigned to social care
Public survelliance
Banned real time monitoring
• Allowed remote uses of the collected footage when the
individual is no longer present (significant delay)
Banned use by the police and law enforcement agencies
• Allowed for individual businesses (malls, schools),
municipalities, and central government
Banned the biometric identification
• Banned for identify individuals against a watchlist of
people of interest as an unacceptable violation of
rights to privacy and free expression and assembly
• Allowed for biometric validation against pre-stored
individual personal data to access doors (airports,
stadiums) and unlock the use of devices
 Higly controlled uses
Business cases
• AI is a safety component or a product subject to
Significant existing safety standards and assessments
• Use an AI model to verify a diagnosis and
potential for
maintenance in a medical devise
harm to • AI has a sensitive purpose
fundamental • Use AI in biometrics, critical infrastructure,
rights education and vocational training, employment,
workers management, access to essential services,
law enforcement, migration, asylum and border
high risk control management, administration of justice and
democratic processes
• AI that can impact voters and make recommendations
on social media platforms with +45M users
 Higly controlled uses
Business cases
• Law enforcement under discussion
high risk • Identification of individuals with a significant time
delay
• Assessment of the likelihood of an individual
committing or being a victim of a crime
• Prediction of potential criminal activities using
historical behavior, social connections, and profiles
• Aggregation of data from various sources into a
profile for investigating detained individuals
• Emotion detection (e.g., polygraph analysis)
• Evaluation of the credibility of evidence
 Higly controlled uses
Business cases
• AI models used for the safety of
high risk • Cars, jet boats, planes, vessels and trains → self
driving, operation of traffic
• Medical devices → diagnosis, monitoring
• Operation supply of water, gas, heating and
electricity
• Machinery, lifts and cable installations
• Personal protective equipment → video detection,
use alerts
• Fuel appliances → energy storage and
management, internet of things
• Toys
 Higly controlled uses
Requirements
• Conduct risk and control assessments
high risk • Assess health, safety, discrimination and privacy
risks to demonstrate and document controls on the
AI model
• Perform controls on the AI model
• Monitor and be able to explain the use of data, the
human oversight, the accuracy of the model and
the cyber security controls
• Ex-ante self-certify the conformity of the AI
systems and governance
• Report serious incidents to authorities
• Register in EU-wide public database
Provider controls
Demonstrate the appropriate accuracy, robustness and cybersecurity

• Assess risks to health, safety, fundamental rights, natural

environment, democracy and rule of law
• Ensure the quality of the training data (quality system)
• Prevent biases and black boxes affecting outputs
• Protect access to modify the AI model
• Undergo conformity assessment and reassessment for significant
modifications
• Register the AI system in EU database
• Ensure human oversight
• Keep automatic logs on the functionality of the model lifecycle
• Maintain technical documentation and user guideline
• Collect data on reliability, performance and safety for post-market
monitoring
• Inform summaries of the copyrighted data used for training
User controls
Address risks in the intended uses

• Operate AI system in accordance with instructions of use and

consistently for their intended purpose
• Ensure human oversight on the outputs
• Monitor and address risks in operation
• Inform the provider or distributor about any serious incident or any
malfunctioning

Risk of shadow AI
• Use software, in particular add-ons and web applications, without
knowing that there is an embedded AI algorithm
Risk assessments
Implement and document risk assessments in all the AI lifecycle
integrated into the already existing risk management procedures

• Assess risks: Examine risks affecting health, safety, and

fundamental rights, considering the intended purpose and the
children’s vulnerabilities
• Incorporate incident data: Integrate incident and post-market
monitoring data into risk analysis as a continuous iterative process
• Test lifelong controls: Evaluate controls at all stages of the AI
lifecycle to refine risk analysis and evaluate components
• Implement balanced safeguards: Establish controls to mitigate or
eliminate risks in AI systems
• Educate users: Inform and train users about identified risks,
including those that cannot be mitigated
Known and foreseeable risk
Facilitate the identification of imaginative risk scenarios involving architects,
modelers, data scientists, ethical experts, privacy and compliance officers, as
well as users

• Biased data: Gender, discrimination, and racial bias may be present in both
training data and feedback, leading to potential discrimination, issues with
protected characteristics
• Misuse: Improper or malicious use of your AI system for fraud,
misinformation, cyberbullying, private purposes, cyberattacks or crime
• Wrong decisions: Unacceptable error rates can result in false positives,
impacting the system's reliability
• Inability to detect unwanted behaviors: Unacceptable false negatives in the
model may hinder the system's effectiveness
• Lack of prior testing: Using an AI system that has not been previously tested
or tested in a different environment can introduce uncertainties
• Performance Issues: AI system performance may degrade, affecting its
overall functionality and utility
Impact assessments
Risks must be assessed measuring the health, safety and fundamental rights impact of the
following scenarios

• Human dignity → financial and psychological well-being, loss of control on decisions

(manipulation), misuse for fraud and crime, safe environment
• Privacy → personal and family life, protection of personal data
• Free expression and information → deep fakes and misinformation, filter bubbles
• Free assembly and association → censor online discussions, public surveillance to
discourage demonstrations, online browsing data to predict individuals' associations
• Non-discrimination and disabilities rights → biased decision-making, exclusion
• Consumer protection → prices increases in vulnerable conditions, client type exclusions
• Workers´ right → job displacement, over monitoring, discriminatory recruiting
• Effective remedy. fair trial, defense and the presumption of innocence → predictive
policing lead to biased profiling and arrests, overreliance on AI to decide bail
determinations and judicial actions, manipulation of legal evidence
• Good administration → access to public services, segregate data invisibles
 Cyber security risk
Implement and document risk assessments to ensure the AI trustworthiness
objective in all the AI lifecycle

AI creates new threat vectors and weaknesses potentially exploited by cyber

threat actors

• Align the taxonomy and control for privacy and ISO 27000 requirements
• Incorporate vulnerabilities on both, software and the hardware
• Implement data integrity controls on interfaces (governance)
• Log and monitor the user activity, decisions and events (record-keeping)
• Implement the security by design at the model conception (transparency)
• Collect and address reported vulnerabilities (post-market monitoring)
 Confidentiality Integrity Availability
Disclosure: employee or Poisoning: attacker may Denial of service: attacker may
consultant may expose the model manipulate the training data, flood the system with numerous
or training data through either aiming to distort outputs, requests or data to overwhelm its
negligence or intention introduce backdoors, or sabotage capacity and slow down service
the model, causing it to produce consumption
Oracle: attacker may input the inaccurate predictions or
model to analyze its outputs, classifications Perturbation: attacker may input
aiming to reverse engineer and data to exploit the model's
extract either the model itself or Evasion: attacker may modify a fragility, intentionally inducing
the training data minor portion of the training data errors or unexpected behavior
to trigger significant changes in
outputs, thereby influencing Abuse: user may misuse the
decisions made by the model model for fraudulent or unethical
purposes
Bias: developer may use
incomplete training data leading Third party: vendor may fail to
to discriminatory outcomes deliver expected supporting
services
Data governance
A data quality management system is crucial for the proper functioning of
AI systems and the documentation of the controls

• Data management: implement quality and feasibility requirements and

ownership for training, validation, and testing data sets (statistical
properties, free of error, complete, representative, relevant)
• Representativeness: ensure that data sets are relevant, representative,
and possess the appropriate statistical properties with respect to the
targeted individuals
• Accuracy: ensure that data sets are free of errors and as complete
considering technical feasibility, state of the art, data availability, and
risk management measures
• Privacy-preserving techniques: protect, minimize and avoid the use of
personal data, in particular, sensitive personal data
• Contextual considerations: ensure that data sets accounts for features,
characteristics, and elements specific to the geographical, behavioral, or
functional context in which the AI system will be used
Quality management system
Providers should implement a control system to ensure the quality of the AI model and data
proportionate to the size of the organization

• Procedures to ensure compliance, system management, and modification

• Risk assessment to justify the implementation of balanced controls
• Procedures for allocating responsibilities to staff
• Controls and tests for designing, verifying, and developing the system
• Controls to prevent unauthorized or banned system uses
• Controls for data collection, storage, retention, analysis, labeling, mining, aggregation,
sharing, and all operations for the intended use
• Procedures for setting up and conducting post-market monitoring
• Procedures for reporting incidents and notifying authorities and users
• System for producing and storing documentation
• System for managing resources and third-party relationships
Conformity assessments
Primary providers, as well as distributors and
importers, need to assess their internal controls either
through internal employees or third-party auditors for
periodic checks

• Verify that the established quality management

system is in compliance with the requirements
• Examine the information in the technical
documentation to assess whether the requirements
for high-risk AI systems are met
• Verify that the design and development process of
the AI system and its post-market monitoring is
consistent with the technical documentation
 Permited special uses
• Impersonation: Inform users that there is an interaction
with AI at the time of the first interaction
• Chatbots, virtual assistants and interactive games
Low potential • Biometric categorization: Inform users
impacts with • Fingerprint, facial, voice and iris recognition

additional • Typing and signature speed and walking style

• Image, audio and video generation: Disclose that the
transparency content was generated by AI, prevent illegal content,
requirements prevent deep-fakes, and publish the copyrighted data
used for training
Limited risk • Emotion recognition: Inform users
 Unrestricted uses
• Spam filters
• Recognize common keywords, phrases, senders,
headers, user feedback and elements to classify
No potential emails as spam
impacts • Video games
• Animate non-player characters as enemies in
combat, allies and quest givers to adapt the
No risk difficulty
• Content recommendation algorithms
• Language translators
• Weather forecasting
Prof. Hernan Huwyler
Compliance, Risk and Governance
Academic Director at the IE Law School

Artificial Intelligence Risks, Quantitative

Risk, Audit, Cyber Security

Zürich, Zug, Genève, Copenhagen, Madrid

hernanwyler

hewyler