Module3

• Moral Framework of Justice in AI: on the

Limits, Failing and Ethics of Fairness,
Accountability in Computer Systems-
Responsibility and AI, The concept of Handoff
as a Model for Ethical Analysis and Design.
• AI systems are increasingly used in important areas
such as hiring, loan approval, criminal justice,
healthcare, and education.
• Because these systems influence people’s rights and
opportunities, questions of justice and fairness become
very important.
• The moral framework of justice in AI examines
whether AI systems treat individuals and groups
ethically, fairly, and responsibly.
Justice in AI:
• Justice generally refers to fairness, equality, and
protection of human rights.
• In the context of AI, justice means:
– AI systems should not discriminate.
– Individuals should receive fair treatment.
– Human dignity and rights must be respected.
– Decisions should be accountable and explainable.
• AI systems do not have moral values of their own.
• They reflect the values and assumptions of the
people who design and deploy them.
• Understanding Fairness in AI
– Fairness in AI usually refers to the absence of bias
and discrimination.
• An AI system is considered fair when:
• It does not favor or disadvantage individuals based on
race, gender, religion, caste, or economic background.
• It provides equal opportunity.
• It applies consistent standards in decision-making.
• However, fairness is complex because it can be
defined in different ways, and these definitions
may sometimes conflict.
• Defining Fairness
• Individual Fairness
– Similar individuals should be treated similarly.
• Group Fairness
– Equal outcomes across different demographic
groups.
• Procedural Fairness
– Transparent and consistent decision-making
processes.
• Distributive Justice
– Equitable allocation of benefits and burdens.
 Biased Training Data in Healthcare AI

• Ethical Implications of Biased Training Data

• Sources of Bias
– Historical data reflects past inequities.
– Under-representation of minority groups.
– Socioeconomic and geographic data gaps.
– Label bias from medical professionals.
• Ethical Consequences
– Higher misdiagnosis rates in marginalized groups.
– Perpetuates systemic health disparities.
– Erodes patient trust in AI systems.
– Unequal access to accurate AI-assisted care.
 Healthcare AI Bias – Real World Impact

• Pulse Oximetry Devices

– Devices designed and tested primarily on lighter skin tones showed less
accurate readings for darker-skinned patients.
– This contributed to delayed COVID-19 diagnoses and treatment.
• Dermatology AI Algorithms
– Skin cancer detection models trained mostly on light-skinned datasets
performed worse on darker skin.
– Resulted in higher rates of missed malignant lesions.
• Kidney Function Estimation
– Race-based adjustments in eGFR algorithms historically classified Black
patients as having better kidney function than they did.
– This delayed referrals for specialist care and transplant eligibility.
• Key Takeaway:
Bias in healthcare AI is not just a technical flaw — it is an ethical
failure that can reinforce systemic inequality and cause real harm.
• The Impossibility Theorem
– Mathematical proof shows that multiple fairness criteria (e.g., equal
accuracy, equal false positive rates, calibration) cannot all be satisfied at the
same time.
– Trade-offs are unavoidable.
• Fairness is Context-Dependent
– What counts as fair varies across cultures, legal systems, and use cases.
– No single universal fairness metric exists.
• Proxy Discrimination
– Removing protected attributes (e.g., race, gender) does not eliminate bias.
– Correlated proxies (e.g., zip code, name) can still produce discriminatory
outcomes.
• Feedback Loops
– Biased AI decisions create biased outcomes.
– These outcomes reinforce biased training data, compounding inequality
over time.
• Measurement Problems
– Training data often reflects existing societal biases.
– Fairness evaluations may encode these biases instead of correcting them.
 Human Oversight in AI Systems

• Role of Human Oversight in Maintaining Fairness &

Accountability
• Algorithmic Auditing
– Human experts periodically review AI decisions for disparate impact.
– Ensures no group is systematically disadvantaged.
• Interpretability & Explainability
– AI systems should provide human-understandable reasoning.
– Supports accountability and meaningful review.
• Diverse Review Boards
– Include stakeholders from affected communities.
– Bring lived experience into technical governance.
• Override Mechanisms
– Clear human override protocols for critical decisions.
– Essential in high-stakes areas like justice and healthcare.
– Ensures critical decisions remain under human control.
• Different moral approaches:
• Equality-Based Ethics
– Treat everyone the same.
• Equity-Based Ethics
– Treat people differently to correct disadvantages.
• Rights-Based Ethics
– Protect privacy and human dignity.
• Utilitarian Ethics
– Maximize overall social benefit.
• “Different ethical theories lead to different AI
design choices.”
 Accountability in Computer Systems.

• Computer systems are widely used in modern

society for important decisions such as:
• Banking transactions
• Medical diagnosis
• Autonomous vehicles
• Recruitment systems
• Criminal justice
• When these systems make mistakes or cause harm,
an important question arises:
– Who is responsible for the decision made by the
system?
• This leads to the concept of Accountability in Computer
Systems.
• Accountability ensures that there is clear
responsibility for the design, operation, and
outcomes of computer systems, especially those
using Artificial Intelligence.

• Accountability helps ensure that technology is used

safely, ethically, and responsibly.
• Definition of Accountability
• Accountability refers to the ability to identify, trace,
and hold responsible individuals or organizations for
actions taken by computer systems.
• In simple terms:
• Accountability means ensuring that someone is
responsible when a computer system causes an error,
harm, or unfair decision.

• Accountability helps in:

• building trust in technology
• ensuring ethical use of systems
• preventing misuse of AI technologies
• Accountability is very important in artificial intelligence systems for several
reasons.
• First, it helps build trust between users and technology. People are
more likely to trust AI systems when they know that the system’s
decisions can be explained and verified.
• Second, accountability helps prevent misuse of technology.
Without accountability, AI systems could be used in ways that harm
individuals or society.
• Third, accountability helps identify and correct errors. When
systems fail or produce incorrect results, accountability allows
developers to understand what went wrong and improve the
system.
• Finally, accountability helps ensure that organizations follow ethical
and legal standards while developing AI technologies.
Components of Accountability:
• There are several important principles that support
accountability in AI systems.
• Transparency
– Transparency means that the system’s operations and
decision-making processes should be clear and
understandable.
• Users should know how data is collected, processed,
and used by the system.
• Explainability
– Explainability means that AI systems should provide clear
explanations for their decisions.
• For example, if an AI system rejects a loan application, it
should explain the reasons for the rejection.
• Fairness
– AI systems should treat all individuals equally and should
not produce biased or discriminatory results.
• Bias can occur if the training data used to develop AI models
contains historical inequalities.
• Responsibility
– Developers and organizations must take responsibility for
the outcomes produced by their AI systems.
• They should ensure that systems are properly tested and
monitored.
• Auditability
– AI systems should allow independent auditing and
evaluation to ensure that they operate correctly and
ethically.
Challenges in AI Accountability:

• Although accountability is important, implementing it in AI

systems is not always easy.
▪ One major challenge is the use of complex machine learning
models, especially deep learning systems.
▪ These models often function as black boxes, meaning it is
difficult to understand how they arrive at a particular decision.
▪ Another challenge is data bias. AI systems learn from data, and
if the data contains bias, the system may produce unfair
results.
▪ There is also the challenge of multiple stakeholders involved
in AI development, including developers, data scientists,
companies, and users.
▪ Determining who is responsible when something goes wrong
can be complicated.
▪ Additionally, laws and regulations related to AI are still
evolving, making accountability more difficult to enforce.
Ensuring Accountability in AI Systems
To ensure accountability in AI systems, organizations should
follow several practices.
• They should develop ethical AI guidelines and ensure
that AI systems are designed responsibly.
• Regular algorithm audits should be conducted to detect
bias and errors.
• Proper documentation of data and models should be
maintained to improve transparency.
• Human oversight should be included in critical decision-
making processes.
• Finally, governments and regulatory bodies should
establish clear policies and standards for AI
development.
• Role of Engineers and Developers
• As future engineers and AI professionals, have an
important role to play in building responsible
technology.
• They should focus on developing systems that are fair,
transparent, and explainable.
• carefully select datasets, test models thoroughly, and
consider the social and ethical impact of our
technologies.
• Responsible AI development helps create systems that
benefit society while minimizing risks.
 Examples of Accountability Issues in AI

• Example 1: Self-Driving Cars

• Companies such as Tesla and Waymo are developing
autonomous vehicles.
– If a self-driving car causes an accident,
responsibility could lie with:
• the car manufacturer
• the software developer
• the owner of the car
• the AI system itself
• This creates legal and ethical challenges.
 Example 2: AI in Healthcare

• AI systems are used for:

– disease prediction
– medical image analysis
– treatment recommendations
• If an AI system suggests an incorrect diagnosis,
responsibility could lie with:
• the hospital
• the doctor using the system
• the AI developers
– Currently, doctors are expected to verify AI
recommendations before making final decisions.
 Methods to Ensure Accountability in AI Systems

• Several strategies can improve accountability.

1. Explainable AI (XAI)
– Explainable AI focuses on making AI systems easier to
understand.
– These systems provide reasons for their decisions.
• Example:
Explaining why a loan was rejected.
2. AI Governance Policies
– Organizations develop internal guidelines for
responsible AI use.
– Companies like Google and IBM have published AI
ethics principles.
[Link]
• Developers should document:
– datasets used
– algorithms applied
– system limitations
• This helps identify problems when they occur.
[Link] Audits
– Independent experts should periodically examine AI systems
to check for:
• bias
• accuracy
• fairness
[Link] Regulations
– Governments are introducing laws to regulate AI technologies.
• These laws require organizations to take responsibility
for the systems they deploy.
The Concept of Handoff as a Model for Ethical Analysis
and Design

• Modern computer systems are complex and involve

many participants such as:
• software developers
• system designers
• organizations
• users
• regulators
• During the development and use of technology,
responsibility is often transferred from one person or
group to another.
• This transfer of responsibility is called Handoff.
• A handoff occurs when:
– A human gives control to an AI system
– An AI system returns control to a human
– Control shifts during task execution
• It focuses on “who is responsible at each stage of a task.”
• Handoff as an Ethical Model
• The handoff model helps evaluate:
– Who makes decisions?
– Who is accountable for outcomes?
– Whether the transition of control is safe and clear
• Types of Handoffs
• 1. Human → AI
• Example: Activating autopilot in a vehicle
• Issue: Over-reliance on AI
• 2. AI → Human
• Example: AI requests human intervention
• Issue: Human may not be attentive or ready
• 3. Shared Control
• Both human and AI operate together
• Issue: Confusion about authority and responsibility
• Ethical Issues in Handoff
• 1. Responsibility Gap
– Unclear who is responsible for failure
• 2. Automation Bias
– Humans blindly trust AI decisions
• 3. Lack of Awareness
– Users may not know when control changes
• 4. Timing Problems
– Delayed or premature handoffs can cause harm
• Design Principles for Ethical Handoffs
• To ensure ethical AI systems, handoffs must be
designed carefully:
• Clarity: Clearly indicate who is in control
• Transparency: Explain why the handoff occurs
• Smooth Transition: Provide warnings before
switching
• Human Readiness: Ensure users can take
control
• Accountability: Clearly assign responsibility
• Why is Handoff Important?
• “In real-world systems, problems happen because:
• People don’t clearly understand their responsibility
• Information is not properly shared
• One group blames another

• The handoff model helps us:

• Understand responsibility clearly
• Avoid confusion
• Design better and safer systems”
• The Handoff Model helps us analyze ethical
responsibilities in technological systems.
• It helps answer questions like:
• Who is responsible for a system’s design?
• Who is responsible when something goes wrong?
• Instead, responsibility moves between different actors at
different stages.
• Example stages:
– Design stage
– Development stage
– Deployment stage
– Usage stage
– Maintenance stage
• At each stage, responsibility is handed off.
• Stages of Handoff
• “Let us look at how responsibility moves in
different stages.”
• 1. Design Stage
• “In this stage, designers decide what the
system should do.
• They must think about:
– Is the system fair?
– Will it cause harm?
• If design is wrong, the whole system can fail.”
• 2. Development Stage
• “Here, developers write code and build the
system.
• They must:
– Write correct code
– Avoid bugs and errors
– If they make mistakes, the system may not work
properly.”
• 3. Deployment Stage
• “In this stage, companies release the system.
• They must:
– Test the system
– Make sure it is safe
• If they release it too early, it can cause harm.”
• 4. Usage Stage
• “Now users use the system.
• Users must:
– Use it properly
– Not misuse it
• But users should also understand that the
system is not always perfect.”
• “After release, the system needs updates.
• Engineers must:
– Fix errors
– Improve the system
– Ignoring problems can make things worse.”
• Ethical Problems at Handoff Points
• “Most ethical issues happen at the handoff
points.
• Some common problems:
– No one takes responsibility
– Poor communication
– Users trust systems too much
– Loss of human control
• These problems happen when responsibility is
not clearly passed.”
• Simple Example
• “Let’s take an example of an AI system:
• Data is collected
• Model is built
• Company deploys it
• Users depend on it
• If the system gives wrong results, we cannot blame
just one person.
• The problem may have happened at any stage.”
• Summary :
• Handoff means transfer of responsibility
• Responsibility changes at each stage
• Ethical problems occur when handoffs are not
managed properly
• So as future engineers, always remember:
Good systems are built not just with good
code, but with clear responsibility at every
stage.”