Human-Centered AI (BAI613)
MODULE 4
GOVERNANCE STRUCTURES - 1
Topics: Introduction, Reliable Systems Based on Sound Software Engineering Practice, Safety
Culture through Business Management Strategies, Trustworthy Certification by Independent
Oversight.
Introduction
The rapid spread of misinformation, conspiracy theories, and misuse of AI systems has
significantly increased concern about ethical issues in artificial intelligence. While many
organizations have proposed ethical principles like fairness, transparency, and accountability, the
real challenge lies in translating these principles into practical implementation. Ethics alone is not
sufficient; it must be supported by governance structures that ensure responsible design,
deployment, and monitoring of AI systems.
Human-Centered AI (HCAI) represents a shift from traditional AI approaches that focused
primarily on machine autonomy and algorithmic performance. Instead, HCAI emphasizes human
control, usability, trust, and societal benefit. The goal is to build systems that enhance human
capabilities rather than replace them. These systems are applied across domains such as healthcare,
education, cybersecurity, and environmental protection, where they can deliver immense benefits
but also pose risks like bias, privacy violations, and misuse.
HARSH VERMA 1
Human-Centered AI (BAI613)
A key insight is that AI systems today are complex, interconnected, and continuously evolving.
They rely on multiple components such as datasets, algorithms, hardware, and external services.
This complexity makes it difficult to identify failure points and demands stronger governance
mechanisms. Moreover, ethical principles proposed by organizations (like IEEE or Berkman Klein
Center) often remain aspirational, creating a gap between theory and real-world practice.
To bridge this gap, the book proposes a four-layer governance framework:
1. Team level – Software engineering practices
2. Organizational level – Safety culture
3. Industry level – Independent oversight
4. Government level – Regulation
These layers work together like nested systems, ensuring accountability at every stage. The
transition from ethics to practice requires clear responsibility, structured processes, continuous
monitoring, and external accountability.
However, challenges remain. Corporate interests may conflict with societal good, and
policymakers often lack technical understanding of AI systems. Therefore, successful
HARSH VERMA 2
Human-Centered AI (BAI613)
implementation requires collaboration between engineers, managers, regulators, and society.
Ultimately, HCAI aims to create systems that support human values such as creativity,
responsibility, and social connection.
Reliable Systems – Software Engineering Practices
Reliable HCAI systems are built through strong software engineering practices that ensure
correctness, fairness, and transparency. Unlike traditional software, AI systems depend heavily on
data and machine learning models, which introduce new challenges in design, testing, and
deployment.
One of the most important practices is the use of audit trails (logs). These act like “black boxes”
in airplanes, recording system behaviour, decisions, and data used at every stage. Audit trails
enable retrospective analysis of failures, help assign responsibility, and support continuous
improvement. However, they raise issues such as data ownership, privacy, and storage complexity.
Modern HCAI systems also require updated software engineering workflows. Traditional waterfall
models are being replaced by agile approaches, where development is iterative, user-focused, and
adaptive to changing requirements. AI workflows include stages such as data collection, cleaning,
labelling, model training, evaluation, deployment, and continuous monitoring. Importantly, data
quality becomes as important as code quality.
HARSH VERMA 3
Human-Centered AI (BAI613)
Another critical aspect is verification and validation testing. AI systems must be tested not only
for correctness but also for robustness across different datasets and real-world conditions.
Techniques include case-based testing, differential testing, metamorphic testing, user testing, and
red-team attacks. These methods help identify vulnerabilities, including adversarial attacks and
unexpected failures.
Bias testing is essential to ensure fairness. AI systems can inherit biases from data or design
decisions, leading to discrimination in areas like hiring, healthcare, or criminal justice. Bias can
HARSH VERMA 4
Human-Centered AI (BAI613)
be pre-existing, technical, or emergent. Addressing bias requires careful dataset design, fairness
metrics, continuous monitoring, and sometimes dedicated roles such as a bias testing leader.
Explainability is another key requirement. Users must understand why decisions are made,
especially in high-stakes applications. Two approaches exist:
• Post-hoc explanations (after decision)
• Prospective interfaces (before decision, interactive exploration)
The latter is often more effective, as it allows users to explore outcomes and understand system
behavior proactively.
HARSH VERMA 5
Human-Centered AI (BAI613)
Overall, reliable systems are achieved through integration of engineering rigor, data management,
fairness evaluation, and user-centered design.
Safety Culture – Organizational Strategies
While technical practices ensure reliability at the team level, organizations must create a safety
culture that prioritizes long-term trust over short-term gains. Failures in large systems often arise
not from individual errors but from organizational weaknesses.
Four major approaches guide safety culture:
1. Normal Accident Theory – Failures are inevitable in complex systems
2. High Reliability Organizations – Continuous focus on preventing failure
3. Resilience Engineering – Ability to recover from disruptions
4. Safety Culture – Organizational mindset prioritizing safety
HARSH VERMA 6
Human-Centered AI (BAI613)
A strong safety culture begins with leadership commitment. Leaders must clearly communicate
safety as a core value, allocate resources, and ensure accountability. This includes investing in
training, hiring diverse expertise, and maintaining redundancy to handle emergencies.
Organizations must also implement extensive reporting systems for failures and near misses. Near
misses are especially valuable because they occur more frequently and provide insights for
preventing major failures. Open reporting systems, similar to aviation or healthcare, encourage
transparency and learning.
Internal review boards play a crucial role in analyzing incidents and guiding improvements. These
boards evaluate performance, review risks, and ensure alignment with long-term goals. Practices
such as disclosure and apology in healthcare demonstrate how transparency can build trust and
reduce conflicts.
Alignment with industry standards is another pillar. Standards from organizations like ISO, IEEE,
and others provide guidelines for safety, fairness, and usability. Additionally, models like the
Capability Maturity Model (CMM) help organizations evolve from chaotic processes to optimized,
measurable systems.
Ultimately, safety culture transforms organizations from reactive entities to proactive systems that
anticipate risks and continuously improve.
HARSH VERMA 7
Human-Centered AI (BAI613)
Trustworthy Certification – Independent Oversight
Independent oversight introduces external accountability, ensuring that organizations do not
operate in isolation. This layer is crucial for building public trust in HCAI systems.
Oversight focuses on responsibility and liability, ensuring that developers and organizations are
accountable for their systems. Unlike traditional engineering fields, software development has
historically lacked strict certification and accountability mechanisms, making this an important
area for reform.
There are three main types of oversight:
1. Planning oversight – Reviewing systems before deployment
2. Continuous monitoring – Ongoing supervision
3. Retrospective analysis – Investigating failures
Several institutions contribute to oversight:
• Auditing firms conduct external reviews and ensure transparency
• Insurance companies enforce safety through risk assessment and compensation models
• NGOs and civil society raise awareness, highlight issues, and influence policy
• Professional organizations develop standards and ethical guidelines
Independent oversight ensures that systems are not only technically sound but also socially
responsible and aligned with public interest.
Government Interventions and Regulations
HARSH VERMA 8
Human-Centered AI (BAI613)
Government regulation forms the outermost layer of governance. While often seen as restrictive,
well-designed regulation can actually promote innovation and protect society.
Governments play several roles:
• Ensuring fair competition and consumer protection
• Enforcing safety and accountability standards
• Protecting privacy and human rights
Examples like GDPR have driven advancements in explainable AI and data protection. Similarly,
regulations in industries like transportation and healthcare have historically improved safety and
innovation.
However, regulation is challenging due to the rapid pace of AI development. Policymakers must
balance innovation with risk control, requiring collaboration with experts, industry, and civil
society.
Summary and Skeptic’s Corner
The final chapter reflects on the proposed governance framework and addresses skepticism. Critics
argue that:
• Regulations may slow innovation
• Oversight mechanisms may be ineffective
• Organizations may prioritize profit over ethics
Despite these concerns, the framework offers a practical path forward. It emphasizes that trust in
AI systems cannot be achieved through principles alone but requires structured governance,
accountability, and continuous improvement.
The future of HCAI depends on integrating human-centered thinking into every stage of
development. While challenges remain, the approach provides a hopeful vision of AI systems that
enhance human life, promote fairness, and support societal well-being.
HARSH VERMA 9