CHAPTER ONE

INTRODUCTION AND PROBLEM FORMULATION

1.1 Introduction

Digital platforms face an escalating scale–complexity crisis in content governance.

Contemporary social media systems process billions of user-generated text items daily,

including posts, comments, and messages, making manual moderation operationally

infeasible (Gillespie, 2018; Roberts, 2019). In response, platforms increasingly rely on

artificial intelligence (AI) and machine learning models to automate content moderation tasks

such as detecting hate speech, harassment, and abusive language (Schmidt & Wiegand, 2017;

Vidgen et al., 2020).

While AI-based moderation systems provide significant gains in speed and scalability,

empirical studies consistently show that automation alone is insufficient for resolving

context-dependent and ethically sensitive moderation decisions. Automated classifiers often

struggle with sarcasm, implicit meaning, reclaimed slurs, and culturally specific language,

leading to high rates of false positives and false negatives (Sap et al., 2022; Zhao et al.,

2023). These limitations raise concerns about fairness, freedom of expression, and

algorithmic accountability, particularly when moderation decisions directly affect users’

visibility, participation, and speech rights (Raji et al., 2020; Floridi et al., 2018).

Recent research therefore emphasizes the need for hybrid moderation systems that integrate

human judgment into automated workflows. Human-in-the-Loop (HITL) systems combine

the computational efficiency of AI with human contextual reasoning and ethical oversight,
enabling more reliable moderation outcomes (Holzinger et al., 2021; Nguyen et al., 2023).

Rather than replacing human moderators, HITL frameworks reposition humans as decision

authorities in low-confidence or ambiguous cases, while AI handles high-confidence routine

classifications. This study adopts this hybrid perspective to address structural weaknesses in

fully automated moderation systems.

1.2 Statement of the Problem

Despite advances in natural language processing and deep learning, fully automated content

moderation systems remain unreliable when handling low-confidence and context-

sensitive inputs. Existing systems typically rely on static decision thresholds and dataset-

driven pattern recognition, which limits their ability to interpret nuanced language or adapt to

evolving online discourse (Vidgen et al., 2020; Zhong et al., 2024).

The fundamental problem addressed in this study is the absence of a reliable mechanism

for resolving low-confidence moderation decisions under constraints of fairness,

transparency, and scalability. While AI models can efficiently classify large volumes of

content, they lack moral reasoning, situational awareness, and sociocultural understanding

required for ethically sound moderation decisions (Floridi et al., 2018; Sap et al., 2022).

Consequently, automated systems may disproportionately flag minority dialects, suppress

legitimate speech, or fail to detect subtle forms of harm (Raji & Buolamwini, 2019).

Furthermore, many existing moderation systems do not incorporate structured feedback

mechanisms that allow human decisions to systematically improve AI performance over time.

Without such feedback loops, models remain static and increasingly misaligned with

emerging language patterns and moderation norms (Nguyen et al., 2023). This gap highlights

a systemic design failure rather than a purely algorithmic limitation.

Formally, the problem can be expressed as follows: Current moderation systems lack a

principled method for routing content based on model uncertainty, resulting in inefficient

allocation of human oversight and ethically fragile decision-making processes.

1.3 Aim of the Study

The primary aim of this study is to design and implement a Relative AI Content Moderation

System with a Human-in-the-Loop dashboard that dynamically allocates moderation

authority between automated classifiers and human moderators based on confidence levels

and contextual complexity.

1.4 Objectives of the Study

This study is guided by the following objectives:

1. To design an AI-based text classification model capable of identifying

harmful or inappropriate content with associated confidence scores

(Schmidt & Wiegand, 2017; Jurafsky & Martin, 2023).

2. To implement a confidence-based Human-in-the-Loop

workflow that escalates low-confidence moderation decisions to

human reviewers (Holzinger et al., 2021; Nguyen et al., 2023).

3. To develop an interactive moderation dashboard that supports

transparency, auditability, and efficient human decision-making (Lai

et al., 2022).

4. To evaluate the proposed system against a fully automated baseline

using metrics such as accuracy, precision, recall, false-positive

 reduction, and moderation latency (Powers, 2020; Zhong et al.,

2024).

1.5 Significance of the Study

This study is significant within the fields of Computer Science, Artificial Intelligence, and

Human–Computer Interaction. Technically, it contributes a system-level framework for

integrating AI classification with structured human oversight, addressing known weaknesses

of automated moderation systems (Raji et al., 2020; Holzinger et al., 2021).

Academically, the study contributes to ongoing research on human-centered AI by

demonstrating how ethical principles such as fairness, accountability, and transparency can

be operationalized through system design rather than treated as abstract ideals (Floridi et al.,

2018; Amershi et al., 2019).

Practically, improved moderation accuracy and transparency can enhance user trust and

platform integrity while reducing unnecessary censorship and exposure to harmful content

(Jhaver et al., 2019; Roberts, 2019). For undergraduate research at AE-FUNAI, the study

demonstrates the practical application of AI, system architecture, and ethical computing

principles within a controlled academic environment.

1.6 Scope of the Study

This study is limited to the design and implementation of a prototype text-based content

moderation system. The system focuses on detecting harmful textual content such as hate

speech and harassment using supervised machine learning techniques (Vidgen et al., 2020).

Multimedia content moderation, large-scale deployment, and cross-platform enforcement are

beyond the scope of this research.

The Human-in-the-Loop mechanism is implemented through a web-based dashboard that

enables human moderators to review low-confidence cases and provide feedback for iterative

model improvement (Holzinger et al., 2021; Lai et al., 2022). The system is evaluated in a

controlled academic setting suitable for undergraduate-level system development.

 CHAPTER TWO

LITERATURE REVIEW

2.1 Conceptual Review of Content Moderation Systems

Content moderation refers to the processes through which digital

platforms regulate user-generated content to enforce community

standards and legal requirements (Gillespie, 2018). Early moderation

systems relied almost exclusively on manual human review, where

moderators assessed content based on predefined platform rules. While

human moderation offers contextual sensitivity and ethical reasoning, it is

widely recognized as unsustainable at scale due to high operational costs,

slow response times, and significant psychological risks to moderators

(Roberts, 2019).

The exponential growth of online content has led platforms to adopt

automated moderation systems, primarily driven by advances in artificial

intelligence and machine learning. Automated systems aim to classify

content as acceptable or violative using computational models trained on

labeled datasets (Schmidt & Wiegand, 2017). These systems provide

speed and scalability but are inherently constrained by the quality of

training data and the limitations of algorithmic interpretation of human

language.

Recent scholarship emphasizes that content moderation is not solely a

technical problem but a socio-technical challenge, where computational

decisions intersect with ethics, culture, and governance (Gillespie, 2018;

Floridi et al., 2018). As a result, modern research increasingly explores

hybrid approaches that combine automation with human judgment.

2.2 Artificial Intelligence in Text-Based Content Moderation

AI-based content moderation systems typically employ natural language

processing (NLP) and supervised machine learning techniques to identify

harmful textual content such as hate speech, harassment, and abusive

language. Early approaches relied on keyword filtering and rule-based

systems, which proved ineffective due to their inability to capture context

and evolving language patterns (Schmidt & Wiegand, 2017).

More recent systems utilize machine learning classifiers and deep learning

architectures, including convolutional and transformer-based models,

which have demonstrated improved performance in detecting nuanced

linguistic patterns (Goodfellow et al., 2016; Jurafsky & Martin, 2023).

Large-scale datasets such as hate speech corpora have enabled models to

achieve high accuracy under controlled conditions (Vidgen et al., 2020).

However, multiple studies reveal that high accuracy does not equate to

reliable moderation in real-world settings. AI classifiers often misinterpret

sarcasm, coded language, and reclaimed slurs, leading to biased or

inconsistent moderation outcomes (Sap et al., 2022; Zhao et al., 2023).

Furthermore, algorithmic systems may disproportionately misclassify

content associated with minority groups, raising concerns about fairness

and discrimination (Raji & Buolamwini, 2019; Raji et al., 2020). These

findings indicate that while AI is effective for large-scale filtering, it lacks

the interpretive depth required for ethically sensitive moderation

decisions.

2.3 Limitations of Fully Automated Moderation Systems

Despite technological advances, fully automated moderation systems face

persistent structural limitations. One major issue is model uncertainty.

Most AI moderation systems produce probabilistic outputs, yet many

platforms apply rigid decision thresholds without considering uncertainty

levels (Zhong et al., 2024). This approach results in the automation of

borderline cases that would benefit from human interpretation.

Another limitation is the static nature of trained models. Language on

digital platforms evolves rapidly, with new slang, memes, and coded

expressions emerging continuously. Without structured human feedback

mechanisms, automated systems struggle to adapt, leading to

performance degradation over time (Nguyen et al., 2023).

Ethical accountability also remains a challenge. Automated decisions are

often opaque, making it difficult for users to understand or contest

moderation outcomes. This lack of transparency undermines trust and

raises regulatory concerns, particularly in regions with emerging AI

governance frameworks (Floridi et al., 2018; Amershi et al., 2019). These

limitations have motivated the exploration of system designs that

incorporate human oversight as an integral component rather than an

afterthought.

2.4 Human-in-the-Loop (HITL) Systems

Human-in-the-Loop (HITL) systems are designed to integrate human

judgment into automated decision-making processes, particularly in cases

characterized by uncertainty, ambiguity, or high risk (Holzinger et al.,

2021). In content moderation, HITL systems typically allow AI to handle

high-confidence cases while routing low-confidence decisions to human

moderators.

Research demonstrates that HITL systems significantly improve decision

accuracy and reduce harmful errors compared to fully automated systems

(Lai et al., 2022; Nguyen et al., 2023). By leveraging human contextual

reasoning, HITL frameworks address ethical concerns such as fairness,

accountability, and explainability.

However, effective HITL implementation requires careful system design.

Poorly designed HITL systems may overload human moderators or fail to

utilize human input meaningfully. Studies emphasize the importance of

clear confidence thresholds, intuitive interfaces, and structured feedback

loops to ensure that human involvement enhances rather than hinders

system performance (Amershi et al., 2019; Holzinger et al., 2021).

2.5 Confidence-Based Decision Routing

Confidence-based decision routing is a core principle in modern HITL

systems. It involves using model confidence scores to determine whether

a decision should be automated or escalated to human review. Research

on “learning to defer” demonstrates that selectively deferring uncertain

cases to humans improves overall system accuracy and reliability (Madras

et al., 2018; Lykouris & Weng, 2024).

In content moderation, confidence thresholds enable platforms to allocate

human resources more efficiently by focusing attention on ambiguous

cases rather than routine decisions. Studies show that confidence-aware

systems reduce false positives and improve user trust when compared to

fixed-threshold automation (Raji et al., 2020; Zhong et al., 2024). This

study adopts a confidence-based routing mechanism as a foundational

design principle, aligning with contemporary best practices in hybrid AI

systems.

2.6 Moderation Dashboards and Human–Computer Interaction

The effectiveness of HITL systems depends not only on algorithmic

performance but also on interface design. Moderation dashboards serve

as the primary interaction point between human moderators and

automated systems. Poorly designed interfaces can increase cognitive

load and decision fatigue, reducing moderation quality (Jhaver et al.,

2019).

Recent HCI research emphasizes the need for dashboards that provide

clear explanations, audit trails, and decision support tools to assist

moderators in making consistent and informed judgments (Lai et al.,

2022). Transparent interfaces also enhance accountability by enabling

traceability of moderation decisions.

By incorporating an interactive dashboard, this study aligns with human-

centered AI design principles that prioritize usability, transparency, and

ethical oversight.
2.7 Research Gap

Existing literature establishes the effectiveness of AI, HITL systems, and

confidence-based routing independently. However, there remains a gap in

integrated system implementations that combine confidence-aware AI

moderation, structured human feedback, and transparent dashboards

within a single deployable framework, particularly in undergraduate-level

research contexts.

Most studies focus either on algorithmic performance or ethical theory,

with limited emphasis on end-to-end system design and real-time human–

AI collaboration. This study addresses this gap by designing and

implementing a Relative AI Content Moderation System that

operationalizes human-in-the-loop principles through confidence-based

escalation and an interactive moderation interface.

2.8 Summary of Literature Review

This chapter reviewed existing literature on content moderation, AI-based

classification, human-in-the-loop systems, confidence-based decision

routing, and moderation interfaces. While automated moderation offers

scalability, its limitations necessitate hybrid approaches that integrate

human judgment. The reviewed studies collectively support the feasibility

and relevance of a confidence-driven HITL moderation system, providing a

strong theoretical foundation for the methodology presented in the next

chapter.
 CHAPTER THREE

METHODOLOGY AND SYSTEM DESIGN

3.1 Research Design

This study adopts a Design Science Research (DSR) methodology. Design Science is

appropriate for Computer Science research that seeks to build, evaluate, and refine an

artefact intended to solve a real-world problem (Hevner et al., 2004; Peffers et al., 2007). In
this study, the artefact is a Relative AI Content Moderation System with a Human-in-the-

Loop (HITL) dashboard.

The choice of Design Science is justified because the research does not merely analyze

existing moderation systems but designs and implements a functional prototype, evaluates

its performance, and demonstrates its relevance to content moderation challenges. The

methodology follows the core DSR stages: problem identification, system design,

implementation, evaluation, and communication of results (Hevner et al., 2004).

3.2 System Overview

The proposed system is a hybrid content moderation framework that combines automated

AI-based text classification with structured human oversight. Content submitted to the system

is first processed by an AI classifier, which assigns both a moderation label and a confidence

score. Based on predefined confidence thresholds, the system either:

 Automatically moderates the content (high-confidence cases), or

 Escalates the content to a human moderator via a dashboard (low-confidence cases).

Human moderation decisions are logged and stored, enabling transparency, auditability, and

potential feedback for system improvement. This architecture ensures scalability while

preserving ethical oversight in ambiguous cases (Holzinger et al., 2021; Nguyen et al., 2023).

3.3 System Architecture

The system architecture follows a modular client–server design, consisting of the following

components:

1. User Content Interface: Receives text-based user-generated content.

 2. AI Classification Module: Performs automated moderation and computes confidence

scores.

3. Decision Routing Engine: Determines whether content is auto-moderated or

escalated based on confidence thresholds.

4. Human Moderation Dashboard: Allows human reviewers to inspect, classify, and

justify decisions.

5. Database and Audit Log: Stores content, decisions, confidence scores, and

moderation history.

6. Deployment and Workflow Layer: Manages version control, deployment, and

automation processes.

This layered architecture improves maintainability, scalability, and transparency by

separating decision logic from interface and infrastructure concerns (Amershi et al., 2019).

3.4 AI Classification Module

The AI module performs text-based content classification using supervised machine

learning techniques. Input text is preprocessed through tokenization, normalization, and

vectorization before being passed to the classifier. The model outputs both a predicted class

(e.g., acceptable or harmful) and an associated probability score representing prediction

confidence (Jurafsky & Martin, 2023).

Confidence scores play a central role in the system. Rather than enforcing fixed binary

decisions, the system interprets confidence as a measure of uncertainty. Predictions below a

predefined confidence threshold are considered unreliable and are automatically routed for

human review (Madras et al., 2018; Lykouris & Weng, 2024).

This approach aligns with contemporary research advocating uncertainty-aware AI systems

for high-risk decision environments such as content moderation (Raji et al., 2020).

3.5 Human-in-the-Loop (HITL) Workflow

The HITL workflow is designed to ensure that human involvement is purposeful and

efficient. Only content that the AI classifier identifies as ambiguous or low-confidence is

escalated to human moderators. Moderators review the content via a web-based dashboard,

where they can:

 View the original content

 Inspect AI confidence scores

 Assign a final moderation decision

 Provide optional justification notes

Human decisions override automated predictions and are permanently logged. This structure

reduces moderator overload while ensuring that ethically sensitive cases receive appropriate

human judgment (Holzinger et al., 2021; Lai et al., 2022).

3.6 Moderation Dashboard Design

The moderation dashboard serves as the primary human–computer interaction layer. Its

design prioritizes usability, clarity, and transparency. Key features include:

 Real-time moderation queues

 Clear labeling of confidence levels

 Decision history and audit trails

  Basic analytics on moderation outcomes

The dashboard is intentionally designed to minimize cognitive load by presenting only

essential information needed for decision-making. This aligns with human-centered AI

principles, which emphasize that interfaces should support, rather than burden, human

judgment (Amershi et al., 2019; Jhaver et al., 2019).

3.7 Tools, Technologies, and Deployment Pipeline

To ensure reproducibility and real-world relevance, the system adopts a modern software

engineering toolchain, as guided by the supervisor-provided document.

 GitHub: Used for version control, collaboration, and code traceability. Version

control ensures reproducibility and accountability throughout development.

 Render: Serves as the deployment platform, enabling continuous deployment and

operational stability.

 MongoDB: Used as the database due to its flexible schema, which supports evolving

moderation data and audit logs.

 n8n Workflow Automation: Handles automation of AI inference workflows,

escalation triggers, and logging processes.

This pipeline supports transparent deployment, persistent auditability, and system

scalability, aligning the project with industry-standard software engineering practices.

3.8 Data Collection and Evaluation Metrics

Evaluation is conducted by comparing the hybrid HITL system with a fully automated

baseline. Performance is assessed using standard classification metrics, including:

  Accuracy

 Precision

 Recall

 False-positive and false-negative rates

Additional system-level metrics include:

 Number of escalated cases

 Average moderation latency

 Reduction in automated misclassification

These metrics provide quantitative evidence of the effectiveness of confidence-based human

intervention (Powers, 2020; Zhong et al., 2024).

3.9 Ethical Considerations

Ethical considerations are central to the system design. The HITL framework promotes:

 Fairness: reducing biased automated decisions

 Accountability: through traceable moderation logs

 Transparency: via explainable confidence-based routing

By ensuring that humans retain authority over ambiguous decisions, the system aligns with

ethical AI principles and emerging governance frameworks (Floridi et al., 2018; Raji et al.,

2020).
 CHAPTER FOUR

SYSTEM IMPLEMENTATION AND EXPECTED RESULTS

4.1 System Implementation Overview

This chapter presents the implementation details of the proposed Relative AI Content

Moderation System with Human-in-the-Loop (HITL) and outlines the expected results

based on system design and evaluation metrics. The implementation follows the architectural

and methodological specifications described in Chapter Three, translating conceptual design

into a functional prototype.

The system was implemented as a web-based application, integrating automated AI text

classification, confidence-based decision routing, and a human moderation dashboard.

Emphasis was placed on modularity, transparency, and auditability to ensure that the system

remains extensible and ethically aligned.

4.2 Implementation of System Components

4.2.1 User Content Submission Interface

The user content interface serves as the system’s entry point, allowing text-based content to

be submitted for moderation. Submitted content is validated and forwarded to the AI

classification module for processing. This component ensures input consistency and prevents

malformed data from entering the moderation pipeline.

4.2.2 AI Text Classification Module

The AI classification module was implemented using supervised machine learning techniques

for text classification. Incoming text is preprocessed through normalization, tokenization, and

vectorization before classification. The model produces:

 A predicted moderation label (e.g., acceptable or harmful)

 A corresponding confidence score indicating prediction certainty

The confidence score is a critical output, as it directly informs the decision routing

mechanism. Rather than enforcing absolute decisions, the system treats classification

confidence as an uncertainty indicator, aligning with uncertainty-aware AI design principles

(Raji et al., 2020; Zhong et al., 2024).

4.2.3 Confidence-Based Decision Routing Engine

The decision routing engine evaluates confidence scores against predefined thresholds. If the

confidence exceeds the threshold, the system automatically applies moderation rules.

Conversely, content with low or borderline confidence is escalated to the human moderation

dashboard.

This mechanism ensures that human reviewers focus only on ambiguous or high-risk cases,

reducing cognitive load while maintaining ethical oversight. The routing logic operationalizes

the Human-in-the-Loop framework described earlier.

4.2.4 Human Moderation Dashboard

The moderation dashboard provides a structured interface for human reviewers. Moderators

can:

 View flagged content

 Inspect AI confidence scores

 Assign final moderation decisions

 Provide optional justification notes

Human decisions override automated outcomes and are logged for accountability. The

dashboard emphasizes clarity and minimalism to support efficient human decision-making

and reduce fatigue, in line with human-centered AI design guidelines (Amershi et al., 2019).

4.2.5 Database and Audit Logging

All system activities, including content submissions, AI predictions, confidence scores,

routing outcomes, and human decisions, are stored in a centralized database. Persistent audit
logs ensure traceability and transparency, supporting ethical accountability and potential

system review.

The use of a flexible database schema allows the system to accommodate evolving

moderation categories and decision metadata.

4.2.6 Deployment and Automation Pipeline

The system adopts a modern deployment pipeline using:

 GitHub for version control and collaboration

 Render for deployment and hosting

 MongoDB for data persistence

 n8n for workflow automation and process orchestration

This pipeline supports reproducibility, operational stability, and automated handling of

moderation workflows, aligning the project with real-world software engineering practices.

4.3 System Integration and Workflow Execution

Upon submission, user content flows through the AI classifier, confidence evaluation, and

decision routing components. Depending on confidence thresholds, content is either

automatically moderated or escalated to human review. Final decisions are logged and stored,

completing the moderation cycle.

Figures 3.1 and 3.2 (presented in Chapter Three) visually illustrate this interaction and

workflow sequence, reinforcing the coherence between system design and implementation.
4.4 Evaluation Criteria

Since the system is implemented as a prototype, evaluation focuses on expected

performance outcomes rather than large-scale deployment metrics. The system is designed

to be evaluated using the following criteria:

 Classification accuracy

 Precision and recall

 Reduction in false positives and false negatives

 Percentage of content escalated to human review

 Average moderation latency

 Transparency and auditability of decisions

These metrics align with standard evaluation practices for content moderation systems

(Powers, 2020).

4.5 Expected Results

Based on system design and existing literature, the following outcomes are expected:

1. Improved Moderation Accuracy

The integration of human oversight for low-confidence cases is expected to reduce

misclassification errors compared to fully automated moderation systems.

2. Reduction in False Positives

By routing ambiguous cases to human reviewers, the system is expected to reduce

wrongful content removal, thereby protecting legitimate user expression.

 3. Efficient Allocation of Human Resources

Confidence-based routing is expected to limit human review to a manageable subset

of cases, improving moderation efficiency and reducing reviewer fatigue.

4. Enhanced Transparency and Accountability

The inclusion of audit logs and decision histories is expected to improve system

transparency, enabling traceability of both automated and human decisions.

5. Ethically Robust Moderation Decisions

Human involvement in ambiguous cases is expected to strengthen ethical compliance

by incorporating contextual judgment beyond algorithmic capability.

4.6 Discussion of Expected Outcomes

The expected outcomes align with findings from prior studies on human-in-the-loop and

confidence-aware AI systems (Holzinger et al., 2021; Lykouris & Weng, 2024). By

combining AI scalability with human contextual reasoning, the proposed system is positioned

to achieve a balance between efficiency and ethical reliability.

While the prototype is not evaluated under real-world production conditions, its design

demonstrates feasibility and practical relevance within an academic setting. The system

architecture allows future extension to larger datasets, additional content types, and adaptive

learning mechanisms.