Scribd
Upload
11 views2 pages

LLMs Enhancing AAC Communication

This project aims to enhance Augmentative and Alternative Communication (AAC) systems by integrating Large Language Models (LLMs) to create personalized dialogues for individuals with speech impairments. The system will utilize the Empathetic Dialogues dataset and Retrieval-Augmented Generation (RAG) to ground responses in personal narratives, improving expressiveness and engagement. Key tasks include fine-tuning dialogue models, integrating RAG, and developing a prototype for user interaction, with evaluations focusing on automatic metrics and human assessments of relevance, sincerity, and personalization.

Uploaded by

amruthavallir01
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
11 views2 pages

LLMs Enhancing AAC Communication

This project aims to enhance Augmentative and Alternative Communication (AAC) systems by integrating Large Language Models (LLMs) to create personalized dialogues for individuals with speech impairments. The system will utilize the Empathetic Dialogues dataset and Retrieval-Augmented Generation (RAG) to ground responses in personal narratives, improving expressiveness and engagement. Key tasks include fine-tuning dialogue models, integrating RAG, and developing a prototype for user interaction, with evaluations focusing on automatic metrics and human assessments of relevance, sincerity, and personalization.

Uploaded by

amruthavallir01
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Project 2 - LLMs for Societal Good - AAC use case

Background: This project explores the integration of Large Language Models (LLMs) with
Augmentative and Alternative Communication (AAC) systems to enhance communication for individuals
with speech impairments. Current AAC tools often produce generic responses that fail to reflect the
unique personality, experiences, and conversational intent of users. This project leverages LLMs to
generate personalized dialogues that are contextually relevant and represent an AAC user's
communication style. By incorporating Retrieval-Augmented Generation (RAG), the system can ground
responses in personal narratives, ensuring that AI-generated text resonates with an individual’s lived
experiences. The goal is to improve expressiveness, user engagement, and efficiency in AAC-mediated
conversations.

Your system should be able to:


●​ Classify emotion if required.
●​ Respond with empathy.
●​ Share opinions whenever needed.
●​ Have Feedback Mechanism
●​ Share facts about personal experiences if required.

Dataset: You are free to use any relevant dataset that you want. Here are some recommendations, and
one of them is one that you have to use.
●​ [Required] The Empathetic Dialogues (ED) dataset comprises 25k conversations grounded in
emotional situations. ([Link]
●​ GoEmotions: [Link]
●​ DailyDialog: [Link]
●​ [Link]
●​ Additionally, participants may scrape publicly available sources, such as social media forums like
Reddit, for autobiographical journal entries or personal narratives to use in Retrieval-Augmented
Generation (RAG). These resources can provide real-world experiences that help personalize the
conversational system. You can also prompt and synthetically generate such content.

Tasks: Participants will implement a system that enhances AAC-mediated conversations by


incorporating personalization techniques. The core tasks include:

1.​ Fine-Tuning a Dialogue Model


○​ Use the Empathetic Dialogues dataset and one additional dataset to fine-tune a
conversational LLM.
○​ Ensure responses capture nuanced emotions and remain contextually relevant.
2.​ Integrating Retrieval-Augmented Generation (RAG)
○​ Implement RAG by retrieving personal narratives or situational context from an external
database (e.g., a corpus of autobiographical texts).
○​ Optimize retrieval strategies to enhance factual grounding and reduce hallucinations.
3.​ Improving Conversational Style Adaptation
○​ Develop prompting strategies that adapt the model’s responses to an AAC user's
communication preferences.
○​ Experiment with response refinement techniques, such as injecting personality traits and
maintaining user-specific language patterns.
4.​ Developing a System Prototype
○​ Build a simple web-based demo where users can interact with the model and evaluate its
personalized responses.

Evaluation

To assess the effectiveness of the personalized conversational AI, participants will conduct both automatic
and human evaluations:

1.​ Automatic Metrics


○​ BLEU, ROUGE, METEOR: Measure fluency and coherence.
○​ BERTScore: Evaluates the semantic similarity between generated and reference
responses.
2.​ Human Evaluation - You can also use LLM-based evaluation
○​ Relevance: Does the response align with the context and conversation flow?
○​ Sincerity: Does the response feel authentic rather than generic?
○​ Understandability: Is the generated text clear and easy to comprehend?
○​ Fluency: Does the response read naturally, without awkward phrasing?
○​ Personalization: Does the response reflect a unique user’s personality and
communication needs?

References
●​ [Link]
●​ [Link]
●​ Huggingface Transformers: [Link]
●​ TransferTransfo: [Link]

Common questions

Powered by AI

To ensure LLM-generated responses are perceived as sincere and authentic, strategies such as embedding personal narratives, fine-tuning on emotion-rich datasets, and using prompting techniques that reflect the user's personality can be employed . Additionally, integrating retrieval systems that provide factual and contextually relevant information can help the model generate responses that resonate genuinely with the user's experiences .

The integration of LLMs with AAC systems enhances communication for individuals with speech impairments by generating personalized dialogues that reflect the user's unique communication style, personality, and experiences . By using Retrieval-Augmented Generation (RAG), the system can ground responses in personal narratives, making conversations more contextually relevant and expressive . This approach improves user engagement, expressiveness, and efficiency in AAC-mediated conversations .

Retrieval-Augmented Generation (RAG) reduces hallucinations in conversational AI systems by retrieving relevant personal narratives or situational contexts from an external database, ensuring responses are factually grounded . By optimizing retrieval strategies, the system can provide accurate information that resonates with the individual's lived experiences, minimizing the generation of irrelevant or inaccurate content .

The proposed metrics for evaluating the effectiveness of personalized conversational AI systems include automatic metrics like BLEU, ROUGE, METEOR, and BERTScore, and human evaluations focusing on relevance, sincerity, understandability, fluency, and personalization . These metrics assess aspects such as the fluency and coherence of the response, semantic similarity with reference responses, and how well the AI captures the unique communication needs and personality of the user .

Implementing a feedback mechanism contributes to the improvement of AAC-mediated conversations by allowing users to provide real-time input on the quality and relevance of responses . This continuous feedback loop helps in refining the dialogue model, ensuring responses are better tailored to individual preferences and styles over time, thereby enhancing personalization and overall user satisfaction .

Incorporating personal experiences in dialogue generation is crucial for AAC systems because it ensures that each conversation reflects the user's personality and lived experiences, increasing engagement and expressiveness . This can be achieved using Retrieval-Augmented Generation (RAG), which pulls personal narratives or contextual information from a database to create responses that are contextually rich and emotionally resonant .

The primary datasets recommended for fine-tuning a conversational LLM for AAC systems include the Empathetic Dialogues dataset and potentially others like GoEmotions or DailyDialog . The Empathetic Dialogues dataset comprises 25k conversations grounded in emotional situations, which helps the model capture nuanced emotions and maintain contextual relevance in communication .

Adapting conversational styles to meet diverse AAC users' needs poses challenges such as accurately capturing varied communication preferences, personality traits, and linguistic patterns . Each user may have unique requirements, necessitating the development of flexible prompting strategies, personalized training data, and refined response generation methods that respect individual differences while maintaining natural dialogue flow .

Emotion classification plays a critical role in enhancing AAC conversations by ensuring that responses align with the emotional tone and intent of the user, making interactions more empathetic and contextually appropriate . In LLM systems, this can be implemented by training models on emotion-rich datasets, like the Empathetic Dialogues, and integrating emotion detection capabilities to classify and respond to emotions accurately .

Key considerations in developing prompting strategies for adapting LLM responses to an AAC user's preferences include capturing the user's unique communication style, introducing personality traits, and maintaining user-specific language patterns . This involves experimenting with response refinement techniques to ensure that the model's output aligns with the user's conversational needs and expresses their individuality .

You might also like