Volume 11, Issue 2, February – 2026 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165 [Link]

Multimodal Command System for Human-

Computer Interaction
Harikumar M.1; Prasanth D.2; Reuben Abraham George3

1,2,3
Department of Computer Science and Engineering,
Rajalakshmi Engineering College, Thandalam, 602105.
Chennai, Tamilnadu, India.

Publication Date: 2026/03/07

Abstract: Recent advances in computing have increased the demand for interaction paradigms that enable intuitive and
efficient communication between users and digital systems. Traditional interfaces such as keyboards and mice often limit
accessibility and natural interaction, particularly in complex or hands-free environments. This paper proposes a multimodal
human–computer interaction system that integrates hand gesture recognition and voice command processing to enable
seamless desktop control and interaction with external applications. The system employs MediaPipe-based hand landmark
extraction and speech-to-text processing, coordinated through an AI agent using the Model Context Protocol (MCP) for task
orchestration and service integration. Experimental evaluation demonstrates that the proposed framework achieves high
recognition accuracy with low latency, supporting real-time interaction across local and cloud-based services. The results
indicate that multimodal fusion combined with agent-based automation enhances usability, responsiveness, and accessibility,
positioning the system as a scalable solution for next-generation human–computer interaction.

Keywords: Human–Computer Interaction, Hand Gesture Recognition, MediaPipe, Lightweight CNN, Multimodal Systems.

How to Cite: Harikumar M.; Prasanth D.; Reuben Abraham George (2026) Multimodal Command System for Human-Computer
Interaction. International Journal of Innovative Science and Research Technology, 11(2), 2902-2907.
[Link]

I. INTRODUCTION capture and action execution, enabling extensible integration

with external applications while promoting interpretability.
The evolution of human-computer interaction has As a result, multimodal HCI systems are positioned to deliver
increasingly prioritised user interfaces that combine multiple not only a more natural user experience but also streamlined
input modalities to create more accessible and efficient digital automation and greater accessibility for a diverse audience.
experiences. Integrating gesture recognition and voice
command processing allows users to interact naturally with This paper presents a unified multimodal human–
computing systems, reducing reliance on traditional input computer interaction platform that combines hand gesture
devices and supporting a wider variety of tasks and recognition and voice command input with agent-driven
environments. Contemporary gesture recognition algorithms automation to enhance system responsiveness, usability, and
achieve high levels of accuracy and adaptability by utilising extensibility. The proposed system is architected to advance
advanced sensor fusion and machine learning techniques, the state-of-the-art in input recognition, system
making them suitable for real-world deployments. Similarly, responsiveness, and extensibility for both desktop and smart
improvements in voice command technology, supported by ecosystem environments.
growth in natural language processing, enable robust hands-
free operation for both local and web-based applications. II. LITERATURE REVIEW

Despite these advances, genuine synergy between Jia J et al. [1] presented an integrated framework that
modalities remains a significant challenge for system leverages both speech and gesture modalities to enable fluid
designers. Robust multimodal frameworks must address educational interactions. They demonstrate that multimodal
ambiguity in user input, achieve effective data fusion, and deep learning models, when paired with sensor fusion
ensure system transparency to maintain user trust. In strategies, enhance both dynamic gesture recognition and
response, the introduction of AI agent-based orchestration spoken command accuracy in real-world environments,
and modular communication frameworks, such as the Model setting a technical baseline for future HCI systems.
Context Protocol (MCP), helps bridge the gap between intent

IJISRT26FEB1459 [Link] 2902

Volume 11, Issue 2, February – 2026 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 [Link]

Ridhun M et al. [2] proposed attention-based hybrid combining complementary input channels such as speech and
models that achieve advanced performance in recognizing gesture. The study emphasized that multimodal systems
concurrent hand gestures and speech input. Their solution reduce user cognitive load and error rates by allowing flexible
illustrates, through rigorous testing, that context-driven interaction strategies depending on context and user
modality fusion is vital for supporting quick, reliable user preference. The findings highlight that effective fusion of
commands across multimedia and smart device applications. modalities improves robustness and accessibility, particularly
in environments where traditional input devices are
Ravanbakhsh S et al. [6] offers a multi-hypothesis impractical. This work provides foundational justification for
approach, reducing classification errors and increasing integrating gesture and voice inputs in modern interactive
flexibility when recognizing user gestures, particularly in systems, reinforcing the design choices adopted in the
larger gesture vocabularies. proposed multimodal framework.

Gao Q et al. [7] presented parallel CNNs for image and Jaimes A et al. [19] presented a comprehensive survey
sensor data that creates a scalable recognition model with that maps the multidisciplinary requirements for HCI,
improved reliability, which is critical for both consumer and emphasizing that systems benefit from adaptively leveraging
industrial human-computer interaction applications. both vision and audio modalities. This foundational review
highlights the importance of agent-based orchestration and
Merge AI et al. [12] proposed an MCP client-server context-aware decision-making in resolving input ambiguity
methodology that is formalized as a modular, extensible and ensuring robust user experiences.
interface for connecting AI agents with diverse tools and
cloud applications. The paper details server features such as Dysnix et al. [20] proposed an in-depth analysis of the
dynamic tool discovery, secure capability negotiation, and MCP server infrastructure is presented, outlining both data
context-aware request handling, highlighting how this and transport layer innovations. Key features such as JSON-
architecture enables seamless orchestration and cross- RPC 2.0 message structure, lifecycle management, and
environment automation for modern agent-based HCI support for both stateful and stateless communications are
systems. discussed, while security paradigms like zero-trust and
federated learning are positioned as critical for scalable,
Oviatt et al. [16] examined the role of multimodal reliable, and adaptive deployments in edge, cloud, and hybrid
interfaces in enhancing human–computer interaction by contexts.

III. METHODOLOGY

Fig 1 System Architecture Diagram of the System

A. System Architecture routed to a core AI agent, which maps the user's actions to
The system is designed on a modular architecture corresponding tasks or external application functions.
combining multiple user input modalities and AI agent
orchestration. The user interacts with a cross-platform B. Gesture and Voice Command Recognition
desktop or web app, implemented with standard JavaScript Hand gesture recognition is implemented using the
frameworks for usability and accessibility. The application MediaPipe Hand Tracking framework, which detects and
captures hand gesture input via webcam and processes voice tracks 21 three-dimensional hand landmarks in real time from
commands through an integrated speech recognition module. webcam input. These landmarks provide a compact and
All recognized intentions—irrespective of input type—are robust representation of hand posture and movement,

IJISRT26FEB1459 [Link] 2903

Volume 11, Issue 2, February – 2026 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 [Link]

enabling reliable gesture interpretation under varying lighting

conditions. The extracted landmark coordinates are passed to
a lightweight Convolutional Neural Network (CNN)
classifier for gesture classification. The lightweight CNN
classifier is a custom-designed neural network optimized for
real-time performance. It consists of convolutional layers
with ReLU activation, followed by max-pooling and fully
connected layers for classification. The model architecture IV. RESULT AND ANALYSIS
minimizes computational overhead while maintaining high
recognition accuracy, making it suitable for interactive A. Overview of Evaluation
human–computer interaction scenarios. The proposed multimodal command system was
experimentally evaluated to measure its performance in
C. AI Agent Orchestration gesture recognition, speech understanding, intent
A centralized AI agent interprets all incoming gesture interpretation, and system-level responsiveness. The
and voice inputs, determines the requested task, and evaluation aimed to assess how effectively gesture-based and
coordinates downstream actions. For standard desktop voice-based commands can enhance human–computer
operations (e.g., opening files, switching windows), the agent interaction through an intelligent AI agent supported by an
executes the required action directly. For interactions MCP server connectivity.
involving external services—like fetching emails, updating Figure 2 illustrates the complete operational workflow
calendars, or querying third-party APIs—the agent generates of the proposed multimodal system, comprising two
and transmits structured requests to back-end connectors, independent yet synchronized input pipelines—gesture and
utilizing the Model Context Protocol (MCP) for secure and voice. The left branch captures visual gestures via webcam
scalable integration. frames processed through MediaPipe for hand keypoint
extraction, while the right branch handles voice input through
D. MCP Integration and Service Connectivity a microphone, converting speech into a textual form using a
External application access is achieved through a series Speech-to-Text (STT) engine. Both recognized modalities are
of lightweight MCP connectors, each exposing necessary forwarded to the Intent Extraction and Resolution layer,
capabilities and tool functions to the AI agent. MCP- which identifies user goals and determines the type of action
compliant connectors can be developed in any mainstream required. The AI Agent then plans and routes the commands
programming language and deployed locally or on cloud toward the MCP/Dispatcher layer, responsible for invoking
infrastructure. The protocol ensures that agents can only appropriate service calls—either local (e.g., file system or OS
invoke allowed functions, and all data exchange is secured control) or external (e.g., Gmail, GitHub, or browser APIs).
according to best practices suitable for non-commercial Once the requested operation completes, the returning data is
project environments. All integrations are tested using simple re-interpreted by the AI Agent and presented to the user
authorization tokens or session-bound access, allowing rapid through a unified feedback interface that includes on-screen
demonstration and end-user testing. messages, auditory confirmation, or visual UI updates. This
hierarchical flow ensures asynchronous multimodal
E. Workflow and User Operation processing while maintaining low latency and modular
System flows are built to demonstrate end-to-end scalability.
functionality typical for multimodal workplaces. A user may
launch the application, enable webcam and microphone, and The evaluation was conducted using a labeled dataset of
use gestures for common shortcuts while issuing voice hand gestures derived from MediaPipe hand landmark
commands for more complex tasks (e.g., “Check my coordinates and spoken voice commands collected under
upcoming meetings” or “Send this file via email”). Results controlled conditions. The dataset was divided into training
and feedback are displayed directly within the application and testing sets using an 80:20 split. Gesture classification
interface. Scenarios are tested iteratively to ensure the fusion accuracy was computed as the percentage of correctly
of modalities and correct orchestration through the agent and classified gestures over the test set. Speech recognition
MCP connections, without requiring dedicated cloud accuracy was measured using word-level transcription
infrastructure or production-scale security. correctness, while intent recognition accuracy was evaluated
based on successful task execution. The lightweight CNN
achieved high accuracy due to effective landmark-based
feature extraction and reduced model complexity, which
limited overfitting and improved generalization.

IJISRT26FEB1459 [Link] 2904

Volume 11, Issue 2, February – 2026 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 [Link]

Fig. 2. Data Flow and Processing Sequence of the Proposed Multimodal System.

B. Gesture Recognition Analysis

Gesture-based commands form the visual modality of the system. The gestures were designed for real-world usability,
including cursor movements, clicks, swipes, and control signals such as screenshot or cancel. Detection and classification were
performed using MediaPipe Hand Tracking followed by a lightweight CNN classifier.

Table 1: Gesture Recognition Metrics

Gesture Detection Accuracy (%) Avg Latency (ms)
Move Cursor 97.2 18
Left Click 95.1 22
Right Click 94.3 21
Swipe (Next/Previous) 93.5 25
Open Palm (Cancel) 96.0 20
Closed Fist (Screenshot) 95.8 19

The gesture recognition module achieved an overall accuracy of 95.3% with minimal latency. Static gestures, such as open
palm, showed higher reliability compared to dynamic gestures like swipe due to frame overlap. The system maintained robust
performance under different lighting conditions, validating MediaPipe’s stability.

C. Speech Recognition and Intent Interpretation

Voice commands are processed through the Speech-to-Text (STT) pipeline, which converts spoken input into text. This text is
then interpreted by the AI agent for intent extraction and execution. The system supports a range of natural commands such as
opening applications, fetching emails, or searching the web.

Table 2: Speech and Intent Recognition Results

Command Type STT Accuracy (%) Intent Accuracy (%) Example Command
Open App 97.6 95.1 Open Chrome
Fetch Emails 95.3 91.8 Get emails from Oct 10
Search Web 96.8 93.9 Search for AI games
GitHub Repositories 94.5 91.0 Show my GitHub repositories
System Control 96.1 94.2 Increase volume, Take screenshot

IJISRT26FEB1459 [Link] 2905

Volume 11, Issue 2, February – 2026 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 [Link]

The STT module maintained transcription accuracy above 95% even in moderate noise. Intent accuracy was slightly lower due
to contextual complexity. This confirms that the STT–AI Agent link effectively bridges natural language input and system actions.

D. AI Agent and MCP Integration Analysis

The AI Agent acts as the decision-making core responsible for interpreting commands and routing them through MCP
connections to relevant APIs or local modules. For instance, voice commands like 'Get my emails on Oct 10' trigger Gmail MCP
connectors, while gestures such as 'Closed Fist' activate local system controls. This modularity ensures seamless interoperability
across various services.

E. Module-Level Performance Metrics

Table 3: Module-Level Performance

Module Model Used Precision (%) Recall (%)
Gesture Detector MediaPipe + CNN 96.8 96.0
STT (Speech-to-Text) Google STT / Wav2Vec2 97.1 96.3
Intent Model (Text Parser) BERT Classifier 95.6 95.0
AI Agent Rule-based Planner + RL Logic 92.0 91.5

The STT model demonstrated the highest precision with minimal transcription errors, while the gesture module remained
stable across lighting variations. The AI agent achieved slightly lower recall due to occasional intent overlap when multiple MCP
services were active. Overall, the system achieved an average precision of 95.3%.

F. Overall System Performance

Table 4: System-Level Metrics

Metric Value
Average Response Time (Gesture) 100 ms
Average Response Time (Voice + MCP Query) 350 ms
Command Throughput 40 commands/min
False Trigger Rate 1.6 %

Gesture-based interactions exhibit minimal latency as VI. CONCLUSION

they execute locally, while voice commands incur slightly
higher delays due to network-based MCP requests. User This paper presented a multimodal command system
surveys indicated a high satisfaction level owing to the designed to enhance human–computer interaction through the
naturalness of multimodal interactions. integration of gesture and voice inputs. The framework
employs MediaPipe-based gesture detection and Speech-to-
V. DISCUSSION Text processing for accurate recognition, which are
interpreted by an AI agent capable of executing system-level
The experimental results demonstrate that integrating and external operations through MCP connectivity.
gesture and voice inputs within a unified multimodal Evaluation results demonstrated strong system performance,
framework significantly enhances interaction flexibility and achieving recognition accuracy above 95%, precision of
system usability. Compared to unimodal approaches, the 95.3%, and average execution latency below 400 ms. These
proposed system enables context-aware task execution, outcomes highlight the efficiency and stability of the
allowing users to perform both simple and complex proposed design in enabling real-time interaction. The
operations efficiently. The low response latency and high modular and scalable architecture further supports easy
recognition accuracy indicate that MediaPipe-based gesture extension to additional services or domains, making the
tracking combined with a lightweight CNN is suitable for system a promising foundation for next-generation
real-time applications. The AI agent and MCP-based interactive and assistive computing environments.
integration further contribute to modularity and scalability,
enabling seamless interaction with external services such as REFERENCES
email and repositories. While the system performs robustly
under controlled conditions, future work may focus on [1]. Jia J, Hu Z, Wang R, et al. "A Multimodal Human-
improving intent disambiguation in noisy environments and Computer Interaction System and Its Applications."
expanding adaptive learning capabilities for personalized Sensors, 2020;20(11):3215.
interaction. [2]. Ridhun M, et al. "Multimodal Human Computer
Interaction Using Hand and Speech Recognition."
Human-Computer Interaction. ICICT, 2022.

IJISRT26FEB1459 [Link] 2906

Volume 11, Issue 2, February – 2026 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165 [Link]

[3]. Wu J, et al. "Fusing multi-modal features for gesture [23]. Wu Y, et al. "A new human-computer interaction
recognition." Proc. 15th ACM Int. Conf. Multimodal paradigm: Agent interaction model based on large
Interaction, 2013:453-6. models and its prospects." Virtual Reality & Intelligent
[4]. Siddiqui N, Chan RHM. "Multimodal hand gesture Hardware, 2025;7(3):237-266.
recognition using single IMU and acoustic [24]. Rusan HA, et al. "Human-Computer Interaction
measurements at wrist." PLoS ONE, Through Voice Commands Based on Deep Learning."
2020;15(1):e0227039. Proc. 2022 Int. Conf. Electrical and Computing
[5]. Agrawal A, et al. "Vision-based multimodal human- Technologies and Applications. IEEE, 2022.
computer interaction technique using dynamic hand [25]. Chaturvedi S. "Voice Recognition Systems: An
gesture recognition." 2013 IEEE Int. Conf. Image example of human–computer interaction." SSRN
Information Processing. Electronic Journal, 2024.
[6]. Ravanbakhsh S, Pitsikalis V, Katsamanis A, et al. [26]. Kettebekov S, et al. "Understanding gestures in
"Multimodal Gesture Recognition via Multiple multimodal human-computer interaction." International
Hypotheses Rescoring." J. Machine Learning Research, Journal of Human–Computer Studies, 2000;53:153-
2015;16:261-294. 170.
[7]. Gao Q, Liu J, Ju Z. "Hand gesture recognition using
multimodal data fusion and multiscale parallel CNN."
Expert Systems, 2021.
[8]. Cohen PR, Oviatt S, Wu L, et al. "The role of voice input
for human-machine communication." PNAS,
1995;92(22):9921-9927.
[9]. Liu J, Li Y, Sun J, et al. "A survey of speech-hand
gesture recognition for the development of multimodal
interfaces in human-computer interaction." IEEE Trans.
Human-Machine Systems, 2010;40(6):465-79.
[10]. El-Azazy AAMEH, et al. "Enhancing Human-
Computer Interaction through Speech Recognition and
AI." Engineering Research Journal, 2025;54(1):59-102.
[11]. Wu X, et al. "Multimodal gesture recognition." Proc.
19th ACM Int. Conf. Multimodal Interaction, 2017.
[12]. Merge AI. "5 real-world Model Context Protocol
integration examples." Merge AI Blog, 2025.
[13]. OpenAI. "Model Context Protocol (MCP) - OpenAI
Agents SDK." 2025.
[14]. Anthropic. "Introducing the Model Context Protocol."
Anthropic News, 2024.
[15]. Cyclr. "Model Context Protocol (MCP) for AI
Integration." [Link], 2025.
[16]. Oviatt S. "Multimodal Interfaces." CRC Press, 2003.
[17]. Montero CS, et al. "Multimodal interaction: A review."
Computer Science Review, 2022; 43:1-15.
[18]. Cardenas EJE, et al. "Multimodal hand gesture
recognition combining temporal information." J. Visual
Communication and Image Representation, 2020.
[19]. Jaimes A, Sebe N. "Multimodal human–computer
interaction: A survey." Computer Vision and Image
Understanding, 2007;108(1–2):116-34.
[20]. Dysnix, “MCP Architecture: Advanced Techniques
Review,” Blog, 2025. [Link]
architecture-review.
[21]. Katsamanis A, et al. "Multimodal Gesture Recognition
for HCI." Artificial Intelligence Review, 2016;43(1):1–
54.
[22]. Katsamanis A. "Understanding Gesture and Speech
Multimodal Communication." ACM Digital Library,
2020.

IJISRT26FEB1459 [Link] 2907