Tribhuvan University

Institute of Computer Science and Information Technology

Nepali Speech to Text & Summarization System with

Wav2Vec 2.0 and Text Rank

A PROJECT REPORT

Submitted to

Department of Computer Science

and Information Technology

Texas International college

In partial fulfillment of the requirements for the Bachelors in Computer Science and Technology

Submitted by

Biplop Khadka(26227/077)
Bishal Thakur(26230/77)
Sabin Kumar Thokar (26246/077)

Under the Supervision of

Sukraj Limbu
1
 Tribhuvan University

Institute of Computer Science and Information

Technology

Texas International College

Supervisor’s Recommendation

I hereby recommend that this project prepared under my supervision by Biplop

Khadka, Bishal Thakur and Sabin Kumar Thokar entitled “Nepali Speech to
Text & Summarization System with Wav2Vec 2.0 and Text Rank” in partial
fulfillment of the requirements for the degree of Bachelors in Computer Science
and Information Technology is recommended for the final evaluation.

SIGNATURE

Mr. Sukraj Limbu

SUPERVISOR

CSIT
Texas International College, Mitrapark Kathmandu

2
 LETTER OF APPROVAL

This is to certify that this project prepared by Biplop Khadka, Bishal Thakur and
Sabin Kumar Thokar entitled “Nepali Speech to Text & Summarization
System with Wav2Vec 2.0 and Text Rank” in partial fulfillment of the
requirements for the degree of Bachelor in Computer Science and Information
Technology has been evaluated. In our opinion, it is satisfactory in the scope and
quality as a project for the required degree.

SIGNATURE of Supervisor SIGNATURE of HOD/ Coordinator

……………. ………………

Lecturer Coordinator

Texas International College Texas International College

Mitrapark, Kathmandu Mitrapark, Kathmandu

SIGNATURE of Internal Examiner SIGNATURE of External Examiner

………… ……………..

3
 ABSTRACT

This project aims to develop a web-based system that efficiently transcribes spoken Nepali into text
and generates concise summaries of the transcribed content. By leveraging deep learning models
such as Wav2Vec2 for automatic speech recognition (ASR) and the TextRank algorithm for
summarization, the system will provide an accessible tool for users needing transcription and
summarization services. It will be particularly beneficial for journalists, educators, legal
professionals, and content creators who require accurate speech-to-text conversion and summarized
insights from lengthy discussions or recordings.

The primary objectives of this project are to develop an efficient Nepali ASR system using deep
learning techniques, fine-tune existing speech recognition models with Nepali language datasets for
improved accuracy, implement an automatic text summarization module that extracts key points
from transcribed speech, create a user-friendly web application that integrates ASR and
summarization functionalities, and evaluate and improve the system’s performance by addressing
challenges related to Nepali language processing. By addressing these objectives, the project aims to
contribute to the advancement of Nepali NLP tools, making speech recognition and summarization
more accessible for various real-world applications.

However, this system has certain limitations. The accuracy of speech recognition may be affected by
factors such as background noise, speaker accents, and variations in pronunciation. Additionally, due
to the limited availability of high-quality Nepali language datasets, the model's performance may not
be as robust as its English counterparts. The summarization process may also struggle with
understanding complex contexts and generating completely accurate summaries, especially for
longer conversations or technical content.

Keywords: Nepali Speech Recognition, Summarization, Wav2Vec, TextRank, Automatic Speech

Recognition , Python, Deep Learning, Nepali Language.

4
 ACKNOWLEDGEMENT

The successful completion of this project would not have been possible without the
guidance and support of several individuals and institutions. First and foremost, we would
like to express our deepest gratitude to our supervisor, Sukraj Limbu, for his unwavering
guidance, patience, and insightful suggestions throughout the duration of this project. We
are also thankful to the faculty of the Department of Computer Science and information
Technology, Texas International College, for providing us with the resources and a
supportive environment conducive to learning and innovation. Lastly, we extend our
heartfelt thanks to our families, friends, and colleagues for their constant encouragement
and support, which motivated us throughout this journey.

Biplop Khadka(26227/077)
Bishal Thakur(26230/77)
Sabin Kumar Thokar (26246/077)

5
 Table Of Contents
Chapter 1: Introduction..........................................................................................................................1

1.1 Introduction..............................................................................................................................1

1.2 Problem Statement...................................................................................................................2

1.3 Objectives................................................................................................................................2

1.4 Scope and Limitations.............................................................................................................2

1.5 Development Methodology.....................................................................................................4

1.6 Report Organization.................................................................................................................5

Chapter 2: Background Study and Literature Review...........................................................................7

2.1 Background Study........................................................................................................................7

2.1 Literature Review.........................................................................................................................9

Chapter 3: System Analysis and Design..............................................................................................11

3.1 System Analysis.........................................................................................................................11

3.1.1 Requirement Analysis.........................................................................................................11

3.1.2 Feasibility Analysis.............................................................................................................13

3.1.3 Object Modeling using Class Diagram...............................................................................14

3.1.4 Dynamic Modelling using State and Sequence Diagrams..................................................16

3.1.5 Process Modelling using Activity Diagrams.......................................................................18

3.2 System Design............................................................................................................................19

3.2.1 Component Diagram...........................................................................................................19

3.2.2 Deployment Diagram..........................................................................................................20

3.3 Algorithm Details.......................................................................................................................21

Chapter 4 : Implementation and Testing..............................................................................................24

4.1 Implementation...........................................................................................................................24

4.1.1 Tools Used...........................................................................................................................24

4.1.2 Implementation Details of Modules....................................................................................25

4.2 Testing........................................................................................................................................27
6
 4.2.1 Test Cases for Unit Testing.................................................................................................27

4.2.2 Test Cases for System TestingTest.....................................................................................30

4.3 Result Analysis...........................................................................................................................34

Chapter 5: Conclusion and Future Recommendations.........................................................................35

5.1 Conclusion..................................................................................................................................35

5.2 Future Recommendations...........................................................................................................35

Appendices...........................................................................................................................................36

References............................................................................................................................................38

7
List Of Figures
Figure 1 Agile Methodology................................................................................................................11
Figure 2 Use case diagram...................................................................................................................20
Figure 3 Class Diagram........................................................................................................................22
Figure 4 State Diagram.........................................................................................................................23
Figure 5 Sequence Diagram.................................................................................................................24
Figure 6 Activity Diagram...................................................................................................................25
Figure 7 Component Diagram..............................................................................................................26
Figure 8 Deployment Diagram.............................................................................................................27

8
Abbreviation
NLP Natural Language Processing
ASR Automatic Speech Recognition
AI Artificial Intelligence
STT Speech to text
Wav2Vec Wave-to-Vector (A Deep Learning Model)
ML Machine Learning
BERT Bidirectional Encoder Representations from Transformers

9
 Chapter 1: Introduction
1.1 Introduction
With the rapid advancement in natural language processing (NLP) and deep learning, automatic
speech recognition (ASR) has become a focal point of innovation, enabling machines to understand
and process human speech. While languages like English, Spanish, and Chinese benefit from robust
speech recognition systems due to abundant resources and research, low-resource languages like
Nepali face significant challenges. This project seeks to address the gap by developing a system
capable of recognizing Nepali speech and generating meaningful summaries of the transcribed text.

Nepali is a rich and complex language spoken by millions, characterized by a variety of dialects,
regional variations, and unique linguistic traits. These factors, coupled with limited datasets and
resources, pose considerable challenges in achieving high accuracy for ASR systems. Recognizing
these issues, this project employs cutting-edge models such as Wav2Vec 2.0, a transformer-based
model designed for speech recognition, and the TextRank algorithm, a graph-based approach for
summarization. By fine-tuning these models on Nepali-specific datasets, we aim to overcome
linguistic complexities and data scarcity, ensuring improved accuracy and relevance in both
transcription and summarization.

The integration of ASR and automatic summarization in this system has wide-ranging applications. It
can significantly reduce the time and effort required for tasks like media transcription, legal
documentation, and educational content processing. This project not only highlights the potential of
deep learning techniques in addressing challenges in underrepresented languages but also sets a
foundation for future advancements in Nepali NLP.

This project highlights the adaptation of advanced AI models to address the unique challenges of
low-resource languages like Nepali. By fine-tuning Wav2Vec 2.0 on Nepali speech data, the system
effectively handles variations in pronunciation and dialects, while customizing the TextRank
algorithm ensures accurate and coherent summaries. These efforts demonstrate the potential of
modern AI tools in making technology more inclusive and accessible for underrepresented linguistic
communities.

1
1.2 Problem Statement
The primary challenge addressed by this project is the lack of a reliable speech recognition and
summarization system for the Nepali language. Despite the growing demand for automatic speech
recognition (ASR) tools, the scarcity of large Nepali language datasets, minimal research, and the
linguistic diversity within Nepali complicate the development of accurate ASR systems.
Additionally, most existing speech recognition tools are designed for widely spoken languages like
English, making it difficult to adapt them for Nepali.

Another key issue tackled in this project is the time-consuming process of manual content
processing. In industries such as media, transcription, and education, summarizing long speeches,
lectures, or interviews is often labor-intensive. By integrating automatic summarization into the
system, this project aims to reduce manual efforts and enhance efficiency by providing concise
summaries of extensive text, significantly benefiting content processing workflows.

These challenges highlight the need for a more reliable, accurate, and accessible summarization tool
that overcomes the limitations of current solutions.

1.3 Objectives
 To develop an accurate speech-to-text system for the Nepali language using deep learning.
 To implement an automated text summarization system that generates concise summaries
from transcribed speech.
 To create a user-friendly interface where users can input speech and receive both the
transcription and summary.
 To ensure high accuracy and efficiency for practical use cases like media transcription
and educational content summarization.

1.4 Scope and Limitations

Scope:

 Development of a system that converts spoken Nepali into text using automatic speech
recognition (ASR).
 Implementation of a summarization module that reduces transcriptions into concise
summaries.
 Applicability in sectors such as media, education, and government where Nepali is the
primary language.
 Potential for future extension to support various Nepali dialects and regional accents.
2
Limitations:
 Scarcity of Resources: Limited availability of large, high-quality Nepali speech and text
datasets, which may affect model training and accuracy.
 Diversity of Dialects: The variety of Nepali dialects and accents poses challenges in creating
a universally effective model without comprehensive data covering these variations.
 Speech Variability: Variations in speaker accents, speaking speeds, and background noise
may reduce the accuracy of speech recognition.
 Computational Requirements: The need for powerful computational resources to train and
fine-tune deep learning models, which may limit accessibility and increase project costs.

1.5 Development Methodology

Agile methodology is a project management and software development approach that promotes
iterative progress, collaboration, and flexibility. It emphasizes adaptive planning, evolutionary
development, early delivery, and continual improvement, allowing teams to respond to change more
effectively than traditional project management methods. The Agile framework is based on the Agile
Platform, which outlines values and principles aimed at enhancing customer satisfaction through
early and continuous delivery of valuable software.

Figure 1.1 Agile Methodology

Key Characteristics of Agile Methodology Iterative Development:

Projects are divided into small, manageable increments called iterations or sprints, typically lasting
from one to four weeks. Each iteration involves planning, design, development, testing, and review.

3
Collaboration:

Agile encourages collaboration between cross-functional teams, including developers, testers,

business stakeholders, and end-users. Regular communication fosters a shared understanding of
project goals and requirements.

Customer Feedback:

Frequent feedback loops with customers and stakeholders ensure that the project aligns with their
needs and expectations. This helps in making timely adjustments to the product.

Flexibility:

Agile allows for changes in requirements even late in the development process. This adaptability is
crucial in dynamic environments where user needs may evolve.

Continuous Improvement:
At the end of each iteration, teams conduct retrospective meetings to reflect on what worked well
and what can be improved. This fosters a culture of continuous learning and enhancement.

1.6 Report Organization

The report for the project is organized into the following chapters, each addressing key aspects of the
project’s development and implementation:

Chapter 1 Introduction

This chapter provides an overview of the project, including its objectives, significance, and scope. It
outlines the problem addressed by the project, the need for Nepali language support in speech
recognition and summarization, and a brief introduction to the techniques and models used in the
system.

Chapter 2 Background Study and Literature Review

This chapter reviews existing technologies and research related to speech recognition and
summarization. It covers the development of Automatic Speech Recognition (ASR) systems, with a
focus on challenges in low-resource languages like Nepali. The chapter also explores the TextRank
algorithm for summarization and reviews relevant work done in the field.

4
Chapter 3 System Analysis and Design

This chapter presents the system analysis and design for the Nepali speech recognition and
summarization system. It covers the system architecture, design diagrams, and functional
requirements. This chapter also discusses the choice of models and algorithms used for speech
recognition and summarization, as well as the system's overall workflow.

Chapter 4 Implementation and Testing

This chapter explains the development process of the system, including the tools and technologies
used in implementing both the speech recognition and summarization modules. It also details the
testing methodologies employed to evaluate the system's performance, including the metrics used for
both recognition accuracy and summarization quality.

Chapter 5 Conclusion and Future Recommendations

This final chapter summarizes the outcomes of the project, highlighting the key findings and
achievements. It also provides suggestions for future improvements and enhancements to the system,
including potential extensions to support different Nepali dialects, improve recognition accuracy, and
further optimize performance.

5
 Chapter 2: Background Study and Literature Review
2.1 Background Study
The field of speech recognition and summarization has witnessed rapid advancements, especially
with the rise of deep learning models and their application in natural language processing (NLP).
This section explores the background and fundamental concepts related to Automatic Speech
Recognition (ASR) and summarization, focusing on the unique challenges and developments in the
context of Nepali language processing.

Automatic Speech Recognition (ASR) involves converting spoken language into written text. It is a
core technology of Natural Language Processing (NLP) and has a wide range of applications, such as
voice assistants (e.g., Siri, Alexa), transcription services, and real-time translations. ASR systems can
be broadly classified into two types: speaker-dependent and speaker-independent. Speaker-dependent
systems require training on a specific speaker's voice, while speaker-independent systems can
process speech from any speaker without prior training.

Traditional ASR systems used techniques like Hidden Markov Models (HMMs) and Gaussian
Mixture Models (GMMs) for modeling speech. However, recent breakthroughs with deep learning
models have significantly improved the performance of ASR systems, making them more accurate
and adaptable. One of the main challenges in developing ASR systems for languages like Nepali is
the scarcity of large, high-quality labeled datasets for training models. Many existing ASR systems
are developed for high-resource languages like English, which have abundant data for training and
fine-tuning models.

Nepali, being a low-resource language, has fewer datasets and research resources, making it difficult
to create accurate and robust ASR models. Moreover, the complex morphology, rich inflections, and
regional variations of Nepali further complicate the task. Additionally, there are limited ASR
systems or tools for Nepali, which increases the need for dedicated research and development in this
area.

Deep learning models, particularly Deep Neural Networks (DNNs) and Convolutional Neural
Networks (CNNs), have gained significant traction in speech recognition, due to their ability to learn
and generalize from large datasets. More recently, transformer-based models like Wav2Vec 2.0 have
shown state- of- the-art performance in ASR. Wav2Vec 2.0 is a self-supervised learning model that
learns representations from raw speech data, significantly improving speech recognition accuracy.
Wav2Vec 2.0 and similar models have demonstrated great potential in overcoming challenges
6
related to limited datasets by leveraging unsupervised and semi-supervised learning techniques. Text
summarization involves condensing a lengthy piece of text into a shorter, meaningful version while
retaining the key information. This process can be categorized into extractive and abstractive
summarization.

Extractive summarization involves selecting key phrases, sentences, or segments from the original
text, while abstractive summarization generates a new, condensed version of the text that may not
include exact sentences from the original but conveys the same meaning. In this project, the
TextRank algorithm is used for summarization. TextRank is a graph-based ranking model inspired
by Google's PageRank algorithm. It selects key sentences in a document based on their importance,
which is determined by how well they are connected to other sentences. Speech summarization
involves both recognizing speech and generating a summary of the transcribed text. Combining ASR
with summarization adds another layer of complexity, especially when dealing with long audio files
or multiple speakers.

The combination of speech recognition and summarization is particularly useful in industries like
media, education, and healthcare, where large amounts of speech data need to be processed and
summarized quickly. In the context of Nepali, speech summarization remains an under-researched
area, with few systems developed specifically for Nepali speech-to-text and summarization tasks.
This project aims to address this gap by implementing an ASR and summarization system tailored to
the Nepali language.

In summary, the background study highlights the challenges and opportunities in developing a robust
Nepali ASR and summarization system. The adoption of deep learning models like Wav2Vec 2.0,
combined with text summarization techniques such as TextRank, presents a promising approach to
tackling these challenges and improving speech recognition and summarization for Nepali. This
project aims to make significant contributions to the field by addressing these gaps and providing a
system capable of effectively recognizing Nepali speech and summarizing transcriptions.

2.1 Literature Review

Google speech to text

Google Speech-to-Text is a powerful and flexible speech recognition tool that allows users to
accurately translate spoken words into written text. Voice assistants, customer service automation,
and transcription services are just a few of the many applications that can benefit from its support for
batch transcription and real-time streaming. The technology supports more than 125 languages,

7
including Nepali, and can process audio from a variety of sources, such as microphones, recorded
files, and phone conversations. Its capacity to automatically punctuate text, identify various speakers,
and filter out offensive material are some of its main benefits. Additionally, it enables modifications
through voice adaptation, which improves the recognition of vocabularies and terms relevant to a
given sector.

[Link]

[Link] is a cloud-based platform that gives developers and organizations the ability to incorporate
machine learning-based automation into their applications by offering AI-powered tools for text
summarization and speech-to-text (STT). Voice assistants, automated documentation, and
transcription services can all benefit from its speech recognition system, which translates spoken
words into written text. Though the effectiveness for non-English languages like Nepali may rely on
the availability of language-specific datasets, it supports several languages and uses deep learning
models to improve accuracy. [Link] uses extractive techniques to extract important information
from long documents for text summarization, choosing the most necessary sentences rather than
creating original content. Although it provides a rapid and automatic method of summarizing text,
the original content's coherence and structure determine how well it works.

Procesio

PROCESIO offers a free AI-powered tool designed to transcribe meetings and generate insightful
summaries, supporting over 40 languages including English, Spanish, German, and French. Users
can upload recordings or provide links from platforms like Zoom, Google Meet, Microsoft Teams,
Google Drive, YouTube, or directly from their devices. The tool provides options for automatic
transcription or transcription with summarization, delivering precise transcripts with speaker
differentiation. The generated summaries include key topics, participant roles, discussed challenges,
proposed solutions, and actionable next steps. This service is particularly beneficial for various
meeting types, including sales meetings, where it integrates strategic sales methodologies like BANT
and MEDDIC to assist in lead qualification and deal closure.

Notta

Notta is a user-friendly and effective audio-to-text transcription application that enables users to
accurately and swiftly translate spoken words into written text. It is perfect for meetings, interviews,
and content production because it allows both file-based conversion and real-time transcribing.
Through the Notta dashboard, users can upload audio in a variety of formats, such as MP3, M4A,
8
WAV, and WMA. The system will process the audio and produce a transcript immediately. Its
smooth device synchronisation, which enables users to view and modify transcripts at any time and
from any location, is one of its main benefits. To improve transcription accuracy even in loud
settings, Notta also provides AI-powered noise reduction. Additionally, Notta offers a number of
helpful features to increase workflow effectiveness. It facilitates speaker identification, which makes
it simpler to distinguish between several speakers during a discussion. Transcripts can be edited in
real time, exported in a variety of formats, and integrated with Google Docs and Notion for improved
document management. There are certain restrictions, though. There are usage limits on Notta's free
plan, and in order to access more advanced services like limitless transcription minutes and AI
models with improved accuracy, a subscription is required.

9
 Chapter 3: System Analysis and Design
3.1 System Analysis
The system analysis phase involves understanding the problem, defining system requirements, and
establishing an effective approach for solving the problem. For this project, the main challenge is
developing a Nepali speech recognition and summarization system. The goal is to accurately
transcribe Nepali speech and then summarize the transcriptions into concise, meaningful content.

A major issue in this domain is the lack of robust, pre-existing Nepali language resources and
datasets for training models. While models like Wav2Vec 2.0 have shown impressive results for
speech recognition in high-resource languages, their adaptation to Nepali requires fine-tuning to
address the unique phonetic and syntactic characteristics of the language. Additionally, the scarcity
of Nepali-specific datasets makes training models a complex task. This system aims to bridge the gap
by leveraging multilingual pre-trained models and transfer learning techniques.

The challenge of summarization lies in adapting techniques like TextRank or BART to Nepali's
distinct linguistic features and dialects, ensuring the system generates accurate, meaningful, and
contextually relevant summaries from long audio inputs.

This analysis sets the foundation for designing a robust system that will address these challenges by
fine-tuning existing models, developing effective user interfaces, and ensuring that the system is
optimized for real-world applications in media, education, and government sectors.

3.1.1 Requirement Analysis

Functional Requirements:

The functional requirements of this project focus on delivering key features to ensure efficient and
accurate text summarization. The main functionalities include:

10
Use Case Diagram

Figure 3.1 Use case diagram of Nepali speech to text and summarizer

Use Case Diagram Description:

 Record Audio: The user can directly record audio through the system interface.
 Upload Audio File: Users can upload pre-recorded audio files for transcription.
 View Transcribed Audio Text: After uploading or recording, users can view the transcribed
text of the audio.
 Summarize Text: The system will summarize the transcribed text into key points for the user.

Non-Functional Requirements

11
The non-functional requirements define how the system performs and ensures reliability, security,
and ease of use:

 Performance: This system must efficiently handle text processing, providing summaries
within seconds, even for large text inputs.
 Scalability: As more users access the system and upload files, it must scale to support
increased traffic and text processing without performance degradation.
 Security: User login and history features require secure data handling. User credentials,
texts, and summaries must be stored securely, with access control.
 Usability: The interface must remain simple and intuitive, ensuring both logged-in and non-
logged users can easily navigate the system to input or upload text and view summaries.
 Reliability: This system should ensure minimal downtime and maintain high availability, so
users can access their accounts, upload texts, and generate summaries without disruption.
 Data Privacy: Uploaded texts and summaries are linked to user accounts. Non-authenticated
users' texts should not be stored. For logged-in users, data must be retained securely, allowing
access only through their accounts.

3.1.2 Feasibility Analysis

Technical Feasibility:

The project uses established deep learning frameworks like PyTorch and Hugging Face transformers,
along with Wav2Vec 2.0 for speech recognition and the TextRank algorithm for summarization.
Both Wav2Vec 2.0 and TextRank are open-source and can be fine-tuned with Nepali language data,
making the technical implementation feasible.

Operational Feasibility:

The system will be deployed as a web-based application, ensuring accessibility on any device with
internet connectivity. The user interface is designed for simplicity, allowing users to upload audio
files and receive transcriptions and summaries with ease.

Economic Feasibility:

The economic feasibility of this project is high due to the use of open-source tools and pre-trained
models, which minimize development costs. The project has moderate computational requirements,
which can be handled using cloud services or local machines with GPU support, making it cost-
effective.

12
Schedule Feasibility:

This project was developed using an agile methodology, with iterative development phases for
speech recognition, summarization, and the user interface. This approach ensured that the project
was completed within the designated timeline.

Gantt cahrt diagram:

TODO
3.1.3 Object Modeling using Class Diagram

Figure 2.3 Class Diagram of Nepali speech to text and summarizer

13
The diagram illustrates a modular audio transcription and summarization system, where the
APIEndpoints class serves as the interface for handling user requests, routing them to the
APIHandler, which orchestrates the process. Audio files are managed by the AudioProcessor, which
converts formats, adjusts audio properties, and prepares data for the SpeechToText module to
transcribe using pre-trained models. The transcriptions are passed to the Summarizer, which extracts
key points to generate concise summaries.

The AudioFile class represents audio metadata, supporting efficient file handling. This design
ensures smooth integration between components, with APIHandler acting as the central hub to
process audio inputs into summaries.

3.1.4 Dynamic Modelling using State and Sequence Diagrams

Figure 3.4 State Diagram of Nepali speech to text and summarizer

14
 Figure 3.4 Sequence Diagram of Nepali speech to text and summarizer

The sequence diagram depicts the workflow of an audio transcription and summarization system,
starting with the user uploading an audio file to the APIHandler. The APIHandler first interacts with
the AudioProcessor to convert the audio file to a compatible format, receiving the converted file
path. It then calls the SpeechToText module, which loads a pre-trained model and generates a
transcript from the audio file. This transcript is passed to the Summarizer, which processes it to
produce a concise summary. Finally, the APIHandler returns the results as a JSON response
containing both the transcript and the summary, completing the interaction with the user.

15
3.1.5 Process Modelling using Activity Diagrams

Figure 3.5 Activity Diagram of Nepali speech to text and summarizer

The flowchart outlines the process of handling an audio file for transcription and summarization. It
begins with the user uploading an audio file, which the API validates. If the file format is invalid, an
error is returned; otherwise, the file is converted to a compatible format if needed. The system then
loads a Hugging Face Automatic Speech Recognition (ASR) model, and if the model fails to load, an
error is returned.

Once the model is successfully loaded, the system generates a transcript from the audio file and
passes it to the summarization module to create a concise summary. If the summarization fails, an
error is returned; otherwise, the final response, containing both the transcript and summary, is sent to
the user. The flowchart highlights error handling at each stage to ensure the process is robust and
user-friendly.

16
3.2 System Design
3.2.1 Component Diagram

Figure 3.5 Component Diagram of Nepali speech to text and summarizer

The diagram depicts the architecture of an Audio-to-Text Converter System with four main
components: Frontend UI, Backend API, Model Handler, and Database. The Frontend UI allows
users to upload audio files through the Audio Upload Module and view results via the Result Display
Module. It communicates with the Backend API, which handles audio processing, transcription,
summarization, and error management through dedicated services like the Audio Processing Service,
Transcription Service, and Summarization Service.

The Model Handler is responsible for loading and managing models required for transcription and
summarization. It leverages tools such as Hugging Face Model Loader for accessing pre-trained
models via the Hugging Face API and Local Model Loader for managing models stored locally. The
Database stores critical information, including transcription data and generated summaries, for future
retrieval. The connections between components ensure seamless data flow, with the Backend API
interfacing with the database to fetch and store processed data and the model handler to execute
transcription and summarization tasks effectively.

17
3.2.2 Deployment Diagram

Figure 3.6 Deployment Diagram of Nepali speech to text and summarizer

The deployment diagram illustrates the architecture of the Audio-to-Text Converter System,
showcasing the interaction between the client device, backend server, database, and cloud services.
The Client Device runs a [Link] Frontend UI, where users can interact with the system by sending
API requests to the backend server. The FastAPI Backend Service handles these requests, managing
tasks such as model loading, data storage, and querying results. The Database stores both transcribed
text and generated summaries, ensuring data persistence. The backend server utilizes Local Model
Storage for loading pre-trained models and connects to the Hugging Face API under Cloud Services
for additional cloud-based model access. This architecture ensures efficient processing, combining
local and cloud resources to optimize performance and scalability.

18
3.3 Algorithm Details
3.3.1 Wav2Vec 2.0 for Audio-to-Text (ASR)
Wav2Vec 2.0 is a self-supervised learning model designed for transcribing spoken language into text
with high accuracy. It is based on the Transformer architecture and has achieved state-of-the-art
performance in ASR tasks, even with limited labeled data.

Key Components of Wav2Vec 2.0:

Feature Extraction Layer:

The model begins by processing the raw audio waveform using a convolutional neural network
(CNN).

The CNN learns to extract important features, such as phonemes and audio patterns, which are
transformed into latent representations.

Quantization Layer:

Quantization helps the model capture phonetic and acoustic features while improving its ability to
generalize across various audio samples.

The extracted audio features are converted into discrete units via quantization.

Transformer Layer:

The core component of Wav2Vec 2.0 is a Transformer encoder.

This layer processes the extracted features and learns long-range dependencies within the audio,
enabling the model to understand contextual information across time steps.

Sequence Modeling:

Wav2Vec 2.0 employs masked prediction on audio signals, where parts of the input are masked, and
the model learns to predict the missing sections.

This approach is similar to masked language modeling used in natural language processing (NLP),
enhancing the model's ability to understand audio context.

Fine-Tuning:

Once pre-trained on large amounts of unlabeled audio data, the model is fine- tuned on labeled
datasets specific to the target task, such as transcription.

19
Fine-tuning maps the learned representations to specific words, phrases, or sentences.

Working for ASR:

Input: Users upload an audio file (e.g., .wav format).

Preprocessing: The audio file is preprocessed to create a numerical representation, such as a

spectrogram or waveform, for easier computation.

Feature Extraction: The CNN layers of Wav2Vec 2.0 extract latent audio features, transforming the
raw input into a structured format.

Contextualization: The extracted features are passed through the Transformer encoder, which
contextualizes the information by capturing phonetic patterns, accents, and dependencies.

Decoding: The model's output is decoded into human-readable text, resulting in an accurate
transcription of the spoken content.

Frameworks and Libraries for Wav2Vec 2.0:

PyTorch:
Used for training and fine-tuning the model. Hugging Face Transformers:
Provides pre-trained Wav2Vec 2.0 models that can be customized for specific tasks using labeled
datasets
3.3.2 TextRank for Text Summarization
TextRank is an unsupervised graph-based ranking algorithm designed for natural language

processing tasks such as keyword extraction and text summarization. It applies concepts from

Google's PageRank algorithm to identify the most important sentences in a document and generate

an extractive summary.

Key Components of TextRank:

 Graph-Based Ranking: Constructs a graph where sentences are nodes, and edges represent
similarity between sentences.
 Sentence Similarity: Computes similarity scores using cosine similarity or other distance
metrics based on word embeddings or TF-IDF.
 Iterative Scoring: Assigns importance scores to sentences based on their relationships
within the graph.

20
Text Summarization Workflow:

 Input: The user submits the transcribed text from ASR.

 Preprocessing: Sentences are tokenized and converted into vector representations.
 Graph Construction: A similarity matrix is built where nodes represent sentences, and
edges represent similarity scores.
 Ranking & Selection: The TextRank algorithm assigns scores iteratively and selects the
most relevant sentences.
 Output: The user receives an extractive summary composed of key sentences from the
input text..

21
 Chapter 4 : Implementation and Testing
4.1 Implementation
4.1.1 Tools Used
Case Tools:

 Visual Studio Code (Code Editor): It used as the primary code editor for both the frontend
and backend development. Its support for extensions and plugins enhanced productivity
during development.
 Librosa (Audio Processing Library): Android Studio was utilized for the development of
the Android application of this project. It provided an integrated environment for designing,
coding, testing, and debugging the Android app.
 Hugging Face Transformers (Model Framework): Wav2Vec 2.0 and BERT models were
accessed and fine-tuned using the Hugging Face library, which provided pre-trained state-of-
the-art models for transcription and summarization.
 Git (Version Control): Employed for version control during development. It allowed
efficient code management, collaboration, and tracking of changes throughout the
implementation phase.

Programming Languages & Technologies:

 Python (Backend Development): Python was the primary programming language for
backend development, model integration, and API creation.
 HTML, CSS, and JavaScript (Frontend Development): HTML was used for structuring
the web pages, CSS (specifically Tailwind CSS) for styling the interface, and JavaScript for
adding interactive features and client-side functionality.
 [Link] (Frontend Framework): [Link], a React-based framework, was chosen for
developing the web application’s frontend. It provided server-side rendering, improved
performance, and seamless integration with APIs.
 APIs (Backend-Frontend Communicatio): FastAPI was used to develop the backend APIs
due to its speed, simplicity, and automatic generation of OpenAPI documentation. FastAPI
allows the creation of RESTful APIs that communicate between the frontend and backend,
facilitating tasks like user authentication, data processing, and serving machine learning
models.

22
  SQLite (Inbuilt Python Database): SQLite, a lightweight inbuilt database in Python, was
used for local data storage and quick prototyping. Its seamless integration with Python
ensured efficient handling of small-scale data requirements.

4.1.2 Implementation Details of Modules

The SummarizationService module in the project is responsible for the core functionality of
generating text summaries. It utilizes multiple utility classes for text processing and implements two
main algorithms: Wav2Vec 2.0 for audio-to- text(ASR) and BERT for Text Summarization. Below
is a detailed breakdown of the various classes and methods used in this module.

1. FastAPI Class and Methods

FastAPI is a modern, fast (high-performance) web framework designed for building APIs. It
facilitates routing requests, handling HTTP operations, and serving responses.

 @[Link]("/input-audio"): Endpoint for audio processing and summarization.

 @[Link]("/audio"): Endpoint for audio file upload and processing.
 @[Link]("/abstract-file"): Endpoint for file-based abstract text summarization.

2. generateTranscriptForFileUsingHF Method (from main module)

generateTranscriptForFileUsingHF(audio_file_path, hf_model):

 Takes the path to an audio file and the HuggingFace model as inputs.
 Processes the audio and returns a text transcript.

3. get_summary_from_text Method (from summarizer module)

This method generates a summary of the provided text.

 get_summary_from_text(text):
 Takes a text string as input and generates a summary of that text.

4. load_huggingface_model Method (from main module)

This method loads the HuggingFace model, which is used to transcribe and process audio files.

 load_huggingface_model(model_name).
 Loads the HuggingFace model specified by the given model name.

5. load_local_model Method (from main module)

23
This method loads a local model for transcription, possibly in cases where a HuggingFace model is
not used.

 load_local_model(model_path, processor_path).
 Loads a local transcription model and its associated processor.

6. [Link] Method

This method is used for executing external shell commands, such as converting an audio file format.

 [Link](command, shell=True):
 Executes a command in the shell (used for converting audio files to a specific format like
FLAC).

7. JSONResponse Class (from [Link])

This class is used to send JSON responses back to the client.

 Sends the processed data (transcript and summary) as a JSON response.

 JSONResponse(content=data):

8. load_local_model Method (from main module)

This method loads a local model for transcription, possibly in cases where a HuggingFace model is
not used:

 Takes paths to model and processor files as input.

 Loads and returns the model and its processor.

9. [Link] Method

 This method is used for executing external shell commands, such as converting an audio file
format:
 Used here to run external commands, like converting audio files from one format to another
(e.g., from WAV to FLAC).

10. JSONResponse Class (from [Link])

 A class used to create a JSON response in a FastAPI application.

 Returns structured data (like transcripts and summaries) as JSON.

Two summarization methods are applied:

24
  TF-IDF: This method scores and selects sentences based on the importance of words in
the text.
 Cosine Similarity: This method compares sentence similarity using a vector-based
approach.
 The top sentences from both methods are combined, deduplicated, and arranged in the
original order.
 The final summary is output as a paragraph for easy readability.

4.2 Testing
4.2.1 Test Cases for Unit Testing

Table 4.1: Test Case for Unit Testing

Test
Case Description Input Data Expected Outcome Result
ID

Test transcription of Clean Nepali Accurate

TC-
clear Nepali audio audio file text Pass
01 transcription

500-word Nepali Concise

TC- Summarization of
article summary within Pass
02 long Nepali text
100 words

TC- 100-word Nepali Concise and

Summarization of
03 article meaningful Pass
short Nepali text
summary

25
 Table 2: Test Case for Unit Testing

Test
Case Description Input Data Expected Result
Outcome
ID

Nepali audio file

TC- Test transcription Correct
containing Pass
03 accuracy for transcription in
conversational
Nepali speech text format
speech

Test summarization Nepali transcript Concise and

TC-
containing narrative accurate summary Pass
04 quality for
complex content generated

transcripts
Upload an audio file System
TC- Verify error handling
in an unsupported display an Pass
05 for unsupported
format (e.g., .aac) error message
formats

Table 3: Test Case for Performance Testing

Test
Case Description Input Data Expected Outcome Result
ID
Test system Nepali audio Transcription
TC- responsetime file containing process completes
Pass
06 for clear and within an acceptable
transcription continuous time frame
speech
Test system Nepali audio file System processes
TC- performance for containing a lengthy audio efficiently and
Pass
07 long-duration conversation or maintains accuracy
audio speech

26
 Table 4: Test Case for File Upload Summary Generation

Test
Case Description Input Data Expected Outcome Result
ID
Verify system valid Nepali audio File is successfully
TC- can upload and file uploaded for uploaded and
Pass
08 process an processing ready for
audio summarization.
file
Test summary
Accurate and
TC- generation A
concise summary Pass
09 after file processed
is generated
upload audio file

Table 5: Test Case for Compatibility Testing

Test
Case Description Input Data Expected Result
Outcome
ID
System works
Test system on Windows, Linux,
TC- seamlessly on
different operating macOS Pass
10 all platforms
systems
System
Mobile
Test mobile browser functions
TC- browser
compatibility correctly in Pass
11 (Chrome,
mobile
Safari)
browsers

27
4.2.2 Test Cases for System TestingTest
Case 1: Speech-to-Text Conversion

Test Case ID: TC-01

Test Case Name: Speech-to-text Conversion

Test Case Description: This test case verifies the system's ability to convert Nepali speech into text.

Preconditions:

 The system is up and running.

 A Nepali audio file is available for processing.

Test Steps:

 Upload the Nepali audio file into the system.

 Start the conversion process.
 Wait for the system to process and generate the transcription.

Expected Results:

 The audio is accurately transcribed into Nepali text.

 No errors or inaccuracies in transcription are found.

Post-Conditions:

 The transcription is stored in the system and is available for further processing.

Test Case 2: Handling Audio Input

Test Case ID: TC-02

Test Case Name: Handling Audio Input

Test Case Description: This test case checks how the system handles audio inputs of varying lengths.

Preconditions:

 The system is operational.

 A Nepali audio file is available for testing.

Test Steps:
28
  Upload the Nepali audio file to the system.
 Observe how the system processes the audio input.
 Ensure the system can process both short and long audio files without errors.

Expected Results:

 The system should process the audio file smoothly, regardless of length.
 There should be no delay or failure in processing the audio input.

Post-Conditions:

 The audio file has been successfully processed, and transcription is available.

Test Case 3: Performance Testing

Test Case ID: TC-03

Test Case Name: Performance Testing with Long Audio Files

Test Case Description: This test case measures the system’s performance when processing long
Nepali audio files.

Preconditions:

 The system is functional and has sufficient resources.

 A long Nepali audio file is available.

Test Steps:

 Upload a long Nepali audio file.

 Initiate the processing of the audio.
 Measure the time taken to process the audio.

Expected Results:

 The system processes the long audio file without significant delays.
 There are no system crashes or performance issues.

Post-Conditions:

 The long audio file is successfully transcribed and stored.

Test Case 4: File Upload and Summary Generation

29
Test Case ID: TC-04

Test Case Name: File Upload Summary Generation

Test Case Description: This test case verifies the system’s ability to upload audio files and generate
summaries.

Preconditions:

 The system is operational.

 The user has a valid audio file ready for processing.

Test Steps:

 Upload the audio file to the system.

 Start the audio processing and summarization.
 Wait for the system to generate a summary of the audio content.

Expected Results:

 The audio file is successfully uploaded.

 The system generates a concise and accurate summary.

Post-Conditions:

 The summary is saved and accessible for further review.

Test Case 5: System Stability under Extended Usage

Test Case ID: TC-05

Test Case Name: System Stability under Extended Usage

Test Case Description: This test case ensures that the system remains stable when used continuously
for an extended period.

Preconditions:

 The system is running.

 Multiple Nepali audio files are available for testing.

Test Steps:

30
  Navigate to the "History" page.
 View the list of generated summaries.

Expected Results:

 The system should handle multiple audio file uploads and processing without crashing.
 There should be no degradation in performance over time.

Post-Conditions:

 The system remains stable and operational after processing multiple files.

4.3 Result Analysis

The testing results of the Nepali Speech Recognition and Summarization system show that the
system performs well in terms of transcription accuracy, summarization quality, and platform
compatibility. It successfully processes individual audio inputs and generates relevant summaries
with satisfactory response times. Performance testing indicated efficient operation without
significant bottlenecks, and compatibility testing confirmed the system works seamlessly across
different operating systems and mobile browsers. However, the system has limitations in handling
simultaneous audio inputs, which could affect its performance in real-time scenarios. Additionally,
improvements can be made by training the models on larger Nepali datasets and integrating speech
synthesis for a more comprehensive solution. Despite these limitations, the system demonstrates
reliable performance and offers a strong foundation for further enhancements.

31
 Chapter 5: Conclusion and Future Recommendations
5.1 Conclusion
This project has successfully developed a Nepali Speech Recognition and Summarization system that
can convert spoken Nepali into written text and summarize the transcriptions efficiently. By
leveraging advanced models such as Wav2Vec 2.0 for Automatic Speech Recognition (ASR) and
BERT for summarization, the system demonstrates the potential of applying cutting-edge machine
learning techniques to low-resource languages like Nepali. Despite the inherent challenges of
processing Nepali, such as limited available data and linguistic complexities, the models performed
adequately, showing that with fine-tuning and appropriate data, effective speech-to-text and
summarization capabilities can be achieved for Nepali. The overall results highlight the potential of
the system to bridge communication gaps and support content accessibility in Nepali.

5.2 Future Recommendations

Looking ahead, several improvements could be made to enhance the system's functionality and
performance. One key area is expanding support for multiple Nepali dialects, as the language has
several variations across regions, which can improve the system’s accuracy and usability.
Additionally, the summarization system could be further optimized by training on larger and more
diverse Nepali datasets, which would enable it to handle a wider range of text types and domains.
Another promising addition would be the integration of speech synthesis to provide a complete
voice-to-voice summarization tool, allowing for seamless interaction. Finally, creating mobile and
offline versions of the system would greatly increase its accessibility, especially in areas with limited
internet connectivity, making it more practical for everyday use by a broader audience.

32
Works Cited
text-to-speech. (n.d.). Retrieved 11 12, 2024, from [Link]:
[Link]

33
 Annex i
{Screenshot}

34
35
 Annex ii
{Source Code}

1. Speech to text
def adjust_volume(data,sr=16000,norm="peak"):
# Peak normalization of all audio to -1dB
meter = [Link](sr) #create BS.1770 Meter
# print(data)
# print([Link](data).shape)
loudness = meter.integrated_loudness([Link](data))
# print(f'Before: {loudness} dB')
if norm == "peak":
# This is peak normalization which depends on the original volume of audio file
peak_normalized_audio = [Link](data,-1.0)
elif norm=="fixed":
# Actually this is loudness normalization to a fixed level irrespective of volume in original file
peak_normalized_audio = [Link](data, loudness, 0)
else:
peak_normalized_audio = data
loudness = meter.integrated_loudness([Link](peak_normalized_audio))
# print(f'After peak normalization: {loudness} dB')
return peak_normalized_audio

def convertAudio(src_audio_path,format="mp3"):
filename, ext = getFileNameAndExtension(src_audio_path)
# if the extension is already flac no need to convert
if ext == format:
return src_audio_path
# create a temporary flac file as flac is supported in torchaudio
dest_audio_path = f"temp/{filename}.{format}"
# using AudioSegment from pydub to convert from any audio format to flac as flac is compressed
format of wav and torchaudio only supports wav and flac
audio = AudioSegment.from_file(src_audio_path,format="m4a")
36
 # Export the audio to flac file
[Link](dest_audio_path, format=format)
return dest_audio_path

def deleteTempAudio(dest_audio_path):
[Link](dest_audio_path)

def writeOutputToFile(output, audio_file_path,translated=False):

filename,ext = getFileNameAndExtension(audio_file_path)
destination_file_path = f"./transcripts/{filename}.txt"
with open(destination_file_path,"w",encoding="utf-8") as f:
[Link](output)
return destination_file_path

def generateTranscriptFromHuggingFaceModel(audio_input,model):
# set mono=True as the SpeechRecognitionPipelince can only work with mono audio
speech_array, sr = [Link](audio_input,mono=True,sr=16000)
array_length = speech_array.shape[0]
# print(speech_array, array_length)
# for longer audio, segmentation needs to be done to prevent program from consuming RAM
which may cause error, so I am diving the entire audio to smaller segments and will process these
segments
if array_length > 250000:
list_of_segments = segmentLargeArrayforHuggingface(speech_array,array_length, 200000)
# print(list_of_segments)
output = ''
for segment in list_of_segments:
output += model(segment)["text"]
else:
output = model(audio_input)["text"]
return output

def generateTranscriptFromLocalMode(input_file,model, processor,device="cpu",dosegment=True):

37
 speech_array, sampling_rate = [Link](input_file)
speech_array = speech_array.numpy()
speech_array = adjust_volume(speech_array,sampling_rate,norm="fixed")
speech_array = torch.from_numpy(speech_array)
resampler = [Link](sampling_rate, 16000)
resampled_array = resampler(speech_array).squeeze()
if len(resampled_array.shape) == 1:
resampled_array = resampled_array.reshape([1,resampled_array.shape[0]]
if resampled_array.shape[1] >= 200000 and do_segment == True:
print('The input file is longer than 10 seconds')
list_of_segments = segmentLargeArrayForLocal(resampled_array)
# print(list_of_segments)
output = ''
for segment in list_of_segments:
if [Link]()[1] > 0:
logits = model([Link](device)).logits
# print(logits)
pred_ids = [Link](logits,dim=-1)[0]
output += [Link](pred_ids)
else:
output += ''
else:
print('The input file is less than 10 seconds')
logits = model(resampled_array.to(device)).logits
# print(logits)
pred_ids = [Link](logits, dim = -1)[0]
# print("Prediction:")
output = [Link](pred_ids)
return output

def generateTranscriptForFile(input_file_path, hf_model, local_model, local_processor,

model_type="huggingface"):
audio_extensions = ['mp3','wav','flac','m4a']

38
 filename, ext = getFileNameAndExtension(input_file_path)
if ext in audio_extensions:
print(f"{input_file_path} is a valid audio file, so proceeding to generate transcript")
if ext == "m4a":
print(f"{filename} is in m4a format, so converting it to mp3 format")
input_file_path = convertAudio(input_file_path,"mp3")
start_time = [Link]()
if model_type == "huggingface":
output = generateTranscriptFromHuggingFaceModel(input_file_path,hf_model)
print(f"Transcript generated in {[Link]() - start_time} seconds for {filename}")
elif model_type == "local":
output = generateTranscriptFromLocalModel(input_file_path,local_model,local_processor)
print(f"Transcript generated in {[Link]() - start_time} seconds for {filename}")
if ext == "m4a":
print(f"Deleting temporarily created mp3 file")
deleteTempAudio(input_file_path)
destination_file_path = writeOutputToFile(output,input_file_path)
print(f"Transcript for {filename} is written at {destination_file_path}")
else:
print(f"{input_file_path} is not a valid audio file, please enter a valid audio file with extension
mp3, wav or flac")

def generateTranscriptForFileUsingHF(input_file_path, hf_model):

audio_extensions = ['mp3','wav','flac','m4a']
filename, ext = getFileNameAndExtension(input_file_path)
if ext in audio_extensions:
print(f"{input_file_path} is a valid audio file, so proceeding to generate transcript")
if ext == "m4a":
print(f"{filename} is in m4a format, so converting it to mp3 format")
input_file_path = convertAudio(input_file_path,"mp3")
start_time = [Link]()
output = generateTranscriptFromHuggingFaceModel(input_file_path,hf_model)
print(f"Transcript generated in {[Link]() - start_time} seconds for {filename}")

39
 if ext == "m4a":
print(f"Deleting temporarily created mp3 file")
deleteTempAudio(input_file_path)
destination_file_path = writeOutputToFile(output,input_file_path)
print(f"Transcript for {filename} is written at {destination_file_path}")
return output
else:
print(f"{input_file_path} is not a valid audio file, please enter a valid audio file with extension
mp3, wav or flac")
return None

2. Text Rank algorithm

stop_words = open("./text_rank/[Link]",'r',encoding="utf-8").read()

word_endings = open("./text_rank/word_endings.txt",'r',encoding='utf-8').read()

kriyapads = open("./text_rank/minimal_kriyapad.txt",'r',encoding="utf-8").read().split("\n")

samyojaks = open("./text_rank/[Link]",'r',encoding="utf-8").read().split("\n")

valid_chars = "./text_rank/valid_chars.json"

def get_summary_from_text(text,force_use_purnabiram_model=False):

global stop_words, word_endings, kriyapads, samyojaks

is_complete_sentence = True

# if "।" not in text:

purnabiram_count = [Link]("।")

if not force_use_purnabiram_model:

if purnabiram_count*100 < len(text):

is_complete_sentence = False

else:

is_complete_sentence = False

40
 valid_characters = text_rank_tokenizer.get_valid_chars(valid_chars)

if not is_complete_sentence:

text = text_rank_tokenizer.add_purnabiram(text,kriyapads,samyojaks)

# Split the sentence into array of words and patagraph in its array. (as Array of Array of the words)

sentences = text_rank_tokenizer.get_sentences_as_arr(text)

text = text_rank_tokenizer.remove_useless_characters(text,valid_characters)

sentences = text_rank_tokenizer.remove_repeating_sentences(sentences)

if len(sentences) == 0:

return "It is not a valid text. Please try again with a valid text."

elif len(sentences) == 1:

return sentences

words_arr = text_rank_tokenizer.get_words_as_arr(sentences)

# Remove the stop words from the array

words_arr=text_rank_tokenizer.remove_stop_words_and_filter_word_arr(words_arr,word_endings,
stop_words)

# remove empty sentences and lone word sentences and update sentences accordingly

sentences, words_arr = text_rank_tokenizer.remove_empty_sentences(sentences, words_arr)

# Tokenize the words and sentences into numbers

tokens, token_dict = text_rank_tokenizer.tokenize(words_arr)

# Create a association matrix

association_matrix,counter_vector=text_rank_ranker.create_association_matrix(tokens,No_of_uni
que_chars= len(token_dict))

# Calculate influence of each word on the paragraph

41
 word_influence_vector=text_rank_ranker.calculate_word_ranks(association_matrix,
counter_vector)

# Based on the word importance ranking, calculate t sentence importance ranking.

sentence_influence=text_rank_ranker.calculate_sentence_influence(tokens,word_influence_vector)

# Get first n sentences from the given text as summarized text.

n=[Link](len(sentences)*0.33))

summary_sentences=text_rank_ranker.get_n_influencia_sentence(sentence,sentenceinfluence,n=n)

summarized_text = text_rank_ranker.get_summarized_text(summary_sentences)

return summarized_text

3. Fast API

@[Link]("/input-audio")
async def summary():
try:
# hf_model = load_huggingface_model("Biplop/wav2vec2-nep")
l_model, l_processor = load_local_model("./model", "./processor")
t1 = [Link]()
transcript_op = generateTranscriptForFileUsingHF(
"./input/audio.mp3", hf_model
)
print(transcript_op)
t2 = [Link]()
print(f"Time for huggingface model: {t2-t1} seconds")
t1 = [Link]()
summary = get_summary_from_text(transcript_op)
t2 = [Link]()
# return summary
return {
"transcripts": transcript_op,

42
 "summary": summary,
"time": round(t2 - t1, 4),
}
except Exception as e:
print(e)
return "fail"
@[Link]("/audio")
def create_upload_file(audio: UploadFile = File(...)):
try:
hf_model = load_huggingface_model("Biplop/wav2vec2-nep")
ext = [Link](".").pop()
[Link]("static/audio", exist_ok=True)
file_location = f"static/audio/{uuid.uuid1()}{[Link]}"
with open(file_location, "wb+") as file_object:
file_object.write([Link]())
if ext == "wav" or ext == "flac":
transcript = generateTranscriptForFileUsingHF(file_location, hf_model)
summary = get_summary_from_text(transcript)
[Link](file_location)
else:
dest_path = f"static/audio/{uuid.uuid1()}[Link]"
command = f"ffmpeg -i {file_location} {dest_path}"
[Link](command, shell=True)
transcript = generateTranscriptForFileUsingHF(dest_path, hf_model)
summary = get_summary_from_text(transcript)
[Link](dest_path)
[Link](file_location)
return JSONResponse(content={"transcript": transcript, "summary": summary})
except:
print(f"Error: {e}")
return JSONResponse(
content={"error": "Failed to process audio file"}, status_code=500
)

43
44