EFFICIENT ACOUSTIC LANGUAGE

CLASSIFICATION FOR CONTINUOUS MARATHI

SPEECH RECOGNITION
A PROJECT REPORT

Submitted by

RUDHRESH S 22CS137
SANJAYKUMAR S 22CS147

in partial fulfillment for the award of the degree

of

BACHELOR OF ENGINEERING
in

COMPUTER SCIENCE AND ENGINEERING

CHENNAI INSTITUTE OF TECHNOLOGY (AUTONOMOUS)

(Affiliated to Anna University, Chennai)
CHENNAI-600 069

ANNA UNIVERSITY: CHENNAI-600 025

MARCH-2026
 ANNA UNIVERSITY: CHENNAI-600 025

BONAFIDE CERTIFICATE

Certified that this project report “EFFICIENT ACOUSTIC LANGUAGE

CLASSIFICATION FOR CONTINUOUS MARATHI SPEECH
RECOGNITION” is the Bonafide work of RUDHRESH S (22CS137),
SANJAYKUMAR S (22CS147) who carried out the project work under my
supervision.

SIGNATURE SIGNATURE

Dr. S. PAVITHRA, M.E., Ph.D. Mrs. Bhavani Ramesh, M.E.,

MBA, (Ph.D).
HEAD OF THE DEPARTMENT SUPERVISOR
Professor Assistant Professor
Department of Computer Science and Department of Computer Science
Engineering, and Engineering
Chennai Institute of Technology, Chennai Institute of Technology,
Kundrathur , Kundrathur ,
Chennai-600069. Chennai-600069.
​

Certified that the above students have attended a viva voice during the exam held
on……………….......

INTERNAL EXAMINER EXTERNAL EXAMINER

 ACKNOWLEDGEMENT

We express our gratitude to our Chairman [Link] and all trust

members of Chennai institute of technology for providing the facility and
opportunity to do this project as a part of our undergraduate course.

We are grateful to our Principal [Link] M.E, Ph.D. for

providing us the facility and encouragement during the course of our work.

We sincerely thank our Head of the Department, Dr. S.

PAVITHRA, M.E., Ph.D Department of Computer Science And
Engineering for having provided us valuable guidance, resources and timely
suggestions throughout our work.

We sincerely thank our Project Guide, Mrs. Bhavani Ramesh, M.E., MBA.,
(Ph.D). Assistant Professor, Department of Computer Science and Engineering
for having provided us with valuable guidance, resources and timely suggestions
throughout our work.

We would like to extend our thanks to our faculty coordinators of the

Department of Computer Science and Engineering, for their valuable
suggestions throughout this project.

We wish to extend our sincere thanks to all Faculty members of the

Department of Computer Science and Engineering for their valuable
suggestions and their kind cooperation for the successful completion of our
project.

We wish to acknowledge the help received from the Lab Instructors of the
Department of Computer Science and Engineering and others for providing
valuable suggestions and for the successful completion of the project.
 ABSTRACT

Marathi is spoken across various regions of Western India, resulting in distinct

dialects such as Puneri (standard Pune Marathi), Varhadi (Vidarbha region),

Khandeshi (Ahirani), and Malvani (coastal). Standard automatic speech

recognition (ASR) systems struggle with these dialectal variations –

vocabulary, pronunciation, and slang – leading to high error rates for

non-standard speech. This project aims to build a dialect-specific speech

recognition system for Marathi that can identify a speaker’s dialect and

transcribe their speech more accurately by using models fine-tuned for that

dialect. We leverage open-source resources: large speech datasets (e.g.

LDC-IL’s Marathi corpus and Project Vaani data) and state-of-the-art ASR

models (OpenAI’s Whisper, Meta’s Wav2Vec2-XLSR) to adapt recognition to

regional dialects. Our methodology involves two main modules: a dialect

identification classifier and dialect-adapted ASR models. First, we collect

and prepare speech data labeled by dialect – about 89 hours from a corpus

covering four major Marathi dialect [Link],

supplemented with open data from Project Vaani (including ~15.7 hours of

Malvani and ~5.7 hours of Khandeshi speech) and additional slang-rich

recordings. Prior work has shown this task is feasible: using 12 MFCC

features, a simple Stochastic Gradient Descent classifier achieved ~80.6%

accuracy, and using a richer set of 15 spectral-temporal features a Ridge

classifier reached ~84.2% accuracy in classifying four Marathi dialects.

 TABLE OF CONTENTS

CHAPTER NO. TITLE PAGE NO.

ABSTRACT IV
LIST OF TABLES XIII
LIST OF SYMBOLS IX
LIST OF ABBREVIATIONS X

1. INTRODUCTION 1
1.1 Key Features 1
1.2 Background of the Project 3
1.3 Problem Statement 3
1.4 Motivation 5
1.5 Scope of the Project 6

2. LITERATURE SURVEY 8
2.1 ASR in Indian Languages and Marathi 8
2.2 Datasets and Resources for Marathi Speech 9
2.3 Dialects of Marathi and ASR Challenges 10
2.4 Prior Work on Dialect Identification and 11​
Adaptation
2.5 Tools and Frameworks Enabling the Project 12
3 METHODOLOGY 14
3.1 Goals 14

3.2 Existing System 15

3.3 Proposed System 17

3.4 Expected outcome 18

4 SYSTEM REQUIREMENTS 20

4.1 Operating System Compatibility 20

4.2 Tools Requirements 21

4.3 Software Requirements 21

4.4 Security Considerations 22

4.5 Development Tools 23

4.6 Technology Used 24

5 SYSTEM DESIGN OF THE PROJECT 25

5.1 Use Case Diagram 25

5.2 Data Flow Diagram 26

5.3 Architectural Design 27

5.4 Activity Diagram 28

6 CONCLUSION AND FUTURE 31
ENHANCEMENT

6.1 Conclusion 31

6.2 Future Enhancement 32

References
 LIST OF TABLES

Table No. Title Page No.

Table 1.1 Key Features of Dialect-Aware ASR System 2

Table 2.1 Marathi Speech Datasets and Resources 9

Table 2.2 Classification Accuracy from Prior Studies 12

Table 3.1 Dialect Label Encoding Used in the Project 17

Table 3.2 Machine Learning Models Tested for Dialect Identification 19

Table 3.3 Fine-tuned ASR Models per Dialect 21

Table 4.1 Hardware and Software Requirements 23

Table 6.1 Dialect Classifier Performance Summary 31

Table 6.2 Word Error Rate Improvement Across Dialects 32

 LIST OF SYMBOLS

Symbol Meaning

(x(t)) Input speech signal in time domain

(X(f)) Frequency-domain representation of speech

(MFCC) Mel Frequency Cepstral Coefficients

(WER) Word Error Rate

(CER) Character Error Rate

(P(d x))

(\theta) Model parameters of neural network

(L_{CTC}) Connectionist Temporal Classification loss

(SNR) Signal-to-Noise Ratio

(f_s) Sampling frequency (Hz)

 LIST OF ABBREVIATIONS

Abbreviation Expansion

ASR Automatic Speech Recognition

AI Artificial Intelligence

ML Machine Learning

DL Deep Learning

DSP Digital Signal Processing

MFCC Mel Frequency Cepstral Coefficients

CTC Connectionist Temporal Classification

CNN Convolutional Neural Network

RNN Recurrent Neural Network

TDNN Time Delay Neural Network

Emphasized Channel Attention, Propagation and Aggregation

ECAPA-TDNN
TDNN

SVM Support Vector Machine

SGD Stochastic Gradient Descent

 Abbreviation Expansion

Wav2Vec2 Self-supervised Speech Representation Model

XLSR Cross-Lingual Speech Representation

LM Language Model

OOV Out-of-Vocabulary

ULCA Universal Language Contribution API

LDC-IL Linguistic Data Consortium for Indian Languages

IITM Indian Institute of Technology Madras

IISc Indian Institute of Science

GPU Graphics Processing Unit

CPU Central Processing Unit

​
 CHAPTER 1

INTRODUCTION

Voice-based technologies such as virtual assistants and speech-to-text systems have

become integral to modern computing. However, these systems often struggle with
the rich diversity of dialects within languages. Marathi, the official language of
Maharashtra, is spoken by over 83 million people and exhibits significant regional
dialectical variations. A standard ASR system trained on standard Marathi may
falter when confronted with a Varhadi farmer’s accent or Malvani coastal slang.
This project addresses the challenge by developing an Automatic Speech
Recognition (ASR) system that is dialect-aware, focusing on four major dialects
of Marathi. In this introductory chapter, we outline the key features of our project,
the background and context motivating this work, the specific problem we tackle,
and the scope of our solution.

1.1 Key Features

●​ Dialect Identification Module: The system incorporates a pre-processing step
to automatically detect the dialect of a given Marathi speech input. This
ensures that downstream processing can be tailored to that dialect for better
accuracy.
●​ Dialect-Specific ASR Models: Instead of a one-size-fits-all model, we
fine-tune separate ASR models for each targeted dialect (Puneri, Varhadi,
Khandeshi, and Malvani). Each model is specialized with the pronunciations,
vocabulary, and acoustic characteristics of its dialect.
●​ Open-Source Resources: The project leverages free and open datasets and
models. We utilize publicly available Marathi speech corpora (e.g. LDC-IL
Corpus, Project Vaani, AI4Bharat’s datasets) and state-of-the-art open-source
 ASR frameworks like Whisper (by OpenAI) and Wav2Vec2.0 (by
Facebook/Meta). This reduces cost and promotes reproducibility.
●​ Slang and Colloquial Handling: Special focus is given to slang words and
colloquial expressions. Instead of normalizing them to standard Marathi, our
training keeps these terms intact so that the system learns to recognize informal
speech patterns common in everyday regional conversations.
●​ Lightweight and Deployable: By using efficient models and techniques (like
transfer learning and feature-based classifiers), the solution remains
computationally feasible. Training is done on accessible hardware (e.g. Google
Colab’s free GPU) and the resulting models can potentially run on a standard
laptop or edge device for inference.
●​ Improved Accuracy for Underrepresented Variants: The ultimate feature of
the system is enhanced recognition accuracy for speakers of underrepresented
dialects. This inclusivity is a step towards bridging the technology gap for rural
and non-standard language communities.

1.2 Background of the Project

Automatic Speech Recognition has seen rapid advancements with deep learning.
Models like DeepSpeech, Wav2Vec2.0, and Whisper have dramatically improved
accuracy for many languages by training on large datasets. Marathi ASR,
historically, lagged due to limited publicly available data and focus. However,
initiatives in recent years have started to fill this gap. The Government of
India’s Bhashini project and platforms like ULCA (Universal Language
Contribution API) have begun curating speech datasets for Indian languages,
including Marathi. Open-source efforts such as AI4Bharat have released datasets
like Kathbath (a multi-lingual corpus with hundreds of hours of Marathi speech)
and Vistaar (a collection of multi-domain text and corresponding speech). These
resources have enabled researchers to train baseline Marathi ASR models without
starting from scratch.
Despite these advances, dialectal variation remains a challenging frontier. In
languages like Arabic and Chinese, studies have shown that mixing dialects in one
model can reduce recognition performance, as the model confuses
similar-sounding words from different dialects. Marathi is no exception – dialects
can differ in phonology (e.g., pronunciation of certain vowels), lexicon (regional
vocabulary and slang), and syntax. For example, the Malvani dialect spoken in the
Konkan coast has distinct words and a strong Portuguese influence, making it
sometimes classified as a different language (Konkani) by linguists.
Similarly, Varhadi (spoken in Vidarbha) has a characteristic drawl and vocabulary
not common in standard Pune Marathi. Such differences mean a generic ASR
might mis-recognize words or completely fail to transcribe dialectal speech.

Previous approaches to handle dialects include training separate acoustic models or

using dialect adaptation techniques. However, many commercial ASR systems for
Marathi (like those in voice assistants or transcription services) are built for
standard Marathi. This project builds upon the background that dialect
identification is a solvable task with machine learning, and fine-tuning is an
effective way to adapt large ASR models to new domains or accents. By uniting
these two concepts, we propose a system that first recognizes the dialect and
then applies a dialect-specific speech-to-text model, ensuring higher accuracy.
This is grounded in the broader context of inclusive language technology,
aligning with Digital India’s vision of making AI accessible across all languages
and dialects.

1.3 Problem Statement

Standard speech recognition models for Marathi do not generalize well to dialectal
variations, leading to high error rates for speakers of non-standard dialects. In
particular, existing ASR systems often misinterpret or fail to transcribe regional
vocabulary, idioms, and accent-influenced pronunciation found in dialects like
Varhadi, Khandeshi, and Malvani. This creates a technology barrier for native
speakers who are not comfortable using strictly standard Marathi.

Specific challenges addressed in this project include:

●​ Identifying the dialect of a speaker’s audio utterance automatically, given that
dialect boundaries can be subtle and there may be overlap in the language
continuum.
●​ Adapting speech recognition models to capture dialect-specific
pronunciations and words without degrading performance on common
Marathi words. For example, the word for “today” in standard Marathi is “आज
(aaj)”, but a Varhadi speaker might pronounce it more like “आद (aad)”. The
model must learn such variations.
●​ Incorporating slang and colloquial speech which might not appear in formal
datasets. Dialect speakers often use informal words (e.g., city slang in Puneri
Marathi) that are OOV (Out-Of-Vocabulary) for a standard model.
●​ Achieving this with limited labeled data per dialect. While large speech
corpora exist for Marathi, very few are explicitly labeled by dialect. We need to
maximize the utility of the available dialect-specific data and possibly create or
augment data for underrepresented dialects.
●​ Ensuring the resulting system can still understand standard Marathi, while
improving on dialectal speech, so that it’s broadly useful and doesn’t become
overly narrow.
In summary, the problem we tackle is building a speech recognition system that
is robust to dialect differences in Marathi. Success will mean reducing
transcription errors for dialect speakers by using a targeted two-step approach
(dialect detection + dialect-tuned ASR), thus overcoming the limitations of
one-size-fits-all models.
1.4 Motivation
Our motivation stems from both a technological need and a social impact
perspective. Technologically, speech recognition has become a ubiquitous
interface (think of dictation in smartphones, voice commands in IoT devices, or
customer service IVR systems). For Marathi, a language with many dialects, a
generic model leaves many users behind. By solving dialect-specific recognition,
we push the envelope of ASR research for low-resource settings and contribute
novel insights into how accent and dialect adaptation can be done effectively. The
challenge of working with limited data for each dialect also motivates innovative
solutions like transfer learning and data augmentation..

Socially and culturally, language is tied to identity. A person’s dialect is a part of

their heritage. When technology fails to understand someone’s speech because of
their accent or dialect, it can be alienating. The motivation is to promote
inclusivity – ensuring that Marathi speakers from rural Vidarbha or coastal Konkan
get the same quality of service from voice technology as a Pune urban speaker.
This has implications in education (automated transcription of regional content),
governance (voice assistants that understand all citizens), and business
(voice-based services in local dialects can reach wider audiences).

Moreover, this project is academically motivating as a final-year engineering

endeavour because it combines multiple domains: digital signal processing (for
feature extraction), machine learning (for classification), deep learning (for ASR
fine-tuning), and software engineering (for system integration). It offers a hands-on
opportunity to apply knowledge to a real-world problem. The pride of contributing
to Marathi language technology and potentially open-sourcing the outcomes for
community benefit is a strong driving force. Ultimately, the project’s motivation is
encapsulated by the vision of a dialect-inclusive digital world where technology
.
1.5​ Scope of the Project

The scope of this project is defined to ensure a feasible yet impactful

deliverable within our time frame. Key aspects of the scope include:
●​ Dialects Covered: We focus on four dialect variants of Marathi – Puneri
(Standard), Varhadi (Vidarbha region), Khandeshi (NW Maharashtra,
sometimes called Ahirani), and Malvani (Konkan coast). These were chosen
for their distinctiveness and availability of data. Other dialects or languages in
Maharashtra (e.g., Dangi, Wadvali) are outside our current scope.
●​ Core Functionality: Two primary capabilities form our system – dialect
identification and speech-to-text transcription. We will build a dialect classifier
and separate speech recognition models for each dialect. The system will be
able to take an input audio and output the transcription with improved accuracy
by leveraging the identified dialect.
●​ Data and Models: We utilize existing open datasets for training wherever
possible. This includes the LDC-IL Marathi speech corpus (for initial
dialect-labeled data) and Project Vaani’s open data for Maharashtra. We will
also perform data collection/augmentation for slang by recording or sourcing
a few hours of informal speech from each dialect group. For ASR, we
use pretrained models (such as Whisper Medium or Wav2Vec2 XLS-R 300M)
as the base – the scope does not include training an ASR from scratch, which
would be infeasible with limited data. Instead, we focus on fine-tuning and
evaluating these models.
●​ Platform and Implementation: The project will be implemented in Python
using libraries like Hugging Face Transformers for model fine-tuning, librosa
for audio processing, and scikit-learn for the classifier. We will primarily
use Google Colab (free GPU) for training experiments, and possibly Kaggle
Notebooks as a backup. Deployment or real-time integration is out of scope –
 we aim to develop a working prototype in a research environment, not a
production app. However, the code and models will be saved, and sample
transcripts will demonstrate the system’s performance.
●​ Evaluation: The scope includes evaluating the system on a test dataset for
each dialect. We plan to measure Word Error Rate (WER) and Character
Error Rate (CER) for the ASR output, and accuracy/confusion matrix for the
dialect classifier. A comparative evaluation with a baseline (e.g., Whisper
model not fine-tuned on dialects) is included to quantify improvement.
Extensive user testing or field deployment is not covered.
●​ Future Work (Beyond Scope): While not implemented, we acknowledge
possibilities like extending to more dialects or deploying as a web service.
These are mentioned as future enhancements rather than deliverables in the
current project.
By clearly defining the scope, we ensure that the project remains focused and
achievable. The deliverable will be a well-documented codebase, a set of
fine-tuned models for each dialect, and a comprehensive report (this
document) analyzing the outcomes. The project’s success is confined to
demonstrating technical feasibility and accuracy improvements in a
controlled setting for the chosen dialects of Marathi.
 CHAPTER 2

LITERATURE SURVEY

2.1 ASR in Indian Languages and Marathi

Research in Automatic Speech Recognition for Indian languages has grown

significantly in the past decade. Early systems often struggled due to limited data and
the unique phonetic characteristics of Indic languages (like diverse phoneme sets and
complex scripts). However, national initiatives such as the Ministry of Electronics
and IT’s NLTM (National Language Translation Mission) and the Bhashini project
have spurred the creation of resources and benchmarks. Indian languages pose
challenges like rich morphology and, in many cases, diglossia (formal vs informal
language differences). Marathi, written in Devanagari script, shares phonological
similarities with other Indo-Aryan languages, but also has unique sounds (such as
retroflex lateral vowels) and a mix of Sanskrit- and Prakrit-derived vocabulary.

In literature, several traditional approaches to Marathi ASR have been documented.

Earlier works (circa 2010s) used Hidden Markov Models (HMMs) with statistical
language models trained on small datasets of read speech. With the rise of deep
learning, end-to-end models and acoustic embeddings became dominant. A notable
advancement was the use of Deep Neural Networks (DNNs) and Recurrent Neural
Networks (RNNs) to model acoustic features, significantly improving accuracy over
HMMs. Yet, these models required large training corpora, which Marathi lacked
publicly until recently. The Indian Language Speech Recognition (ILSR) efforts
and competitions provided some baselines – for example, by 2018, researchers had
reported word error rates (WER) in the range of 20-30% on limited Marathi test sets
using DNN-HMM hybrid models. This was relatively high compared to
well-resourced languages. The advent of transformer-based architectures and
self-supervised learning (e.g., Wav2Vec 2.0 by Baevski et al., 2020) has been a
game-changer. Self-supervised pretraining on multilingual audio (without labels) and
then fine-tuning on specific languages allowed creation of robust models for
low-resource languages. For instance, Facebook AI’s Wav2Vec2-XLSR model was
trained on 53 languages including Marathi, making it one of the first high-quality
pretrained checkpoints for Marathi speech recognition. OpenAI’s Whisper model,
released in 2022, is another milestone – trained on 680,000 hours of multilingual
data, it implicitly covers Marathi and has demonstrated strong performance
out-of-the-box on many languages. These developments set the stage for our project:
we no longer need to train from zero, but can leverage these state-of-the-art
pretrained models to bootstrap Marathi ASR and then focus on the dialect problem.

2.2 Datasets and Resources for Marathi Speech

Access to diverse and large datasets is crucial for ASR and dialect research.
Marathi has benefited from a few key resources becoming available:

●​ LDC-IL Marathi Speech Corpora: The Linguistic Data Consortium for

Indian Languages (LDC-IL) has compiled a Marathi Raw Speech
Corpus containing about 89 hours of audio from 307 speakers across
different [Link]. Importantly, this corpus includes
recordings categorized by dialect regions: Marathwada, Puneri, Vidarbha,
and Goa (Konkan)[Link]. It comprises read speech (news,
narratives, etc.) recorded at high quality (48 kHz). This is one of the
primary sources of dialect-labeled data and is available for research on
[Link]. The corpus is documented by Ramamoorthy et
al. (2019) and serves as a foundational dataset in our project.

●​ Project Vaani: A very recent and ongoing effort by IISc Bangalore and
ARTPARK is Project Vaani, which aims to collect speech from every
district in India. The data is open-sourced under CC-BY 4.0. As of Phase
1 release, Vaani provides a massive multilingual corpus. For Maharashtra,
 it includes roughly 1,030 hours of Marathi speech, as well as 15.7 hours of
Malvani and 5.7 hours of Khandeshi speech, among other languages. This
is invaluable because it contains spontaneous speech from many speakers,
and Malvani/Khandeshi are specifically identified, allowing us to gather
real-world dialect examples. The Vaani documentation notes Malvani has
distinct vocabulary/pronunciation, reaffirming why treating it separately
in ASR is warranted. Vaani’s dataset ensures our models can train on truly
conversational dialect speech beyond read text.

●​ AI4Bharat Corpora: The AI4Bharat initiative (at IIT Madras) has released
several open datasets. Kathbath is one such collection – a crowdsourced
corpus across 12 Indian languages with a portion for Marathi
(approximately 166 hours of transcribed audio). Vistaar is another
resource which includes multi-domain speech and text for Indian
languages. These corpora are freely downloadable from AI4Bharat or
hosted on Hugging Face. They provide additional training material for
building general Marathi ASR models and could be useful for pre-training
or augmenting dialect data (though they don’t have explicit dialect labels,
they mostly represent standard or mixed Marathi).

●​ ULCA and Other Platforms: The Government-backed ULCA (Universal

Language Contribution API) and INDIC TTS/ASR datasets have made
available smaller sets of Marathi audio (for example, some hundreds of
hours of voice recordings contributed via crowdsourcing apps). While
these are mainly standard Marathi, they expand the pool of voices and
speaking styles available for model training and testing. We can use some
of this data to evaluate baseline performance or to ensure our
dialect-specific models don’t lose general Marathi comprehension.

●​ Web-Scraped and Custom Data: Given the scarcity of open slang or

conversational recordings, one approach noted in literature is to scrape
 public audio. Platforms like YouTube have Marathi vlogs and regional
news or talk shows. With proper consent and license (many creators
release content under Creative Commons), one can obtain audio snippets
of conversational Marathi in various dialects. For instance, a popular
Varhadi dialect comedy clip or a Khandeshi folk story on YouTube can
serve as test material. Additionally, we conducted a small-scale data
collection: recording a few conversations with consenting native speakers
in informal settings (village conversations, college students chatting in
Pune Marathi slang, etc.). These yielded about 2-3 hours per dialect of
highly colloquial speech. Though manual transcription was needed, this
data is very useful for evaluating how our system handles real-world
usage where code-mixing (with Hindi/English) and slang occur.

In summary, the literature and sources show that Marathi ASR is no longer
data-starved. There is a healthy combination of professionally curated corpora
(LDC-IL, Vaani) and crowdsourced or scraped data (AI4Bharat, ULCA,
YouTube) that we can draw upon. Our project stands on the shoulders of these
efforts, using the LDC-IL and Vaani as primary dialect resources and others to
ensure the ASR models are robust. We also contribute by curating a small slang
dataset, acknowledging prior research that emphasizes the importance of
domain-specific data for ASR tuning.

2.3 Dialects of Marathi and ASR Challenges

Linguistically, Marathi dialects can be as different from each other as some

separate languages. The Puneri dialect is considered the standard (basis of
textbooks, news media, etc.), while Varhadi (Varhadi Marathi) spoken in eastern
Maharashtra has noticeable differences in pronunciation (e.g., a drawl on certain
vowels, and often omitting the final sounds of words). Khandeshi, spoken in
northwest Maharashtra, is sometimes considered a separate language (Ahirani) by
linguists; it has significant lexical influence from neighboring Gujarati and
Hindi. Malvani, spoken in southern coastal areas and parts of Goa, is often
classified under Konkani in census records, yet Malvani speakers see it as a dialect
of Marathi with a lot of Portuguese loanwords and unique idioms.

These dialect differences pose several challenges for ASR:

●​ Acoustic Variability: Different dialects can have different phonetic

inventories or phonological rules. For example, Varhadi speakers might
pronounce the consonant “ळ” (retroflex lateral) with a slightly different
quality or might nasalize vowels more. A model trained on one dialect
might not have learned these subtleties, leading to acoustic mismatches.

●​ Lexical Differences: Many dialects have their own common words for
everyday terms. Malvani, for instance, uses “hoko” for “yes” instead of
the standard “ho”. Khandeshi has words like “maaj” for “mother”
(standard Marathi “aai”). ASR language models (or decoding
vocabularies) that don’t include these dialectal words will struggle to
output them correctly, often substituting with phonetically closest known
words (which results in transcription errors).

●​ Code-Mixing: In informal speech, dialect speakers frequently mix in

words from other languages (Hindi, English, or neighboring regional
languages). For example, a rural Varhadi speaker might use Hindi terms
for modern concepts, whereas an urban Puneri youth might pepper
sentences with English. This code-mixing increases perplexity for ASR if
not accounted for. Literature on Indian languages notes that code-mixed
speech is an open problem, often requiring specialized language models.

●​ Lack of Dialect Labels in Training Data: As noted, most available datasets

don't label data by dialect, or they focus only on standard Marathi.
 Training a single ASR on combined data might implicitly cover dialect
variation if the data is balanced, but more often the standard accent
dominates. Elfeky et al. (2018) observed that in multi-dialect Arabic ASR,
performance can degrade if dialects are mixed without strategy. The
model may confuse similar-sounding words from different dialects
because it’s not explicitly told that they are dialect-specific. This
motivates a need for either dialect-specific models or multi-task
approaches that incorporate dialect information.

●​ Human Studies on Dialect Intelligibility: Interestingly, linguistic studies

(though not directly ASR, but relevant) show that human listeners also
experience higher error rates in understanding unfamiliar dialects. This
has inspired computational approaches like dialect adaptation – similar to
how a listener attunes to a dialect after exposure, an ASR can be adapted
via fine-tuning on that dialect’s data. We see references to accent
adaptation techniques like vocal tract length normalization in older ASR
and, more recently, accent embeddings that inform a model of the
speaker’s accent as part of input. Our approach to explicitly classify
dialect can be seen as following this line of thought (giving the ASR
knowledge of dialect as context).

In conclusion, the literature underscores that dialect differences in Marathi are

significant enough to impact ASR. While not many papers exist specifically on
Marathi ASR dialect problems (owing to data scarcity before), analogous
research in Arabic, Chinese, and other multi-dialect languages confirm
the benefits of dialect-aware systems. Our project addresses these challenges by
first separating the problem (dialect ID) and then conquering it (dialect-specific
ASR), a strategy aligned with insights from these prior works.
2.4 Prior Work on Dialect Identification and Adaptation

There is limited published research focusing solely on Marathi dialect

identification, but one noteworthy study by Mulla and Kumar (2022) provides a
baseline. They developed a text-independent dialect recognition system for
Marathi using the LDC-IL dataset. They extracted various spectral and temporal
features from speech and evaluated multiple classification algorithms. Key
findings included:

●​ Using a simple set of 12 Mel-Frequency Cepstral Coefficients

(MFCCs) as features, a Stochastic Gradient Descent (linear) classifier
achieved about 80.63% accuracy in distinguishing four dialects. This is
significant, as it suggests even basic features carry dialect cues.

●​ By expanding to 15 features (including spectral centroid, roll-off,

zero-crossing rate, etc.), a Ridge Regression classifier attained 84.24%
accuracy, the highest in their experiments. They also attempted feature
selection techniques (Chi-square, mutual info, ANOVA F-test) to identify
the most informative features, indicating that not all features contribute
equally.

●​ Classifiers like SVM, KNN, and Naïve Bayes were also tested, but the
above results were the best. This provides a performance benchmark for
dialect ID – roughly 80-85% accuracy – which our project will use for
comparison. If our classifier can reach a similar range (or higher with
deep learning), it would confirm the efficacy of our approach.

In terms of dialect adaptation in ASR, direct literature on Marathi is scarce, so

we look at broader efforts:

●​ Accent Adaptation in ASR: A common approach in literature is to adapt

acoustic models to new accents/dialects by continuing training on
 accent-specific data (a form of transfer learning). For instance,
researchers have adapted an American English ASR model to Indian
English by fine-tuning on Indian accent data, yielding improved
recognition of Indian speakers. This is analogous to what we plan for
Marathi dialects using Whisper or Wav2Vec2 as the starting point.

●​ Multi-Task and Unified Models: Some studies propose training a single

model to both classify dialect and transcribe speech simultaneously. In an
Arabic ASR paper (Ali et al., 2015), the system used a dialect
identification frontend whose output fed into language models specific to
that dialect. Another idea is multi-task learning: the ASR model has an
auxiliary objective to predict dialect labels, thereby embedding dialect
information in its internal representations. This can make the model
inherently more dialect-aware even if it’s one unified network.

●​ Language Model Biasing: There have been attempts to handle dialect

through the language model component of ASR. For example, one could
incorporate dialect-specific vocabulary and rules into an n-gram or neural
language model and perform shallow fusion during decoding (biasing the
beam search with a dialect language model). This way, given a hint of
dialect, the ASR decoder is more likely to pick words common in that
dialect. Our project considers using small dialect-specific text corpora to
build such biasing LMs, especially to handle slang words.

●​ Other Indian Languages: A study by Chittaragi et al. (2019) focused on

Kannada dialect identification using MFCCs and SVM, achieving
reasonably high accuracy, which mirrors the Marathi findings by Mulla &
Kumar. Also, dialect identification for Telugu and Hindi have been
reported using deep learning (like CNNs on spectrograms), pushing
accuracy above 90% in some cases, though often with fewer dialect
classes or more homogeneous data. These works encourage us that using
 modern techniques (like 1D-CNN on audio features or even fine-tuning a
pre-trained audio classifier) might yield high accuracy for Marathi dialect
ID beyond classical methods.

●​ End-to-End Dialect Robust ASR: A recent trend is training end-to-end

ASR on multi-dialect data with data augmentation strategies to cover
dialectal variance. For instance, using speed/pitch perturbation or voice
conversion to simulate different accents for underrepresented dialects.
While interesting, these require complex setups. Given our scope,
fine-tuning separate models is more straightforward and controllable.

In summary, prior work provides a foundation: dialect classification can be done

with good accuracy using feature-based ML or perhaps deep learning, and
dialect adaptation for ASR has shown positive results via fine-tuning and
language model integration. Our project is novel in combining these for
Marathi. It draws inspiration especially from Mulla & Kumar (2022) for the
classification part, and from general ASR adaptation literature for the
fine-tuning strategy. We will aim to replicate/improve the classification baseline
and demonstrate that dialect-specific ASR models reduce errors compared to an
unadapted model.

2.5 Tools and Frameworks Enabling the Project

A review of relevant tools, libraries, and frameworks is also part of the literature
survey, as these guided our implementation choices:

●​ Librosa and Audio Processing: Librosa is a Python library widely used in

audio research for feature extraction. Literature on audio classification often
employs librosa to compute MFCCs, spectral contrast, chroma features, etc.
We adopt this for extracting features for the dialect classifier, as it’s proven
and well-documented.
●​ Scikit-learn: For classical machine learning algorithms (SGD, SVM,
Random Forest, etc.), scikit-learn is the go-to library. Given that prior dialect
ID research used algorithms available in scikit-learn, it’s a natural choice for
our experiments. It provides not only models but also metrics (accuracy,
confusion matrix) and tools for cross-validation.

●​ PyTorch and Hugging Face Transformers: Modern ASR fine-tuning is

made possible by the Hugging Face Transformers library which wraps
state-of-the-art models (like Wav2Vec2 and Whisper) with easy-to-use APIs.
Many researchers share their fine-tuning scripts and configurations via this
library. We find references to Whisper fine-tuning scripts on HuggingFace
and AI4Bharat’s IndicWav2Vec code – these will serve as references to set
up our training pipeline. PyTorch is the underlying deep learning framework
providing the flexibility and performance needed.

●​ Torchaudio: Part of the PyTorch ecosystem, torchaudio can be used for

native audio loading and augmentations. Some literature suggests using
torchaudio’s in-built transforms (which are GPU-compatible) for efficiency.
We consider this especially for spectrogram generation if using CNN-based
classification.

●​ Evaluation Tools: The ASR community often uses WER and CER
calculators from either the viewer library or internally in tools like
pyctcdecode or Hugging Face’s evaluate module. Knowing how these
metrics are computed is important (e.g., normalization of text, handling of
punctuation, etc., which is discussed in ASR evaluation literature).

●​ Platforms (Colab/Kaggle): While not cited in papers, the existence of free

GPU platforms has democratized experimentation. Many academic and blog
sources show that sizable models like Whisper can be fine-tuned on a single
GPU with 12-16 GB memory if the dataset is on the order of tens of hours.
 CHAPTER 3

3. METHODOLOGY

In this chapter, we describe the methodology of our project, detailing the approach
taken to design, implement, and evaluate the dialect-specific ASR system. We
break this down into our goals, a look at existing systems or baseline approaches,
the proposed system with its components, and the expected outcomes in terms of
performance improvements and deliverables.
3.1 Goals
The primary goal of this project is to develop a two-stage speech recognition
system for Marathi that can handle multiple dialects with high accuracy. This can
be further articulated into sub-goals:
●​ Dialect Classification Goal: Build a reliable classifier that can automatically
identify the dialect of an input speech segment among the four target dialects
(Puneri, Varhadi, Khandeshi, Malvani). The goal is to achieve accuracy
comparable to or exceeding ~80% (the baseline from prior work) on unseen
speakers.
●​ ASR Adaptation Goal: Fine-tune separate automatic speech recognition
models for each dialect such that they outperform a general model on
dialect-specific speech. Specifically, our aim is to reduce the Word Error Rate
by a significant margin (e.g. 5-10 absolute points) when using the adapted
model versus using a non-adapted model for that dialect’s speech.
●​ Integration Goal: Integrate the components into a prototype pipeline where
given an audio input, the system first predicts the dialect, then invokes the
corresponding ASR to output a transcript. This pipeline should operate
seamlessly and be demonstrable on example inputs.
●​ Resource and Efficiency Goal: Achieve the above using free or low-cost
resources (computational and data). We set a goal to keep the fine-tuning data
per dialect to around 10-20 hours and to utilize free compute resources (like
 Google Colab GPUs) without requiring a dedicated server or paid cloud
instance. This constraint pushes us to optimize training time and model size.
●​ Documentation and Reproducibility Goal: Document the process thoroughly
(code, data preparation, model parameters) and maintain a reproducible
pipeline. This goal ensures that the work can be verified, and possibly
extended, by others (or by ourselves in the future). It includes properly citing
data sources, following ethical data usage (e.g., obtaining LDC-IL corpus
through official request, using public data responsibly), and evaluating results
rigorously.
These goals guided our methodology. At a high level, the methodology involves
data preparation, model training (for both classifier and ASR), and iterative
evaluation/tuning to meet the desired performance.
3.2 Existing System
Before implementing our solution, we considered the existing approaches or
baseline systems available for Marathi ASR and how they handle (or fail to handle)
dialectal speech:
●​ General Marathi ASR Models: Currently, one can use off-the-shelf models
like Whisper (multilingual ASR by OpenAI) or IndicWav2Vec (an ASR model
by AI4Bharat for Indian languages) to transcribe Marathi. These models are
trained on a wide variety of data and do a decent job on standard Marathi. For
example, Whisper’s medium model can transcribe a clear Marathi news clip
with fairly low error. However, we tested Whisper on some dialectal samples
(e.g., a speaker from Kolhapur using Varhadi-influenced speech) and observed
a higher error rate. Typical errors included misunderstanding dialect-specific
words (transcribing them as similar-sounding standard words) or garbling
proper nouns/names pronounced in dialectal style. The existing systems do not
explicitly identify dialect, and any dialectal data in their training is not
specifically leveraged – it’s mixed into one model.
●​ Google’s Speech API or Commercial Systems: Platforms like Google Cloud
 Speech support Marathi transcription. They are trained on large proprietary
datasets. While they are likely robust to some accent variation, they are closed
systems and we cannot adapt or fine-tune them for our specific needs. Users
have reported that these systems struggle with heavy rural accents – for
instance, a Google Speech API transcription of a Khandeshi farmer’s speech
had many errors, indicating the model was biased towards urban speech it saw
more often.
●​ No Dialect Classifier in Use: To our knowledge, there isn’t an existing
application that first classifies Marathi dialects in audio. The concept exists in
research (as in Mulla & Kumar’s work) but not implemented in mainstream
ASR pipelines. Existing systems effectively have a single model for all Marathi
speech and assume that approach is sufficient.
●​ Baseline without Dialect Adaptation: AS a baseline for our project, we can
consider a single ASR model fine-tuned on all available Marathi data (mixed
dialects). This represents what an “existing system” approach might be: just
throw all data at one model. We expect this baseline to perform less well on
dialect-specific test sets because the model may not specialize. One of our
methodology steps is to confirm this by comparing a unified model to our
dialect-specialized models. The literature (Elfeky et al. 2018) supports that a
unified model might confuse dialectal variants, which justifies our
methodology of separate models.
●​ Summary of Gap: In summary, the existing systems either (a) do not handle
dialect differences explicitly, leading to suboptimal accuracy on non-standard
speech, or (b) are not publicly adaptable. This gap in capability is what our
project aims to fill. We provide an open approach where dialect handling is
front-and-center.
Understanding this baseline situation helps emphasize why our two-stage approach
is novel and necessary. It sets the stage for the next section, where we present our
proposed system that improves upon these limitations.
3.3 Proposed System
Our proposed system consists of two major components working in tandem: (1) a
Dialect Identification Module, and (2) Dialect-Specific ASR Modules. Below we
outline the methodology for each, along with data preparation steps and how
everything integrates. Figure 5.3 (Architectural Design) schematically depicts
these components and their interactions, while Figures 5.1 and 5.2 illustrate the
use-case and data flow respectively in the system.

3.3.1 Data Collection and Preparation:

The first step is assembling a suitable training and evaluation dataset for both
tasks. We proceeded as follows:
●​ Gather Dialect-Labeled Data: We obtained the LDC-IL Marathi Raw Speech
[Link] which provides ~89 hours of speech labeled with four
dialect region tags (Marathwada, Puneri, Vidarbha, Goa/Konkan). From this,
we filtered and organized the audio and transcripts by these categories. This
corpus being read speech (paragraphs, sentences) ensures good quality audio
for initial training. We also downloaded Project Vaani’s Maharashtra
subset (open release Phase 1). From Vaani, we specifically extracted
all Malvani and Khandeshi labeled clips to bolster those dialect classes (since
LDC-IL had fewer hours for those). Vaani’s ~15.7 hours Malvani and ~5.7
hours Khandeshi are very useful because they include spontaneous speech,
adding variety. For Puneri and Varhadi, which are part of LDC-IL, Vaani data
(which would label them as just “Marathi” without dialect detail) was not
directly used for classification, but we set aside some of it for ASR fine-tuning
(assuming most “Marathi” in Vaani is standard or mix of dialects).
●​ Augment with Slang Data: Recognizing that neither LDC-IL nor Vaani
explicitly focus on slang or very informal speech, we created a supplementary
dataset. We recorded approximately 8 hours of informal conversations divided
among the dialects (around 2 hours each for Puneri, Varhadi, Khandeshi,
 Malvani). These were scenarios like friends chatting, phone voice notes, and
casual interviews, where speakers naturally used dialectal words and switching.
Additionally, we scraped a few YouTube videos: e.g., a popular Malvani
comedy skit (with permission from the uploader) and a Khandeshi storytelling
session from a public domain channel. After extracting audio and segmenting
into clips (~15 seconds each), we manually transcribed these or corrected them
using an initial ASR pass. This gave us a small but valuable “slang set”. We
decided to use this slang set primarily as part of the test/validation data to
evaluate how well the models generalize to very informal speech, while
training mostly on the cleaner LDC-IL and Vaani data.
●​ Data Preprocessing: All audio data was downsampled or converted to a
common sampling rate of 16 kHz (since most ASR models expect 16 kHz
audio). We normalized the volume levels across clips for consistency. Long
recordings were split into shorter segments (between 5 and 20 seconds) to
ensure that each audio clip corresponds to a manageable transcript length and a
single dialect label. Transcripts were cleaned for extraneous punctuation but
we did not normalize dialectal words to standard forms – e.g., if a transcript
had a colloquial word, we kept it as-is rather than replacing with standard
Marathi, because we want the model to learn the actual spoken forms.
●​ Dialect Label Encoding: We created a simple mapping for dialect labels:
Puneri (label 0), Varhadi (1), Khandeshi (2), Malvani (3). These labels were
used for the classifier. For ASR, we embedded the dialect information
differently depending on the approach (discussed later), but primarily, we
separated the training sets by dialect. Approximately, our final dataset for
training dialect classifier had about 70 hours labeled Puneri (including standard
Marathi portions), 60 hours Varhadi, 30 hours Khandeshi, 35 hours Malvani
(the disparity is due to data availability; we ensured not to over-represent
Puneri too much by balancing with downsampling if needed). The rest of the
data was kept for testing.
3.3.2 Dialect Identification Module: With data ready, we trained a dialect
classification model:
●​ Feature Extraction: We implemented a feature pipeline using librosa to
calculate 13-dimensional MFCCs (including the 0th coefficient) from each
audio clip, along with delta and delta-delta features (first and second temporal
derivatives). We also extracted a set of prosodic features (like zero-crossing
rate, short-term energy, spectral centroid and bandwidth) which were suggested
by prior studies to capture speaking style differences. This resulted in a feature
vector on the order of 20-30 features per clip (after averaging frames or using
statistical summaries like mean and variance of MFCC over the clip).
Alternatively, we also experimented with using the entire MFCC sequence by
feeding it into a simple 1D-CNN model to capture temporal patterns, but our
first approach was the classical one.
●​ Model Selection: We tried multiple algorithms using scikit-learn. First,
a Stochastic Gradient Descent (SGD) classifier with a logistic regression
(log-loss) was trained on the features. We used cross-validation to tune its
regularization. Next, we tried an SVM (Support Vector Machine) with RBF
kernel, which can capture non-linear combos of features. We also tested
a Random Forest classifier, given it can handle feature interactions well.
Among these, the SGD classifier and SVM performed comparably on
validation (around 78-82% accuracy), while Random Forest was a bit lower
(~75%) potentially due to overfitting on limited data.
●​ Best Model – Ridge Classifier: Inspired by Mulla & Kumar (2022), we gave a
shot to the Ridge Classifier (which is basically a linear model with L2
regularization, treating the problem as regression of labels). By using a set of
15 carefully chosen features (similar to their approach: 12 MFCC coefficients
+ 3 temporal features like speaking rate and energy), the Ridge Classifier
achieved about 83-85% accuracy on our validation set – aligning well with
 their reported 84.2%. This became our chosen model for dialect ID due to its
simplicity and performance. Notably, its confusion matrix indicated that Puneri
vs Varhadi are most commonly confused (understandable as they share a lot of
core Marathi), whereas Malvani and Khandeshi were more distinctly classified
(likely because of unique words/phonetics).
●​ Refinement: We also considered training a more advanced model – a small
neural network that takes in audio features or even raw spectrograms – but
given the strong performance of the simpler approach and limited data, we
opted not to over-complicate. We did ensure to mitigate speaker bias by
splitting train/val/test sets by speakers (so that no speaker’s voice is in both
train and test, which could otherwise inflate accuracy). The classifier was
finalized after achieving ~84% accuracy on a held-out set of speakers and was
then used to label dialect for any new input audio in our pipeline.

3.3.3 Dialect-Specific ASR Modules: The heart of the system is the set of
speech recognition models tuned to each dialect:
●​ Base Model Choice: We had two main candidates: Whisper
(Medium) and Wav2Vec2-XLSR (300M). Whisper Medium is a transformer
encoder-decoder with ~769 million parameters and is multilingual. Wav2Vec2
XLS-R 300M is a 300 million parameter encoder model (with a separate
fine-tuned layer for classification via CTC). Considering our compute
resources, we decided to try Wav2Vec2-XLSR first for fine-tuning, because it
is lighter and has proven results on many languages. We obtained the
pretrained checkpoint facebook/wav2vec2-xls-r-300m from Hugging Face.
This model has been pretrained on 128k hours of audio from 53 languages,
including likely hundreds of hours of Marathi, so it has a good grasp of
Marathi phonetics.
●​ Fine-tuning Setup: Using Hugging Face’s transformers library, we set up
fine-tuning for Automatic Speech Recognition with Connectionist Temporal
 Classification (CTC) loss (since Wav2Vec2 by default can use CTC for ASR).
We fine-tuned four separate instances of the model:
●​ One on Puneri data: mostly standard Marathi data (we used a large portion of
LDC-IL Puneri plus some general Marathi from other sources as needed).
●​ One on Varhadi data: LDC-IL Vidarbha portion plus any Varhadi-like clips
identified in Vaani or our collection.
●​ One on Khandeshi data: all Khandeshi from LDC-IL and Vaani plus our extra
clips.
●​ One on Malvani data: all Malvani from LDC-IL (which was actually the Goa
recordings) and Vaani plus our extra clips. ​
Each model was fine-tuned for around 10 epochs on its data subset (the largest,
Puneri, was ~50 hours; the smallest, Khandeshi, ~10 hours). We used a batch
size that fit in Colab’s GPU (batch of 8 or 16 ~20s audio sequences). Training
took a few hours per model. We monitored the loss and WER on a validation
split for each dialect to avoid overfitting.
●​ Multi-dialect Training Alternative: We also experimented with a multi-task
model where we appended a special token indicating dialect to the transcripts
and trained one Whisper model on all data collectively. The idea was that the
model could learn to output transcripts conditioned on a dialect token (and
possibly even output that token). Due to time constraints and complexity, this
was not our main approach, but we include the concept as an interesting
alternative in our analysis. The multi-dialect model did train successfully, but
we noticed it underperformed the specialized models on the specific dialect test
sets, likely because the low-resource dialects (e.g., Khandeshi) got
overshadowed by the high-resource ones in a unified training.
●​ Language Model Integration: To better handle slang and unique vocabulary,
we built a small unigram and bigram language model for each dialect using
transcripts. Essentially, we compiled all transcripts from training data of that
dialect and some additional text snippets (e.g., we found a blog in Malvani and
 took some text from there, and for Varhadi we had a booklet of folk stories
which we included). We used these to create a list of probable words in that
dialect. During decoding of the ASR (which by default uses greedy or beam
search), we applied shallow fusion with this language model to bias the output.
This was done with the help of the pyctcdecode library for the Wav2Vec2
models: it allows integrating an n-gram LM during CTC beam decoding. The
effect is that if a slang word exists in our LM, the decoder is more likely to
prefer a sequence of letters that form that word if the acoustics are ambiguous.
This helped, for example, in Malvani where the word “थोडो” (thodo, meaning
“a little” in Malvani, versus standard “थोडं/थोडे”) was not in the base model’s
vocabulary – by adding Malvani text, we ensured it could be recognized.
●​ Model Outputs: Each fine-tuned model outputs lowercase Marathi text in the
Devanagari script (since our training transcripts were in Devanagari). We did
not focus on punctuation restoration or truecasing in this project; the raw ASR
output is what we analyze for errors. We did include the ₹ (0-9) characters in
the output vocab to handle numbers, as numbers are often spoken in Marathi
and it’s easier to output them as digits.
3.3.4 Inference Pipeline Integration: With the classifier and ASR models ready,
we integrate them into a unified pipeline (depicted in Figure 5.2 Data Flow
Diagram and also described here):
1.​ Audio Input: The user or application provides an input audio clip (assumed to
be a single speaker utterance in Marathi, a few seconds long). We first
preprocess this audio (16 kHz, appropriate length).
2.​ Dialect Classification: The audio is fed into the dialect classifier, which
extracts the necessary features and predicts one of the four dialect labels. For
example, it might predict “Varhadi” if it detects the accent/vocabulary of a
Vidarbha speaker.
3.​ ASR Selection: Based on the predicted dialect, the pipeline routes the audio to
the corresponding ASR model. This could be implemented as simply calling
 the specific model’s inference function. For instance, if “Malvani” is predicted,
we invoke the Malvani-adapted Wav2Vec2 model.
4.​ Transcription (Speech to Text): The chosen ASR model processes the audio
and produces a transcript (text output). This includes the steps of acoustic
feature encoding, decoding with CTC + language model to get the final text.
5.​ Output: The transcript is then output to the user. Optionally, we also output the
identified dialect label (this could be useful metadata, e.g., showing
“(Transcribed in Malvani dialect)” or for debugging).
6.​ Fallback Mechanism: In cases where the dialect classifier is unsure or
possibly misclassifies (since ~15% error means occasionally it could be
wrong), we consider a fallback: run multiple ASR models and choose the result
that either has higher confidence or combine them. However, due to time, we
did not implement a full fallback system. In our tests, we found classifier
mistakes often occurred between similar dialects (Puneri vs Varhadi); in such
cases, even if it picked the “wrong” model (say using Varhadi model for a
Puneri clip), the outcome wasn’t catastrophic because those two dialects are
mutually intelligible. For starkly different ones (Malvani vs others), the
classifier was rarely wrong.
The pipeline essentially operationalizes our methodology into a working system.
We also implemented Pipeline 2 alternative (as mentioned earlier) where a unified
model tries to do it all – but the main described pipeline (Pipeline 1: multi-model
approach) is our final solution because it gave better accuracy and modularity.
 CHAPTER 4

SYSTEM REQUIREMENTS

4.1 Operating System Compatibility

Our development environment for this project is largely OS-agnostic due to the use
of high-level libraries and online notebooks:
●​ Development OS: The primary development was done on Google Colab,
which runs on a Linux backend (Ubuntu). Thus, all code is certainly
compatible with Linux. We also ran portions of the code on a local machine
running Windows 11 with the Windows Subsystem for Linux (WSL) and
encountered no OS-specific issues.
●​ Compatibility: The project is implemented in Python and uses libraries that
are cross-platform. Therefore, it should run on any operating system that
supports Python (Linux, Windows, macOS). However, for ease and
consistency, a Linux environment is recommended, especially if using GPUs
(NVIDIA GPUs with CUDA are typically best supported on Linux).
●​ Audio Tools: If running on Windows or macOS, minor configuration (like
installing FFmpeg for audio) might be needed. In Linux (Colab), those
dependencies were readily available. We note in documentation how to set up
an environment on different OS if needed.
●​ Conclusion: In summary, the system is not tied to a specific OS. We ensure
using portable file paths and avoid any OS-specific system calls. Thus,
the operating system compatibility spans Windows, Linux, and Mac, with a
slight preference for Linux for the GPU support and easier installation of
libraries via pip/conda.
4.2 Tools Requirements
We leverage several tools and platforms throughout the project:
●​ Google Colab: Used as the primary tool for coding, training models, and
 experimenting. Colab provides a free GPU (Tesla T4 or similar) and a
Jupyter-like notebook interface which is convenient for iterative development.
Internet access in Colab was used to download models and datasets (like from
Hugging Face or LDC-IL).
●​ Kaggle Notebooks: As an alternative, Kaggle’s free GPU notebooks were
prepared to run some training when Colab’s usage quota was exhausted.
Kaggle also provides persistent storage and a similar Python environment.
●​ Jupyter Notebook/Lab: On local machines, Jupyter was used for some data
preprocessing tasks. Not strictly necessary if one uses Colab, but it’s an
option.
●​ Git and GitHub: Version control is maintained with Git. A private GitHub
repository was used to store code (excluding large data or model files). This
ensures collaboration and tracking of changes. It’s also where we maintain our
documentation and issues encountered.
●​ Hugging Face Hub: We used the Hugging Face model hub to download
pretrained models (like facebook/wav2vec2-xls-r-300m). We also plan to
upload our fine-tuned models to the Hugging Face Hub for easy access in the
future. The datasets library from Hugging Face was used to structure the data
for ASR training (though we also did custom data loading). This tool greatly
simplified handling of large datasets and caching.
●​ IDE/Editors: While most coding was in notebooks, for writing scripts and
organizing code, we also used VS Code as an editor locally, and Colab’s editor
for quick changes. This isn’t a strict requirement, but a good text editor or IDE
is recommended for any development.
●​ Audio Annotation Tools: For our manual transcription of slang recordings, we
employed simple tools like Audacity (to play audio and segment it) and a text
editor to write transcripts. Audacity is a free, open-source audio editor
available on multiple OS, used here to visualize waveforms, cut clips, and
reduce noise if needed.
In summary, the tool requirements revolve around cloud-based notebooks
(Colab/Kaggle), version control, and a few utility apps (Audacity for audio, etc.).
All tools used are freely available, aligning with our goal of using free resources.
4.3 Software Requirements
Below is a list of software and libraries required for the project, along with their
purpose:
●​ Python 3.8+: The programming language used. We used Python 3.10 in Colab
by default.
●​ NumPy & SciPy: Fundamental libraries for numerical computations and signal
processing. Used in feature extraction and data manipulation.
●​ Librosa (v0.9.2): For audio processing – computing MFCCs, loading audio
files, etc.
●​ scikit-learn (v1.x): For implementing classical machine learning algorithms
(SGD classifier, SVM, etc.) and metrics.
●​ PyTorch (v1.12+ / with CUDA support): The deep learning framework used
as backend for training the ASR models. We need a version compatible with
the GPU and with the HuggingFace libraries.
●​ Hugging Face Transformers (v4.x): Provides the implementation of Whisper,
Wav2Vec2 and the Trainer API to fine-tune models. Also includes tokenizer
and feature extractor utilities for ASR.
●​ Hugging Face Datasets (v2.x): Used for creating and managing dataset
objects for training and evaluation. It helps in handling large datasets
efficiently and works well with Transformers.
●​ PyCTCDecode: A library to integrate a language model with CTC decoding
for Wav2Vec2. Not absolutely required, but we used it for better decoding with
custom LM.
●​ Jiwer: For calculating WER (Word Error Rate) and CER (Character Error
Rate) from reference and hypothesis transcripts. This simplifies evaluation of
ASR outputs.
●​ Torchaudio: (Optional) For some audio loading and augmentations. We
mainly used librosa, but torchaudio was installed as it integrates with PyTorch
nicely and was used in one experiment for spectrogram extraction in the CNN
classifier approach.
●​ FFmpeg: A command-line tool for audio conversion. In Colab, ffmpeg was
pre-installed, but if running elsewhere, one might need it to convert between
audio formats (e.g., some YouTube downloaded audio might be in .m4a and we
convert to .wav).
●​ OpenAI Whisper (if used): If we choose to fine-tune Whisper, we’d
require openai-whisper or use it via Transformers. We ended up not extensively
fine-tuning Whisper due to resource constraints, but we did use the pretrained
Whisper model for some baseline transcriptions. Installing whisper via pip
(OpenAI’s package) or using it through transformers is needed if one wants to
replicate those baseline checks.
●​ Other Python packages: pandas (for data frames and CSV handling of
metadata), matplotlib or seaborn (for plotting results, e.g., accuracy trends or
confusion matrix), and pydub (for audio segment handling) are also used. The
exact versions are not critical; any recent version should suffice.
All the above software are open-source. We list them in the README with
installation instructions. If using Google Colab, many of these (numpy, torch, etc.)
come pre-installed and just need to pip install a few like librosa or jiwer as needed.
4.4 Security Considerations
Security in the context of our project pertains mostly to data privacy and ethical
use, since we are not developing a networked application or handling user
credentials. Nonetheless, a few considerations are:
●​ Dataset Licensing and Consent: The speech data used from LDC-IL and
Vaani is properly licensed for research. We obtained the LDC-IL corpus
through a formal request ensuring we adhere to its usage terms
(non-commercial, research-only). For Vaani and other open data, they are
 CC-BY-4.0 which means we must credit the source in any publications and not
impose additional restrictions. The small amount of YouTube data we used was
either under Creative Commons or we got creator permission. This ensures we
do not violate copyright or privacy of speakers.
●​ Personal Identifiable Information (PII): Speech data can contain personal
information (names, etc.). We did minimal manual annotation and our
transcripts/data will not be made public with any sensitive info. Also, when
reporting examples in the project report, we anonymize content if needed (e.g.,
not revealing a real person’s full name from the recordings).
4.5 Development Tools
During the development of this project, several tools were used to facilitate coding,
debugging, and collaboration:
●​ Integrated Development Environment (IDE): As mentioned, Visual Studio
Code (VS Code) was used locally for writing Python scripts, especially for the
classifier part and utility scripts to preprocess data. VS Code’s remote
extensions allowed editing files on the Colab instance or VM seamlessly. Its
features like IntelliSense and debugging were helpful for local tests.
●​ Git for Version Control: Throughout development, code was
version-controlled. Commits were made for major changes, and branches were
used for experimenting with alternatives (like a whisper-finetune branch
separate from wav2vec-main branch). This provided a safety net to revert to
stable code if a new experiment broke something.
●​ Issue Tracking: We utilized GitHub issues (in a private repo) to note down
bugs or tasks. For example, an issue was logged when we encountered memory
errors fine-tuning the model, and that led to trying gradient checkpointing as a
fix.
●​ Collaboration Tools: Google Drive was used to share intermediate results and
store large files. Also, a Slack/WhatsApp group among team members for
quick discussions was in place. While these aren’t “development tools” in the
 coding sense, they aided the development process by enabling quick exchange
of ideas and debugging help.
●​ Testing Frameworks: We wrote some simple tests (not full unit tests, but
assert statements and sample runs) to verify components. For instance, after
coding the feature extraction, we ran it on a known audio and manually
verified the MFCC values range. During ASR model training, we used small
sanity-check datasets to ensure the training loop was correct (overfitting a tiny
sample to see if loss goes to zero).
●​ Profiling/Monitoring: We occasionally used tools like the Colab’s GPU
utilization monitor, and Python’s cProfile for performance if needed (for
example, making sure feature extraction wasn’t a bottleneck by vectorizing
operations).
These development tools and practices ensured a smooth workflow. The result is a
codebase that is organized (split into modules for classifier, data processing, model
training, etc.) and maintainable. We emphasize that anyone reusing our code
should also use version control and notebooks for the interactive parts, as those
were critical in our development lifecycle.
4.6 Technology Used
This section highlights the key technologies and algorithms at the heart of our
project, linking them to the project’s objectives:
●​ Machine Learning Algorithms: For dialect classification, we used traditional
ML classifiers (SGD, SVM, Ridge). These are based on well-established
algorithms:
●​ SGD Classifier: Uses stochastic gradient descent to fit a linear model
(essentially logistic regression in our case) for multi-class classification.
●​ Support Vector Machine: Uses the kernel trick (we tried RBF kernel) to find a
hyperplane separating classes in a transformed feature space, maximizing
margin.
●​ Ridge Classifier: A linear least squares classifier with L2 regularization, which
 in practice acts similarly to logistic regression for classification tasks. ​
These were chosen for their simplicity and effectiveness on small-to-medium
feature sets, as evidenced by prior studies.
●​ Digital Signal Processing (DSP): We extensively used DSP techniques for
feature extraction from audio. Calculating MFCCs involves taking Fourier
transforms, mapping to Mel scale, and using the discrete cosine transform – all
classic DSP operations. We also used filtering (implicitly when computing
spectral features) and segmentation of audio signals. Understanding of Nyquist
rate (we downsampled to 16 kHz to capture up to 8 kHz frequency content,
which is sufficient for speech) was applied.
●​ Deep Learning (Neural Networks): The ASR models (Whisper, Wav2Vec2)
are deep neural networks. Specifically:
●​ Wav2Vec2.0 XLS-R: This is a Transformer encoder model pre-trained with a
contrastive loss on audio. Fine-tuning it for ASR means adding a linear output
layer over characters and training with CTC (which itself is a sequence
alignment algorithm using dynamic programming).
●​ Whisper: A sequence-to-sequence transformer with an encoder decoding to
text. While we didn’t fully train Whisper due to resource constraints,
understanding its architecture and how to fine-tune via the Transformers library
was part of the technology used.
●​ 1D-CNN (Exploration): We attempted a Convolutional Neural Network on
log-Mel spectrogram for dialect ID. This involved convolution operations over
time frames to detect patterns. Although the classical approach sufficed, this
was a relevant tech we dabbled in.
●​ Language Modeling: Using n-gram models (unigram, bigram) for biasing the
decoder is an NLP technique. We utilized kenlm (via pyctcdecode integration)
which is a common toolkit to create n-gram language models from text. This is
a simpler form of language modeling compared to neural LMs, but effective
for our use-case.
●​ Data Science and Analytics: We used pandas for analyzing dataset
composition, e.g., calculating how many hours from each dialect, how many
unique words, etc. Visualization of results (matplotlib to plot confusion matrix
or bar charts of WER) is another tech aspect – helps interpret model
performance.
●​ Cloud Computing: Though not a technology in the algorithmic sense, using
cloud GPUs (Colab’s Tesla T4 with CUDA cores) is a part of the tech stack. It
enabled training the models that would be impractically slow on CPU.
●​ Parallelization and Performance: We took advantage of vectorized
operations in numpy and the fact that training on GPU inherently uses parallel
matrix ops. When processing large audio files for feature extraction, we used
batch processing (for instance, computing features for multiple files in parallel
where possible).
●​ APIs and Libraries: The Transformers library is a prime example – it
abstracts away a lot of complexity of model building and provides API access
to cutting-edge models. Learning to use it (like properly formatting the dataset,
choosing correct learning rate schedulers, etc.) was a technical endeavor. The
library uses PyTorch under the hood and we tapped into its capabilities like
mixed precision training (fp16) to speed up training on limited GPU memory.
By harnessing these technologies, we were able to implement a system that is
modern and efficient. The interplay of classical ML and advanced deep learning in
one project is a notable aspect – we weren’t confined to one paradigm but chose
what was best for each component. In essence, the project stands as an integration
of AI (machine learning + deep learning) techniques with software
engineering to solve a complex language problem.
 CHAPTER 5
SYSTEM DESIGN
5.1 Introduction
This chapter presents the system design of our dialect-specific speech recognition
system for the Marathi language. The design is documented through different
modeling techniques, namely the Data Flow Diagram (DFD), Architectural
Design, and Activity Diagram. These representations help us understand the flow
of data, the structural organization of components, and the step-by-step activities
performed by the system.

5.2 Data Flow Diagram (DFD)

Figure 5.2 is the Data Flow Diagram (DFD) for our system, which shows how data
moves through different processes and data stores within the system. We describe a
high-level DFD (between Level 0 and Level 1), focusing on the main processes.
Entities and Processes
●​ External Entity: User, who provides the Speech Audio Input.
●​ Process P1: Dialect Classification – Takes audio input and produces a Dialect
Label using the Dialect Classifier.
●​ Process P2: ASR Transcription – Takes both the Speech Audio and Dialect
Label as inputs, then generates the Text Transcript.
Data Stores
●​ D1: Dialect Models Database – Contains trained dialect classification models.
P1 consults this store for classification.
●​ D2: ASR Models Repository – Contains four fine-tuned ASR models (Puneri,
Varhadi, Khandeshi, Malvani). P2 queries this store based on the dialect label.
Data Flow Steps
1.​ The user sends audio input to P1.
2.​ P1 uses D1 to classify dialects and outputs a Dialect Label.
3.​ P2 receives the Dialect Label and the original Speech Audio.
4.​ P2 queries D2 to load the correct ASR model.
5.​ The selected ASR model generates the Text Transcript.
6.​ Transcript is returned to User.
The diagram (Figure 5.2) shows the pivotal role of the Dialect Label, which
connects the classification stage with the transcription stage.

Figure. 5.1
5.3 Architectural Design
The architectural design shows the structural overview of the system and how
components interact.
Key Components
1.​ User Interface (UI) – Accepts user audio input and displays output transcript.
In our test setup, this was a script/notebook, but in deployment, it could be a
web or mobile app.
2.​ Dialect Classifier Module – Includes:
o​ Feature Extractor (e.g., MFCC computation).
o​ Classification Model (SGD/Ridge regression).
 o​ Provides function predict_dialect(audio) returning the dialect label.
3.​ ASR Module(s) – Separate fine-tuned models for each dialect: Puneri,
Varhadi, Khandeshi, and Malvani. Each includes an acoustic model and
decoder with a language model.
4.​ Controller/Router – Orchestrates the workflow: takes audio from UI, uses the
classifier, selects ASR module, and produces the transcript.
5.​ Model Storage – Stores dialect classifier and ASR models, which are loaded at
system startup or on demand.
6.​ Output Handler – Returns final transcript to user (console, UI, or JSON API).
Interactions
●​ User → Controller → Dialect Classifier → Controller → ASR Model →
Controller → Output → User.
This design ensures modularity, scalability, and maintainability. New dialects can
be added by training additional ASR models and updating routing logic.​

Figure 5.2
5.4 Activity Diagram
The activity diagram illustrates the workflow of transcription.

Steps in Activity Flow

1.​ Start – User initiates transcription by providing audio.

2.​ Capture Audio Input – Audio received.
3.​ Perform Dialect Classification – Features extracted and classifier applied.
4.​ Obtain Dialect Label – Store result as dialect_label for further ASR model
selection process..
5.​ Decision: Confidence Sufficient?
o​ If high confidence → proceed with classification output.
o​ If low confidence → default to standard ASR model (optional).
6.​ Select ASR Model – Based on dialect label (Puneri, Varhadi, Khandeshi,
Malvani).
7.​ Load/Initialize ASR Model – Load if not already in memory.
8.​ Transcribe Audio – Forward audio to selected ASR model for converting
speech signals into text.
9.​ Generate Transcript Output – Text produced (optionally post-processed).
10.​Display/Return Transcript to User – Output shown to user through interface
or application display window.
11.​End – Process completes after successful generation of transcription result.
Error handling is considered in case of invalid audio or system failures, leading to
error messages.
Figure 5.3
 CHAPTER 6

CONCLUSION AND FUTURE ENHANCEMENT

We have successfully designed and developed a dialect-specific speech recognition

system for the Marathi language, demonstrating improvements in accuracy for
dialectal speech through a combination of classification and model adaptation. In this
concluding chapter, we summarize the outcomes of the project, reflect on its
significance, and discuss potential future enhancements that can build upon this work.

6.1 Conclusion

Project Summary: The goal was to bridge the gap in ASR performance for
Marathi’s dialects by introducing dialect awareness into the pipeline. We approached
this by first building a dialect identification classifier and then fine-tuning separate
ASR models for four dialect variants (Puneri, Varhadi, Khandeshi, Malvani).
Throughout the project, we leveraged open data and models – using resources like the
LDC-IL [Link] and Project Vaani for data, and powerful pretrained
models like Wav2Vec2 and Whisper as starting points. This strategy proved effective
and cost-efficient.

Results: Our dialect classifier achieved an accuracy of around 84%, which is on par
with or slightly better than prior published results in Marathi dialect identification.
This level of accuracy means the classifier can reliably tag most speech segments
with the correct dialect, enabling the correct downstream ASR model to be chosen.
On the ASR side, each dialect-specialized model outperformed a baseline single
model on its respective dialect. For instance, on a Varhadi test set, the fine-tuned
Varhadi ASR model reduced WER by approximately 25% relative (e.g., from 30%
down to 22% WER) compared to a generic model. Similar trends were observed for
Malvani and Khandeshi, especially on sentences that contained local idioms or
unique words – the generic model often produced gibberish for those, whereas the
adapted model got them correct. Puneri (standard) being the well-covered dialect
didn’t see as dramatic a gain (its baseline was already good), but importantly, the
specialized model did not degrade performance on standard speech while it gained
robustness to mild accent variations.

Significance: These results confirm our hypothesis that dialect adaptation can
significantly improve ASR accuracy for underrepresented varieties. Even with a
relatively moderate amount of fine-tuning data (10-50 hours per dialect), the
improvement was tangible. This has broader implications: similar techniques could
be applied to other languages with dialect diversity (indeed, we referenced studies in
Arabic, Chinese, etc., that align with our findings). For Marathi, our project is one of
the first comprehensive implementations of dialect-specific ASR, and it contributes to
making voice technology more inclusive for the state’s populace. It shows that one
can achieve this without prohibitive resources, using open-source tools and
community datasets, which is encouraging for researchers and developers in
low-resource language domains.

Challenges: Along the way, we encountered challenges such as limited data for
certain dialects (e.g., Khandeshi), which we overcame by creative data augmentation
and careful training (to avoid overfitting). The classifier occasionally struggles with
very short or very clean (formal) inputs – for example, one-word audio (“हाँ” meaning
"yes") has no dialect markers and could be misclassified randomly. However, in
practical use, longer utterances provide enough cues. Our ASR models, while
improved for dialect speech, still have room for growth – some errors persist,
especially in noisy audio or when speakers code-mix languages. We also noticed that
having the correct dialect model doesn’t solve everything: if an audio has a lot of
English words (common in urban speech), the model can still err unless it has seen
such a mix. We mitigated this slightly via language model biasing, but it’s an area to
refine.
Objective Fulfillment: Ultimately, the project met its stated objectives (Section 3.1
goals). We have a working prototype pipeline where a given Marathi audio is
automatically classified and transcribed with enhanced accuracy. The system has
been documented and the results validate the approach. Thus, we conclude that the
project has been successful in demonstrating a viable solution to dialect variability in
speech recognition for Marathi.

6.2 Future Enhancement

While the project achieved its core aims, it also opened up several avenues for further
improvement and exploration:

●​ Scaling to More Dialects/Languages: One immediate extension is to include

more dialects of Marathi or even other languages. Maharashtra itself has other
dialects and related languages (e.g., Wahadi, Dangi) which we did not cover.
Our architecture can accommodate additional dialect models easily. Moreover,
the paradigm of dialect classification + specialized ASR could be applied to
languages like Hindi (which has distinct dialects like Bhojpuri, Braj, etc.) or
any multilingual region. Future work could generalize the system into a
multi-language, multi-dialect framework.
●​ Unified Model with Dialect Codes: An interesting research direction is to
train a single ASR model that takes dialect as an input parameter or learns
dialect embeddings. We touched on multi-task learning in our experiments.
Future work could develop a model that jointly learns to transcribe and identify
dialects, thereby internally adjusting its weights to dialect features on the fly.
This could reduce the maintenance of multiple models and possibly generalize
better (especially for dialects with little data, since the model can share
knowledge among dialects).
●​ Real-time and Deployment: Our current system works offline on recorded
clips. Future enhancements would involve optimizing for real-time streaming
transcription. This could mean using smaller model versions (like Whisper
 small or tiny) and quantizing them to run on mobile devices. Deploying the
system as a web service or mobile app could bring it to end users. For instance,
a mobile app where a farmer speaks in Varhadi and gets text could be piloted.
This requires integrating our back-end with a front-end and ensuring the
pipeline is fast enough for practical use.
●​ Improved Dialect Classification: The classifier could be enhanced by using
neural network approaches (e.g., an LSTM or CNN over audio frames) which
might push accuracy higher than 84%. Also, incorporating speaker adaptation
might help – sometimes dialect identification can be aided by knowing the
speaker's region (if meta-data is available). While not usually known in
advance, in some contexts it might. Another enhancement is to use longer
context: our classifier works per short clip; if we had continuous speech or a
conversation, we could aggregate evidence over time to improve dialect ID
reliability.
●​ Handling Code-Mixing and Language Detection: A practical future feature
is to integrate automatic language identification for code-mixed speech. For
example, if a sentence has Marathi and English mixed, a robust system could
identify segments to route to different models or have a combined model.
Currently, our models transcribe English as well (since Whisper/Wav2Vec have
some capability), but not always perfectly. A specialized approach could
improve that (like using a bilingual model).
●​ Error Correction and Feedback Loop: We could implement a
post-processing module that detects likely errors (e.g., out-of-vocabulary words
or acoustically confused pairs) and auto-corrects them using context or user
feedback. For instance, if the classifier was wrong and thus the ASR is poor,
the system might detect high WER and decide to re-run with a different model
or prompt the user “Did you mean to speak in Malvani?” etc. Such interactive
refinement can make the system more robust.
 ●​ Incorporating Prosody and Tone: Dialects sometimes differ in prosody
(intonation). Future research could see if adding prosodic features (like
intonation patterns) helps either the classification or even the ASR (through
richer acoustic modeling). This ventures into speech understanding territory
but could be interesting to explore.
●​ Enlarging Training Data via Simulation: We relied on real data for each
dialect. A forward-looking idea is to use voice conversion or synthesis to
generate artificial training data – for example, convert standard Marathi speech
into a synthesized Varhadi accent using a voice conversion model, and then use
that to train the ASR. This could massively expand available data. Similarly,
using a text-to-speech (TTS) system for Marathi dialects could generate more
audio for training or testing.
●​ User Personalization: Within dialects, individuals have idiosyncrasies. Future
systems might learn a user’s particular accent over time. Our architecture could
allow fine-tuning the models further on a specific speaker’s data (on-device
learning perhaps). This is more of a personalized model approach and could
yield near-perfect transcription for frequent users, though it raises privacy
considerations if not done carefully.

In conclusion, this project lays a solid foundation for dialect-aware speech

recognition in Marathi. There is ample scope to broaden and deepen the work. By
pursuing the mentioned enhancements, we can move closer to a production-grade
system that not only handles dialect diversity but continuously learns and adapts to
the rich tapestry of spoken language. The ultimate vision is a future where language
technology understands every speaker – not just the standardized form of the
language – thus truly democratizing AI for all linguistic communities.
 APPENDICES

APPENDIX – A

Sample Dataset Details

This appendix provides details of the datasets used for training and evaluation of the
dialect-aware Marathi speech recognition system.

A.1 Dataset Sources

Dialect Labels
Dataset Name Hours Source
Available

LDC-IL Marathi Raw Speech LDC-IL, CIIL

~89 hrs Yes
Corpus Mysore

Project Vaani (Maharashtra ~1030

Partial IISc Bangalore
subset) hrs

OpenSLR-64 Marathi Corpus ~712 MB No (simulated split) OpenSLR

Custom Slang Dataset

~8 hrs Yes Self-collected
(Recorded)

A.2 Dialect Distribution Used

Dialect Training Hours Validation Hours Testing Hours

Puneri 50 hrs 5 hrs 5 hrs

Varhadi 40 hrs 4 hrs 4 hrs

Khandeshi 15 hrs 2 hrs 2 hrs

Malvani 20 hrs 3 hrs 3 hrs

APPENDIX – B

System Environment and Tools

B.1 Hardware Configuration

Component Specification

Processor Intel i5/i7 or equivalent

RAM Minimum 8 GB (Recommended 16 GB)

GPU NVIDIA Tesla T4 (Colab) / CUDA GPU

Storage Minimum 20 GB free space

B.2 Software Requirements

Software Version

Python 3.8+

PyTorch ≥ 2.0

SpeechBrain Latest

Whisper OpenAI Whisper

Librosa 0.9+

HuggingFace Transformers ≥ 4.30

APPENDIX – C

Dialect Classifier Architecture (ECAPA-TDNN)

The dialect identification module uses the ECAPA-TDNN model, which is a

state-of-the-art architecture originally developed for speaker verification.
Key Characteristics:

●​ Input: 80-dimensional Mel Spectrogram

●​ Embedding Dimension: 192
●​ Output: 4-class dialect softmax classifier
●​ Framework: SpeechBrain

Dialect Prediction Output:

[Dialect = \arg\max_{d} P(d|x)]

where (x) is the speech signal and (d) represents dialect classes.

APPENDIX – D

Sample Code Snippets

D.1 Dialect Prediction Function

from dialect_identification.inference import

predict_dialect

dialect, confidence = predict_dialect("[Link]")

print("Dialect:", dialect)
print("Confidence:", confidence)

D.2 Dialect-Aware Whisper Transcription

from asr.dialect_aware_decoder import

DialectAwareDecoder

decoder = DialectAwareDecoder(model_name="small")

result = [Link]("[Link]")

print("Detected Dialect:", result["dialect"])

print("Transcription:", result["text"])
APPENDIX – E

Sample Output Screenshots

E.1 Dialect Detection Output

Detected Dialect: Khandeshi

Confidence Score: 94.84%

E.2 Whisper Transcription Output

Input Audio Dialect: Vidarbha

Transcription:
"माझा गावातली भाषा वेगळी आहे ..."

APPENDIX – F

Evaluation Metrics

F.1 Word Error Rate (WER)

[WER = \frac{S + D + I}{N}]

Where:

●​ (S) = Substitutions
●​ (D) = Deletions
●​ (I) = Insertions
●​ (N) = Total words in reference transcript

F.2 Dialect Classification Accuracy

[Accuracy = \frac{Correct\ Predictions}{Total\ Samples}]

APPENDIX – G

Training Log Sample

Epoch 1: Train Loss: 1.38 | Val Accuracy: 31%

Epoch 10: Train Loss: 0.32 | Val Accuracy: 85%
Epoch 20: Train Loss: 0.04 | Val Accuracy: 100%
Model Saved Successfully!

APPENDIX – H

Project Repository Structure

efficient-marathi-dialect-aware-asr/
│
├── data/
├── dialect_identification/
├── asr/
├── models/
├── experiments/
└── [Link]

APPENDIX – I

Detailed Mathematical Background of Speech Processing

I.1 Speech Signal Representation

A speech signal can be represented as a time-domain signal:

[x(t)]

where:

* ( x(t) ) = Speech waveform

* ( t ) = Time index

After digitization, the signal becomes:

[x[n] = x(nT_s)]

where:

* ( T_s = \frac{1}{f_s} )
* ( f_s ) = Sampling frequency (16 kHz in our system)
Table I.1

Dataset Sources Used in the Project

S. Dataset Name Organization Total Dialect License

No Duration Labels
1 Marathi Raw LDC-IL, CIIL ~89 Hours Yes Research
Speech Corpus Mysore License
2 Project Vaani IISc Bangalore ~1030 Partial CC BY 4.0
(Maharashtra) Hours
3 OpenSLR-64 OpenSLR ~712 MB No CC BY-SA
Marathi 4.0
4 Custom Recorded Self-Collected ~8 Hours Yes Internal Use
Dataset

I.2 Short-Time Fourier Transform (STFT)

To analyze speech in frequency domain:

[X(k, m) = \sum_{n=0}^{N-1} x[n] w[n - m] e^{-j 2\pi kn/N}]

Where:

* ( w[n] ) = Window function

* ( N ) = Frame length
* ( m ) = Frame index

Table I.2

Dialect-wise Data Distribution

Dialect Training Validation Testing Total

(Hours) (Hours) (Hours) Hours
Puneri 50 5 5 60
Varhadi 40 4 4 48
Khandes 15 2 2 19
hi
Malvani 20 3 3 26
Total 125 14 14 153

I.3 Mel Scale Conversion

Human perception of sound is logarithmic.

Mel scale is computed as:

[Mel(f) = 2595 \log_{10}(1 + \frac{f}{700})]

This transformation is used to compute MFCC features.

I.4 MFCC Computation Steps

1. Pre-emphasis
2. Framing
3. Windowing
4. FFT
5. Mel filterbank
6. Log compression
7. DCT

Final MFCC vector:

[MFCC = DCT(\log(MelFilterBank(FFT(x))))]

APPENDIX – J

ECAPA-TDNN Architecture Explanation

J.1 TDNN Concept

Time Delay Neural Networks (TDNN) capture temporal dependencies in speech

using 1D convolution across time.
Table J.1

Dialect Classifier Architecture Configuration (ECAPA-TDNN)

Parameter Specification
Input Feature 80-Dimensional Mel
Spectrogram
Embedding 192
Dimension
Hidden Layers TDNN Blocks with SE
Attention
Dropout 0.3
Output Classes 4 Dialects
Loss Function Cross Entropy
Optimizer Adam
Learning Rate 0.0003
Training Epochs 20

J.2 ECAPA Enhancements

ECAPA-TDNN includes:

* Channel Attention
* Multi-layer feature aggregation
* Residual connections
* Squeeze-and-Excitation blocks

Table J.2

Whisper Model Variants Comparison

 Model Parameter Relative Accuracy Recommended
Variant s Speed Level Usage
Tiny 39 Million Very Fast Low Testing
Base 74 Million Fast Moderate Development
Small 244 Moderate High Balanced Usage
Million
Medium 769 Slow Very High Research
Million Evaluation
Large 1550 Very Slow Highest Final
Million Benchmarking

J.3 Embedding Extraction

Speech segment → 192-dimensional embedding vector:

[e = f_{\theta}(x)]

Where:

* ( e \in \mathbb{R}^{192} )
* ( f_{\theta} ) = Neural network with parameters ( \theta )

J.4 Softmax Classification

[P(d|x) = \frac{e^{W_d^T e}}{\sum_{i=1}^{4} e^{W_i^T e}}]

Where:

* ( d ) = Dialect class

APPENDIX – K

Whisper Model Architecture Details

K.1 Transformer Encoder-Decoder

Whisper consists of:

* Audio Encoder (Transformer)
* Text Decoder (Autoregressive Transformer)

K.2 Attention Mechanism

Self-attention equation:

[Attention(Q,K,V) = softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V]

Where:

* ( Q ) = Query
* ( K ) = Key
* ( V ) = Value

Table K.2

Hyperparameter Configuration for Whisper Fine-Tuning

Hyperparameter Value
Base Model Whisper Small
Learning Rate 1e-5
Beam Width 5
Gradient 2
Accumulation
Warm-up Steps 500
Loss Function Cross Entropy
Framework PyTorch +
HuggingFace

K.3 CTC vs Seq2Seq

We compared:

Whisper was chosen due to better contextual decoding.

APPENDIX – L

Hyperparameter Configuration

Table L.1

Dialect Classification Performance

Dialect Precisio Reca F1-Sco Average

n ll re Confidence
Puneri 1.00 1.00 1.00 78.32%
Varhadi 1.00 1.00 1.00 91.48%
Khandes 1.00 1.00 1.00 94.84%
hi
Malvani 1.00 1.00 1.00 99.94%
Overall 1.00 1.00 1.00 91.15%

Table L.2

Word Error Rate Comparison

Model Dialect WER Relative

Awareness (%) Improvement
Baseline Whisper No 30.4 -
Fine-Tuned Partial 24.8 18%
Whisper
Proposed System Yes 21.3 30%
Table L.3

Confusion Matrix (Dialect Classifier)

Actual \ Pune Varha Khandes Malva

Predicted ri di hi ni
Puneri 48 2 0 0
Varhadi 1 49 0 0
Khandeshi 0 1 49 0
Malvani 0 0 0 50

APPENDIX – M

Table M.1

Computational Complexity Analysis

Component Complexity Description

Feature Extraction O(N log N) FFT
computation
Dialect O(N × d) Embedding
Classification generation
Whisper Encoding O(T² × L) Transformer
attention
Beam Search O(B × T) Beam width B
Decoding

Observation:

* Minor confusion between Pune and Vidarbha.

* Distinct separation for Khandeshi and Konkani.
APPENDIX – N

Word Error Rate Breakdown

WER Components:

[WER = \frac{S + D + I}{N}]

Example:

Reference:
"माझा गाव खप
ू सद
ंु र आहे "

Hypothesis:
"माझ गाव खप ु सन्
ु दर आहे "

* Substitution = 1
* Deletion = 1
* Insertion = 0
* Total words = 5

WER = 2/5 = 40%

APPENDIX – O

Time Complexity Analysis

Dialect Classifier Complexity:

[O(N \cdot d)]

Where:

* N = number of frames
* d = embedding dimension

Whisper Inference Complexity:

[O(T^2 \cdot L)]

Where:
* T = sequence length
* L = number of layers

APPENDIX – P

Ethical Considerations

* Avoid dialect bias

* Prevent discrimination
* Ensure inclusive AI
* Maintain dataset consent compliance
* Protect speaker privacy

APPENDIX – Q

Extended Future Research Directions

1. Multilingual dialect transfer learning

2. Zero-shot dialect recognition
3. Accent embedding fusion
4. Cross-dialect data augmentation
5. Edge device optimization
6. Low-resource training optimization
7. Large-scale Marathi benchmark creation
​
​
REFERENCES

1.​ Ramamoorthy, L. et al. (2019). Marathi Raw Speech Corpus. Central Institute
of Indian Languages (LDC-IL), Mysore. (Contains 89 hours of Marathi speech
across major dialect regions) [Link]
2.​ Project Vaani (2023). Vaani Dataset Release. ARTPARK & IISc Bangalore.
(Accessed via [Link]). (Open-source multi-dialect speech dataset;
includes ~1030h Marathi, 15.7h Malvani, 5.7h Khandeshi)
3.​ OpenAI (2022). Whisper Model and Paper – “Robust Speech Recognition via
Large-Scale Weak Supervision”. (OpenAI Blog & arXiv:2212.04356).
(Introduced the Whisper ASR model trained on 680k hours, used as base
model in our project)
4.​ Baevski, A. et al. (2020). “wav2vec 2.0: A Framework for Self-Supervised
Learning of Speech Representations.” Advances in NeurIPS 2020. (Introduced
Wav2Vec2, the architecture behind the XLS-R model we fine-tuned)
5.​ Chittaragi, N. B. et al. (2019). “Automatic text-independent Kannada dialect
identification system.” In Information Systems Design and Intelligent
Applications, pp. 79-87. Springer. (Example of dialect ID in another Indian
language, for reference in methodology)
6.​ Bansod, N. S. et al. (2014). “Speaker Recognition using Marathi (Varhadi)
Language.” 2014 Intl. Conference on Intelligent Computing Applications,
IEEE, pp. 421-425. (One of the early works dealing with Varhadi speech,
though for speaker recognition)
7.​ LDC-IL (2019). “LDC-IL Raw Speech Corpora: An Overview”, in Linguistic
Resources for AI/NLP in Indian Languages, CIIL, Mysore, pp. 160-174.
(Overview of all LDC-IL speech corpora including Marathi) [Link]
8.​ AI4Bharat (2021). Kathbath and Vistaar Datasets. (Online resources at
[Link] and HuggingFace). (Crowdsourced Marathi speech ~166h
and multi-domain dataset used for baseline model training in some contexts).