UNIT V

SPEECH RECOGNITION
Automatic speech recognition – architecture – applying hidden markov model – feature
extraction: mfcc vectors – computing acoustic likelihoods – search and decoding –
embedded training – multipass decoding: n-best lists and lattices- a*(‘stack’) decoding –
context-dependent acoustic models: triphones – discriminative training – speech
recognition by humans

Automatic Speech Recognition (ASR)

1.​ Early goal of language processing​

○​ Understanding spoken language or converting speech into text was one of the first
aims of computer language processing.​

○​ Speech processing research began even before computers were invented.​

2.​ First speech-recognition toy (1920s)​

○​ Radio Rex (1920s) – a toy dog that came out of its house when it heard a 500 Hz
sound.

○​ 500 Hz matches the first formant of the vowel sound [eh] in “Rex”.​

○​ So, when someone called “Rex,” the dog appeared.​

3.​ Modern ASR goal​

○​ To map speech waveforms (sound signals) into correct written words.​

○​ Example: converting speech into text → “It’s time for lunch!”.

​
4.​ Current limitations​

○​ Perfect ASR for any speaker in any environment is not yet solved.​

○​ But ASR is good enough for many real-world uses.​

5.​ Practical applications of ASR​

○​ Voice interface for appliances, smartphones, assistants (e.g., Siri, Alexa,

chatbots).​

○​ Transcription of audio/video for captions (movies, meetings, lectures).​

○​ Legal and dictation tasks (lawyers, doctors, writers dictating text).​

○​ Assistive technology for people with disabilities (e.g., blind or unable to type).​

○​ Example: Milton dictated Paradise Lost, Henry James dictated novels.​

6.​ ASR system process​

○​ Acoustic feature extraction → turns raw sound into useful features.​

○​ Neural architectures → Convolutional Neural Networks (CNNs) often used

first.​

7.​ Main families of ASR methods​

​
(a) Encoder–Decoder approaches​

○​ Listen, Attend and Spell (LAS): early attention-based model.​

○​ Whisper (OpenAI, 2023): advanced encoder–decoder system; can also do

translation.​

○​ OWSM (2023): open-source Whisper-style model.​

8.​ (b) Self-Supervised Learning (SSL) approaches​

 ○​ Models like Wav2Vec 2.0 and HuBERT.​

○​ Learn abstract speech representations without labeled data.​

○​ Use CTC (Connectionist Temporal Classification) loss for decoding speech

into text.​

9.​ Evaluation metric​

○​ Word Error Rate (WER): standard measure of ASR performance.​

○​ WER = (substitutions + insertions + deletions) ÷ total words.​

○​ Lower WER = better performance.​

The Automatic Speech Recognition Task

I. Dimensions of Variation in ASR

1.​ Vocabulary size​

○​ ASR systems may deal with small vocabularies (like “yes/no” or digits 0–9).
These are very easy because there are few possible words to choose from.​

○​ But for large vocabularies (60,000+ words, like in movies, lectures, or daily
conversations), the system has to consider many more possibilities, which makes
recognition much harder.​

2.​ Speaking style / context​

○​ When humans talk to machines (dictation, chatbots, assistants), they usually

speak slowly, clearly, and carefully → easier for ASR.​

○​ Read speech (like audiobooks) is also easier, since people pronounce clearly
when reading aloud.​

○​ But conversational speech between humans is difficult because people talk

faster, use slang, interrupt each other, and don’t always pronounce clearly.​
 3.​ Channel & noise​

○​ If speech is recorded in a quiet room with a close microphone, ASR works very
well.​

○​ But if recorded in a noisy place (e.g., street, car, dinner party) or with a distant
microphone, the signal quality drops, and ASR struggles.​

4.​ Speaker & accent differences​

○​ ASR systems are usually trained on standard dialects (like US English spoken
by adults).​

○​ If the speaker has a different dialect, accent, or is a child, the system finds it
harder to match the speech to what it learned.​

○​ Example: African American English or regional dialects often lead to higher

errors.​

2. Important Speech Corpora (Datasets)

Researchers use large speech corpora (recordings + transcripts) to train and test ASR. Here are
some key ones:

●​ LibriSpeech (2015)​

○​ Over 1000 hours of audiobook recordings (16 kHz).​

○​ Has two versions:​

■​ Clean: high-quality recordings, close to standard English.​

■​ Other: noisier or accented speech.​

○​ Widely used as a benchmark dataset.​

●​ Switchboard (1990s)​

○​ ~240 hours of telephone conversations between strangers.​

 ○​ Contains about 3 million words.​

○​ Special because it has detailed extra labeling (syntax, dialogue acts, phonetics,
prosody), making it useful for many types of research.​

●​ CALLHOME (1990s)​

○​ 120 conversations (~30 minutes each) between friends or family on the phone.​

○​ Speech is more casual and natural compared to Switchboard.​

●​ CHiME Challenge (various versions)​

○​ Focuses on noisy, real-world speech recognition.​

○​ Example: CHiME-6 has recordings of real dinner parties with 4 people talking at
once in noisy environments (kitchen, dining room, etc.).​

●​ AMI Meeting Corpus (2007)​

○​ 100 hours of group meetings (some real, some staged).​

○​ Includes transcripts and labels (like who is speaking).​

●​ CORAAL (2020)​

○​ Over 150 interviews with African American speakers.​

○​ Goal: to study African American English (AAE) and its variations.​

○​ Useful for testing ASR on non-standard dialects.​

●​ Mandarin corpora​

○​ HKUST: 1200+ telephone conversations in Mandarin (friends and strangers).​

○​ AISHELL-1: 170 hours of Mandarin read speech, mostly from speakers in

northern China.​
 ●​ Multilingual corpora​

○​ Common Voice (Mozilla, 2020):​

■​ 33,000+ hours of speech in 133 languages.​

■​ Crowd-sourced: people record sentences, and others check them.​

○​ FLEURS (2023):​

■​ 3,001 Wikipedia sentences translated into 101 languages.​

■​ Recordings in 102 languages (~12 hours each).​

■​ Useful for multilingual ASR and translation research.​

III. Word Error Rates (WER) in Different Settings

●​ English read speech (audiobooks, LibriSpeech clean): ~2% error.​

→ ASR works almost perfectly here.​

●​ Human conversations (Switchboard, CALLHOME, AMI meetings): 5.8–11% error.​

→ Still fairly good, but harder due to fast, natural speech.​

●​ African American English or other non-standard dialects: higher error rates.​

→ Because systems are usually trained on standard dialects.​

●​ Real noisy conversations (dinner parties, CHiME-6): up to 25.5% error.​

→ Overlapping speech + background noise make it very tough.​

●​ Mandarin read speech: lower errors.​

●​ Mandarin casual conversation: much higher errors.​

●​ Low-resource languages: highest errors (because little training data is available).​

Speech Recognition Architecture (Noisy Channel Model)

1.​ Task of Speech Recognition​

○​ Input: an acoustic waveform (spoken sound).​

○​ Output: a string of words (text).​

○​ Problem: The sound is often unclear or “noisy.”​

2.​ Noisy Channel Idea​

○​ Think of speech as a message sent through a noisy channel.​

○​ The sound we hear is a “noisy” version of the actual words.​

○​ Goal: Recover the original (true) words from the noisy version.​

3.​ How It Works​

○​ The channel adds noise, making words hard to recognize.​

○​ Build a model of the channel to figure out how it changed the words.​

○​ Use this model to guess and recover the most likely original sentence.​

4.​ Key Insight​

○​ If we know how the channel distorts sound, we can test possible sentences.​

○​ Each candidate sentence is passed through the model to see if it could create the
noisy input.​

○​ The best match is chosen as the recognized sentence.

Bayesian Inference in Speech Recognition

5.​ Challenges in Using the Noisy Channel​

○​ Problem 1: The spoken input never exactly matches any sentence in the model.​

○​ Solution: Use probability to measure the “best match.”​

6.​ Why Bayesian Inference?​

○​ Bayesian inference combines probability models to guess the most likely

sentence.​

○​ It has been used in many fields (e.g., handwriting recognition, authorship

detection).​

○​ In speech recognition, it helps calculate:​

 ■​ Probability of an observed sound sequence, given a possible sentence.​

7.​ Process​

○​ Search through possible sentences.​

○​ Choose the one with the highest probability of matching the sound.​

8.​ Efficiency Need​

○​ The number of possible English sentences is huge.​

○​ We need smart algorithms that search only plausible sentences, not all
possibilities.

9. Decoding or Search Problem:​

○​ In speech recognition, the main challenge is finding the best matching sentence
for a given sound input.​

○​ This is called the decoding or search problem.​

○​ Algorithms like Viterbi (learned earlier) are used to solve this.​

10. Why Search Is Hard:​

○​ The number of possible sentences is huge.​

○​ So, searching efficiently is a big part of the problem.​

11. What’s Coming Up:​

○​ The section will talk about the Bayesian (probabilistic) model used in speech
recognition.​
 ○​ It will also explain parts of a modern HMM-based ASR (Automatic Speech
Recognition) system.​

12. Main Goal of Speech Recognition:​

○​ The goal is to answer this question:​

​
“What is the most likely sentence out of all possible sentences in the language L,
given some acoustic input O?”​

13. How Sound Input is Represented:​

○​ The sound input is turned into a sequence of small slices (like one every 10
milliseconds).​

○​ Each slice is a set of numbers showing sound features (like energy or frequency).​

○​ These slices are treated like symbols or observations over time.​

○​ Capital letters represent full sequences of symbols, and lowercase letters

represent individual symbols.​

14. Acoustic Input as Observations:

●​ The sound input (like speech) is represented as a sequence of observations:​

O=o1, o2, o3, ….ot​

●​ These are small chunks of sound, like snapshots taken every few milliseconds.​

15. Sentence as Words:

●​ A sentence is treated as a sequence of words:​

​
W = w1, w2, w3,..., wn​
​
 ●​ This is a simplification. In real life, words may be grouped or need to be broken into
smaller parts (like prefixes or suffixes).​

Note on Simplifications:

●​ Speech recognition systems often treat sentences as sequences of individual

words.​

●​ This is a simplification and doesn't always reflect how language really works.​

●​ Too fine: Modeling individual words may miss important multi-word phrases
(e.g., "in front of").​

●​ Too coarse: It ignores internal word structure (morphology), like "run" vs.
"running".​

●​ Words are usually defined by spelling and are converted to lowercase.​

This causes problems:​

●​ Related words like "oak" and "oaks" are treated as different.​

●​ Homonyms like "can" (verb vs. noun) are treated as the same, despite different
meanings.
●​ The probabilistic implementation of our intuition is expressed as

●​ argmax means finding the value of (x) that makes a function (f(x)) largest.​

●​ We want to find the best sentence (W) that maximizes the probability (P(W|O)), i.e., the
most likely sentence given the sound (O).​

●​ To calculate (P(W|O)), we use Bayes’ Rule, which states

●​ This breaks the problem into modeling (P(O|W)), the likelihood of observing sound (O) if
sentence (W) was said (acoustic model), and (P(W)), the likelihood of sentence (W) itself
(language model).​

●​ This approach helps us find the optimal sentence efficiently.​

●​ The parts (P(W)) and (P(O|W)) are easier to calculate than (P(W|O)).​

●​ (P(W)) tells us how likely a sentence is, and it’s estimated using language models (like
n-grams).​

●​ (P(O|W)) tells us how likely the sound is, given a sentence, and it can be estimated too.​

●​ (P(O)) is the chance of hearing the sound itself, which is harder to figure out.​

●​ But we don’t need to calculate (P(O)) because it’s the same no matter which sentence we
check.​

●​ Since (P(O)) is constant for all sentences, it doesn’t change which sentence is the most
likely.​

●​ To find the best sentence from the sounds we hear, we look at two things for each
sentence.​
●​ First, how likely the sentence is to be said in the language (this is from the language
model).​

●​ Second, how well the sentence matches the sounds we heard (this is from the acoustic
model).​

●​ We multiply these two chances together for each possible sentence.​

●​ The sentence with the highest result is picked as the best match.

●​ The language model (LM) tells us how likely a sentence is in English.​

●​ It helps the system guess which sentences sound natural or common.​

●​ We calculate this likelihood using something called N-gram models.​

●​ An N-gram model looks at groups of words (like pairs or triples) to estimate how
probable a sentence is.​

●​ Basically, it predicts each word based on the words that came before it.

●​ This chapter explains how to use Hidden Markov Models (HMMs) from Chapter
6 to build an Acoustic Model (AM).​

●​ The Acoustic Model calculates the likelihood P(O∣W)P(O|W)P(O∣W), which

means how likely the sound OOO is if the sentence WWW was spoken.​
 ●​ Combining the Acoustic Model (AM) and Language Model (LM) allows us to
find the most probable sentence for a given sound using a search algorithm.​

●​ Figure 9.3 shows how an HMM speech recognizer works in three main steps
during recognition.​

●​ First, in the feature extraction (signal processing) stage, the speech sound is
split into short frames (10 to 20 milliseconds each).​

●​ Each frame is turned into a set of about 39 numbers called spectral features,
which capture the sound’s details like energy and changes over time.​

●​ Section 9.3 gives a brief overview of how these features are extracted from the
sound.

Acoustic Modeling & Decoding

1. What Happens in Acoustic Modeling (Phone Recognition Stage):

●​ The audio (speech) is broken into small time frames (e.g., every 10ms).​

●​ Each frame is represented by a feature vector (describes the sound at that moment).​

●​ The system tries to guess:​

“Which speech sound (phone or sub-phone) could have produced this feature
vector?”​

2. How the Guessing is Done:

●​ Gaussian Mixture Models (GMMs) are used to calculate how likely it is that a certain
phone (like "aa", "s", etc.) produced the current feature vector.​

●​ Each state in the Hidden Markov Model (HMM) corresponds to a phone (or part of it).​

●​ For each time frame, we get a probability vector:​

○​ It shows how likely each phone or sub-phone is for that specific frame.​

3. What Is the Output of This Stage?

 ●​ A sequence of probability vectors, one per time frame.​

●​ Each vector tells us how likely each phone is at that time.​

Decoding Stage – Turning Sounds into Words

4. What Happens in Decoding?

●​ We combine:​

1.​ Acoustic Model (AM) – the likelihoods from GMMs for each time frame.​

2.​ HMM Dictionary – tells us how words are pronounced (as sequences of phones).​

3.​ Language Model (LM) – helps decide which word sequences make sense (e.g.,
using N-grams).​

●​ Goal: Find the most likely sequence of words given the sound.​

5. How Are Words Modeled?

●​ Each word is like its own HMM.​

●​ Phones (or sub-phones) are states in that HMM.​

●​ The GMMs give the likelihood for each state (phone).​

6. How Do We Find the Best Word Sequence?

●​ We use the Viterbi algorithm (a fast search method) to find the most likely path (word
sequence).​

●​ To make it faster and smarter, ASR systems add techniques like:​

○​ Pruning (skip unlikely paths),​

○​ Fast-match (quick filtering),​

 ○​ Tree-structured lexicons (efficient word lookup).​

APPLYING THE HIDDEN MARKOV MODEL

HMM is characterized by the following components

1. What is an HMM in Speech Recognition?
●​ An HMM (Hidden Markov Model) is a mathematical tool to model speech.​

●​ It has:​

1.​ Hidden States → things we want to know but cannot directly observe (e.g.,
phones, words).​

2.​ Observations → the sound information we can measure (e.g., frequencies,

energy).​

3.​ Transitions → probabilities that say how likely it is to move from one state to
another.​

In speech recognition, the hidden states = speech sounds (phones), and the observations =
acoustic features from the audio signal.

2. Algorithms Used with HMMs

●​ Viterbi Algorithm (Decoding)​
 ○​ Helps us decode the speech → find the most likely sequence of phones/words
given the sound features.​

○​ Example: If the sound features look like “s-ih-k-s”, Viterbi finds that the best
matching word is “six.”​

●​ Baum-Welch Algorithm (Training)​

○​ Helps the HMM learn from data.​

○​ It adjusts probabilities so the model better maps sound features to the right
phones/words.​

○​ Needed because real speech is messy (different accents, speeds, background

noise).​

3. Earlier Examples of HMMs (to understand better)

●​ Parts of Speech​

○​ Hidden states = noun, verb, adjective, etc.​

○​ Observations = words in a sentence.​

○​ Task = Guess the POS sequence of a sentence.​

●​ Weather Example ​

○​ Hidden states = weather (sunny, rainy).​

○​ Observations = ice-cream sales.​

○​ Task = Guess weather sequence from ice-cream consumption.​

These examples show how HMMs connect hidden states (things we want to know) with
observations (things we can see).

4. Applying HMMs to Speech

 ●​ Hidden states = phones (basic sound units like /s/, /k/, /i/) or sometimes words.​

●​ Observations = short chunks of sound (acoustic features).​

●​ Task = figure out which phones/words match the observed sound sequence.​

5. Acoustic Features in Speech Recognition

●​ Speech is not processed as raw waveform (too complex).​

●​ Instead, it’s broken into frames of 10 milliseconds each.​

●​ For every frame, we calculate a feature vector with 39 values.​

○​ These values show how much energy is in different frequency bands.​

●​ Example:​

○​ 1 second of speech = 100 frames.​

○​ Each frame = 39 numbers → total 100 × 39 = 3900 values to represent 1 second

of speech.​

So, the input to an HMM is a long sequence of these feature vectors.

6. Small vs. Large Speech Tasks

●​ Small tasks (digits, yes/no recognition):​

○​ Hidden states can represent entire words.​

○​ Example: HMM for “yes”, another HMM for “no.”​

●​ Large tasks (continuous speech):​

○​ Hidden states represent phones.​

 ○​ Words are built by linking phone HMMs.​

○​ Example: The word “six” = /s/ + /ih/ + /k/ + /s/.​

7. Example: HMM for the Word “six”

●​ Suppose the word “six” has phones: /s/ – /ih/ – /k/ – /s/.​

●​ Each phone has its own small HMM (3–5 states, usually).​

●​ To model “six,” we join these phone HMMs one after the other:​

[s-phone HMM] → [ih-phone HMM] → [k-phone HMM] → [s-phone HMM]

●​ When sound is given, the system compares the acoustic features with these states.​

●​ Viterbi finds the best matching path through these states.​

8. Why This Matters

●​ HMMs let us handle:​

○​ Variation in speech (different speakers, speeds).​

○​ Uncertainty (many possible words for the same sound).​

○​ Complex sequences (sentences with many words).​

 1. How HMM Transitions Work in Speech vs. Other Tasks
●​ In earlier HMM tasks (like weather prediction or part-of-speech tagging):​

○​ Any state could, in principle, transition to any other state.​

○​ Example: In weather, you could go from “sunny” directly to “snowy” (though

unlikely).​

○​ In POS tagging, any tag (noun, verb, adjective) could follow another.​

●​ In speech recognition HMMs:​

○​ Transitions are much more restricted.​

○​ Why? → Because speech is sequential and time-ordered. Sounds flow in one

direction.​

2. Left-to-Right HMMs (Bakis Network)

●​ Speech HMMs typically use a left-to-right structure.​

●​ This means:​

○​ A state can stay in itself (self-loop) → allows stretching in time.​

○​ A state can move only to the next state.​

 ○​ No skipping backwards to earlier states.​

●​ This structure is called a Bakis network.​

●​ It matches how speech naturally unfolds → once you say part of a word, you don’t go
back.​

3. Why Self-Loops Are Important

●​ Phones (basic sounds) can vary a lot in length.​

●​ Example (from Switchboard corpus):​

○​ Phone [aa] can last from 7 ms (1 frame) to 387 ms (40 frames).​

○​ Phone [z] can last from 7 ms to 1.3 seconds (130 frames).​

●​ Since we don’t know exactly how long a phone will last, self-loops allow a state to be
repeated many times until enough frames are covered.​

●​ This makes the model flexible for fast or slow speech.​

4. When Phone-Level States Are Enough

●​ For very simple speech tasks (like digit recognition – “zero” to “nine”, or yes/no
recognition):​

○​ Using one HMM state per phone works fine.​

○​ Phones are short and words are limited → no need for deeper detail.​

5. Why We Need More Detail in Large Vocabulary Speech

Recognition (LVCSR)
●​ For real-world speech tasks (large vocabulary, continuous speech):​
 ○​ One phone can last over 1 second = 100+ frames.​

○​ But those 100 frames are not all the same.​

○​ The acoustic properties change within the phone.​

●​ Examples:​

○​ Stop consonants → have two parts:​

1.​ Closure (no sound, very low energy).​

2.​ Release burst (sudden spike of energy).​

○​ Diphthongs → vowels like [ay] where formants (F1, F2) move significantly
during the sound.​

●​ Because of these changes, just one state per phone is too simple.​

●​ Instead, phones are usually broken into sub-states (e.g., 3 states per phone) to model the
beginning, middle, and end portions of the phone.​

6. Example: Word “Ike” ([ay k])

●​ Fig. 9.5 shows the word “Ike” (two phones: [ay] and [k]).​

●​ In the spectrogram:​

○​ [ay] → spectral energy changes over time (formants move).​

○​ [k] → has a closure (silence) and then a burst (sudden energy).​

●​ This shows why one phone ≠ one state.​

●​ We need multiple states per phone to capture all these acoustic variations.​
1. Phones Are Not Uniform Over Time
●​ A phone (basic sound unit) is not the same from start to finish.​

●​ Example:​

○​ A vowel may start soft, get louder, then fade.​

○​ A consonant may begin with silence and then have a burst.​

●​ Because of this, using just one HMM state per phone is too simple.​

2. Splitting a Phone into Multiple States

 ●​ To capture this changing nature, we usually break each phone into 3 HMM states:​

1.​ Beginning state – captures how the phone starts.​

2.​ Middle state – captures the steady portion.​

3.​ End state – captures how the phone finishes.​

●​ This way, the model can describe the progression of a sound over time.​

3. Full Phone Model (5 States)

●​ Each phone model is called a Phone HMM or Phone Model.​

●​ It has 5 states total:​

○​ 2 non-emitting states (one at the start, one at the end).​

○​ 3 emitting states (beginning, middle, end).​

●​ Emitting states = states that actually output/represent sound features.​

●​ Non-emitting states = silent placeholders that help connect phones smoothly.​

4. Word Models Using Phone Models

To build an HMM for a word:​

○​ Take each phone in the word.​

○​ Replace it with its 3-state phone HMM (plus non-emitting boundaries).​

●​ Example:​

○​ In Fig., the word model was built using single-state phones.

○​ Now, we replace each phone with the 3-state phone model.​
5. How Transitions Work in Word Models
●​ In the expanded word HMM (Fig.):​

○​ We don’t keep separate non-emitting states for each phone.​

○​ Instead, we connect phones directly:​

1.​ End of one phone → beginning of the next.​

○​ Only two non-emitting states remain:​

1.​ One at the very start of the word.​

2.​ One at the very end of the word.​

This makes the whole word model simpler and efficient.

6. Why This Approach Is Better

●​ Phones can be long (sometimes 100 frames), and their acoustic features change a lot
inside.​

●​ Splitting them into beginning, middle, end states helps capture:​

○​ Duration flexibility (via self-loops).​

○​ Acoustic variation within the phone.​

○​ More accurate modeling of real speech.​

●​ So, instead of a “flat” phone model, we get a structured phone model that better matches
human speech.
 FEATURE EXTRACTION : MFCC VECTORS

1. Goal of This Section

●​ We want to transform the raw speech waveform (the wavy signal you see when
someone talks) into a sequence of acoustic feature vectors.​

●​ Each feature vector summarizes the sound in a small time window (like 10–20 ms).​

●​ The most widely used feature representation is MFCC (Mel-Frequency Cepstral

Coefficients).​

2. Analog to Digital Conversion (Preprocessing Step)

Before extracting MFCCs, the analog speech signal must be digitized.​
This has two main steps:

(a) Sampling
 ●​ Speech is a continuous wave in air → microphone converts it to an electrical signal.​

●​ We measure the amplitude of this wave at regular time intervals → these measurements
are called samples.​

●​ The sampling rate = number of samples taken per second.​

Key rule:

●​ To capture a wave correctly, we need at least 2 samples per cycle.​

●​ Otherwise, we miss the frequency information.​

This leads to:

●​ Nyquist frequency = maximum frequency that can be captured = half of the sampling
rate.​

(b) Quantization

●​ Each sample is stored as a number (integer).​

●​ Since computers cannot store infinite precision, the amplitude is rounded into fixed
levels.​

●​ Commonly used:​

○​ 8-bit integers (range: -128 to 127).​

○​ 16-bit integers (range: -32,768 to 32,767).​

●​ This rounding process is called quantization.​

●​ Example: Two very close amplitudes may get stored as the same value → loss of fine
detail.​

3. Choosing Sampling Rates for Speech

 ●​ Human speech has important frequency information below 10,000 Hz.​

●​ By Nyquist’s theorem, to capture 10,000 Hz, we need at least 20,000 Hz sampling rate.​

●​ But in practice:​

○​ Telephone speech only keeps frequencies below 4,000 Hz → so 8,000 Hz

sampling rate is enough.​

○​ Microphone / wideband speech often uses 16,000 Hz sampling rate.​

Example:

●​ At 8,000 Hz → 8,000 samples per second.​

●​ Each sample stored in 16-bit format = big data size → efficient storage is important.​

4. Representing the Digital Waveform

●​ Once digitized:​

○​ Each sample is written as x[n], where n is the sample index (time step).​

●​ Now we have a sequence of numbers that describe the speech signal.​

5. Extracting MFCC Features (Overview)

●​ From this sequence of samples, we want to extract MFCC features.​

●​ Why?​

○​ Raw samples are too detailed and noisy.​

○​ MFCCs give a compact representation that matches how humans hear sound.​

●​ The MFCC extraction process usually has 7 steps (shown in Fig. 9.8 in your book).​
 ●​ Each step gradually transforms raw audio into feature vectors.​

●​ Each vector = about 39 numbers, representing the spectrum and energy in that short time
slice.​

Pre-emphasis
●​ Pre-emphasis is the first step in MFCC feature extraction.​

●​ It’s a filtering process applied to the speech signal before further analysis.​

●​ Its main job: boost high-frequency sounds in speech.​

Why Do We Need Pre-emphasis?

●​ Human speech naturally has more energy in lower frequencies and less in higher
frequencies.​

●​ This effect is called spectral tilt.​

○​ Example: In vowels, most energy is concentrated in the low frequencies.​

●​ But higher frequencies also carry important information (like formants that help
distinguish phones).​

●​ If we don’t boost them, the model might ignore important details in higher frequencies.​

So, pre-emphasis balances the spectrum by amplifying high frequencies.

How Does Pre-emphasis Help?

●​ Boosts high-frequency energy → makes the spectral features more even.​

●​ Improves phone detection accuracy → higher formants become clearer, so the system can
better tell phones apart.​
●​ Especially useful for:​

○​ Detecting consonants (which often have higher-frequency energy).​

○​ Capturing formant transitions in vowels and diphthongs.​

How Is Pre-emphasis Done?

●​ Done using a filter (a simple digital operation applied to the signal).​

●​ The filter slightly reduces low-frequency energy and boosts high-frequency energy.​

●​ After filtering → the speech spectrum looks “flatter” (less tilted).​

Example (from Fig.)

●​ Input: Vowel [aa] spoken by the author.​

●​ Before pre-emphasis → spectrum shows strong energy at low frequencies but weak at
high frequencies.​

●​ After pre-emphasis → high frequencies are lifted up, so the overall spectrum is more
balanced.​
Computing Acoustic Likelihoods

Purpose of the Section

●​ We previously learned how to extract MFCC features from a wave file.​

●​ This gives us a 39-dimensional feature vector every 10 milliseconds —

representing the spectral characteristics of the sound.​

●​ Now, we need to understand how to compute the likelihood of these feature

vectors given a specific HMM state.​

●​ In HMMs, this is the B probability function, which gives us:​

○​ p(ot | qi) — the probability of the observation ot given the state qi, also
written as bt(i).​

Comparison with POS Tagging (from Chapter 5)

●​ In part-of-speech tagging, observations (ot) are discrete symbols (like words).​

 ●​ To find p(word | tag), we simply count how many times a tag emits a word in the
training data.​

BUT...

●​ In speech recognition, MFCC vectors are real numbers.​

●​ So, we cannot count how often a certain vector appears — each vector is almost
always unique.​

Need for Real-Valued Likelihood Function

●​ In decoding, we’re given a feature vector ot, and we need to compute p(ot | qi)
for each possible state qi.​

●​ This helps us choose the most likely sequence of HMM states.​

●​ Once we define how to compute this likelihood, we also need to modify the
Baum-Welch algorithm to train this model properly.​

Vector Quantization (VQ)

Main Idea of VQ

●​ Goal: Make MFCC vectors look like discrete symbols — so we can count and
compute probabilities as in text-based tasks.​

●​ VQ is a method to map real-valued vectors into a small number of discrete

symbols.​
 ●​ This allows us to compute p(symbol | state) by counting, like we did for words.​

What is Vector Quantization?

●​ We convert each input feature vector into one of a small number of classes.​

●​ Each class is represented by a symbol.​

●​ More formally, VQ uses:​

○​ A codebook​

○​ A clustering algorithm​

○​ A distance metric​

Codebook

●​ A codebook is a list of possible classes — a vocabulary:​

[ V = {v₁, v₂, ..., v } ]​

●​ Each symbol vk in the codebook is associated with a prototype vector, also

called a codeword.​

●​ For example, if we use 256 codewords, we can represent each input vector by a
number from 0 to 255.​

○​ This is called 8-bit VQ because one byte (8 bits) can represent 256 values.​

●​ Each of the 256 numbers corresponds to a prototype feature vector.​

How the Codebook is Built

●​ Use a clustering algorithm (like K-means) to group all training vectors into 256
clusters.​

●​ From each cluster, choose a representative vector — this becomes the codeword.​

●​ For full details on clustering, refer to:​

○​ Huang et al. (2001)​

○​ Duda et al. (2000)​

Using the Codebook in Practice

●​ For every new incoming feature vector:​

1.​ Compare it to all 256 prototype vectors.​

2.​ Select the closest one (based on a distance metric).​

3.​ Replace the input vector with the index of the closest codeword.​

●​ This process is illustrated in Figure​

Why is VQ Useful?

●​ Once we have discrete classes, we can compute p(vk | HMM state) by:​

○​ Counting how many times each class occurs when the state is active.​

○​ Normalizing these counts to get probabilities.​

Distance Metric

●​ Both clustering and decoding need a way to measure similarity between vectors.​

●​ This is done using a distance (or distortion) metric.​

Euclidean Distance

●​ The simplest metric is Euclidean distance.​

●​ It measures the straight-line distance between two vectors in N-dimensional space.​

●​ Given vectors x and y of length D, Euclidean distance is:​

 ​

●​ In practice, we often use the square of the Euclidean distance (to avoid
computing square roots):​

●​ Even when we say “Euclidean distance,” we usually mean the squared version.​
Gaussian Models

(a) Univariate Gaussian

●​ Models a single feature dimension with:​

○​ Mean (µ): average value of feature.​

○​ Variance (σ²): how spread out the values are.​

●​ Formula:​

●​ In ASR: Each HMM state → has its own µ, σ² for each feature.​

(b) Multivariate Gaussian

●​ Used since MFCC vectors have 39 dimensions.​

●​ Defined by:​

○​ Mean vector (µ) – average of each dimension.​

○​ Covariance matrix (Σ) – captures variance and correlation between

features.​

●​ Types:​

○​ Full covariance: Captures correlations → more accurate but costly (D²

parameters).​
 ○​ Diagonal covariance: Simpler, faster → assumes features are independent.​

●​ Most ASR systems → use diagonal covariance for efficiency.​

Gaussian Mixture Models (GMMs)

●​ Problem with single Gaussian: Real data may not follow a simple bell curve.​

●​ Solution: Use multiple Gaussians combined:​

​
where:​

○​ (M) = number of mixtures,​

○​ cjm​= weight of each mixture,​

○​ μjm​,Σjm​ = mean and covariance of each Gaussian.​

●​ Training:​

○​ Uses Baum-Welch algorithm (Expectation-Maximization).​

○​ Assigns probabilities to mixture components → iteratively refines

parameters.​

●​ Advantage: Flexible, can model complex feature distributions.​

●​ Role: Standard in ASR before neural networks (DNN-HMM hybrid later replaced
GMM-HMM).​
Probabilities vs Log Probabilities

●​ Problem: Multiplying small probabilities across frames causes underflow.​

●​ Solution: Work with log probabilities:​

○​ Multiplication → Addition.​

○​ Faster computation.​

○​ More stable numerically.​

●​ Example:​
Instead of multiplying many tiny values like 0.00001 × 0.0001 → underflow,​
compute log(0.00001) + log(0.0001) = manageable negative numbers.​

●​ Log-likelihood functions often resemble distance metrics (e.g., Mahalanobis).​

Lexicon and Language Model

●​ Lexicon:​

○​ Dictionary mapping words → phone sequences.​

○​ Example: "cat" → /k ae t/.​

○​ Sources: CMU Dictionary (widely used in ASR).​

●​ Language Models (LMs):​

○​ Predict word sequences.​

 ○​ Usually N-grams:​

■​ Trigram and 4-gram for LVCSR (Large Vocabulary Continuous

Speech Recognition).​

■​ Smaller apps → bigram/unigram or finite-state grammars.​

○​ Used to reduce word sequence errors by providing context.​

Search and Decoding

1. Goal of Decoding:​
Decoding in speech recognition is the process of finding the most probable word
sequence Ŵ given a sequence of acoustic observations O.​
According to the noisy channel model,​

​
where:

●​ P(O|W) = Acoustic model likelihood​

●​ P(W) = Language model prior probability​

2. Adjustments in Probability Calculation:​

Since the independence assumption is not fully correct, probabilities are reweighted
using two factors:

●​ Language Model Scaling Factor (LMSF): Increases the weight of the language
model to balance acoustic likelihoods.​

●​ Word Insertion Penalty (WIP): Adjusts for the tendency to prefer longer or
 shorter word sequences.​

Final decoding equation:​

​
or in log form:​

3. Hidden Markov Model (HMM) Components:

●​ Q: States (subphones)​

●​ A: Transition probability matrix​

●​ B: Observation (emission) probabilities​

Each word is represented as a chain of phone HMMs, each expanded into subphones,
often with optional silence at the end.

4. Forward Algorithm:​
Computes total likelihood P(O|W) by summing over all possible state sequences.​

​
It is a dynamic programming method that builds probabilities step-by-step over time.

5. Viterbi Algorithm (Decoding):​

Finds the most likely sequence of hidden states (best path).​
It uses backpointers to trace the optimal path and determine the word sequence.​
Viterbi gives the probability of the best path, while the forward algorithm sums over all
paths.

6. Multi-word Decoding and Language Models:​

Transitions between words are added using bigram or N-gram probabilities (e.g., P(w₂ |
w₁)).​
The system combines intra-word and inter-word transitions in one large trellis.

7. Beam Search (Pruning):​

To make decoding faster, beam search limits computation to the most probable paths:

●​ Keeps only states within a beam width (θ) of the best score.​

●​ Reduces computation by discarding unlikely paths.​

●​ Typically retains 5–10% of the search space with minimal accuracy loss.​

Embedded Training

●​ Definition:​
Embedded training is an automatic method to train HMM-based speech
recognition systems without manual phonetic labeling.​

●​ Model Components:​

1.​ Q: Subphone states​

2.​ A: Transition probabilities (self-loop/next subphone)​

3.​ B: Observation likelihoods (emission probabilities)​

●​ Need:​
 Manual labeling is costly and inconsistent; embedded training uses automatic
alignment through the Baum-Welch algorithm.​

●​ Process:​

1.​ Build “whole-sentence” HMMs using transcriptions and lexicon.​

2.​ Initialize A (0.5 for valid transitions, 0 for others).​

3.​ Initialize B (global mean and variance).​

4.​ Run multiple Baum-Welch iterations to re-estimate A and B.​

●​ Flat Start:​
Equal initial transition and Gaussian parameters for all states.​

●​ Viterbi (Forced Alignment):​

Uses only the most probable path instead of summing all paths.​
Faster approximation of Baum-Welch used to align states with observations.​

●​ Equations:​
Mean: ( \mu_i = \frac{1}{T} \sum o_t )​
Variance: ( \sigma_i^2 = \frac{1}{T} \sum (o_t - \mu_i)^2 )​

●​ Use:​
Also applied in hybrid systems (HMM/MLP, HMM/SVM) for iterative alignment
and training.​