Speech Processing

Lecture #2
Time Frequency Representation of Speech Signals

[Link]
Siddiqui
mandleeb@[Link].e
[Link]
 Fourier transform
 Motivation for using Fourier transform (FT)
 The waveform (a general time-domain representation) provides some
indication of the dynamics and periodicity of audio, but apart from this, it
is not clear, for example, about its frequency distribution.
 Fourier transform provides an alternative representation of the signal,
suitable for displaying other speech characteristics, such as its
frequency information, harmonics, etc.

 Definition
 The Fourier transform of a continuous signal x(t ) is computed as:

X ( )   x(t )e jt dt


where  is the angular frequency:   2f

 Inverse Fourier transform:

x(t )  21 
2
X ()e jt d
2 2
Discrete-time Fourier transform (DTFT)
 The Fourier transform of a discrete-time signal x[n]  x(nT ) is computed as:

X ( )   x[n]e  jn


where  is the angular frequency:   2f

 Inverse discrete-time Fourier transform:

x[n]  21  2
X ()e jn d

3 3
 Discrete Fourier transform (DFT)
 The Fourier transform of a digital signal x[n ] is computed as:
N 1
X [k ]   x[n]e  jk n
n 0

2k
where k is the angular frequency: k  , k  0,1,..., N  1
N
 Inverse discrete Fourier transform:
N 1
1
x[n] 
N
 X
k 0
[ k ]e j k n
, n  0,1,..., N  1

4 4
 Power spectral density (PSD)
 PSD is defined as the magnitude squared of the DFT of the signal:

P[k ]  X [k ]
2

 Examples:

PSD of a vowel spoken by a PSD of a fricative spoken by a male

male speaker speaker 5 5
 Fast Fourier transform (FFT)
 FFT is a fast computation of DFT. The typical FFT algorithm consists
of three conceptual parts:
 Shuffling (bit reversal): shuffling the N-dimensional input into N one-
dimensional signals
 Performing N one-point DFTs
 Merging the N one-point DFTs into one N-point DFT using “Butterfly”
merging equations (requiring that N to be an integral power of 2.)

 The computational complexities of FFT and DFT are respectively:

 FFT: O( N log N )
 DFT: O( N 2 )

6 6
Short-time Fourier transform (STFT)
 STFT (sometimes called short-term FT) can be computed as a N-point
windowed DFT as follows (note that we only consider the discrete form here,
and in practice FFT is usually used to compute the DFT in each frame ):
N 1
x[k , m]   x[m  n]w(n)e  jk n
n 0
2k
where k - the discrete angular frequency: k  , k  0,1,..., N  1
N
m - the time-frame index

 - the hop size

w(n ) - a window function, such as rectangular, Hann

windows

7 7
 Spectrogram
 Spectrogram of a speech signal can be computed as magnitude squared
STFT:

spectrogram{x[n]}  x[k , m]
2

 An example:

Spectrogram of female speaker uttering “warm cloak”

8 8
Decomposition of the speech signal (x[n]) as a
source (e[n]) passed through a linear time-
varying filter (h[n]).
Estimation of the filter, inspired by:
• Speech production models
– Linear Predictive Coding (LPC)
– Cepstral analysis
• Speech perception models (part II)
– Mel-frequency cepstrum
– Perceptual Linaer Prediction (PLP)
Speech recognizers estimate filter
characteristics and ignore the source
• Spectrogram
– Representation of a signal highlighting several
of its properties based on short-time Fourier
analysis
– Two dimensional: time horizontal and frequency
vertical
– Third ‘dimension’: gray or color level indicating
energy
• Spectrogram
– Narrow band
• Long windows (> 20 ms) →
• Narrow bandwidth
• Lower time resolution, better frequency resolution
– Wide band
• Short windows ( <10 ms) →
• Wide bandwidth
• Good time resolution, lower frequency resolution
– Pitch synchronous
• Requires knowledge of local pitch period
• Spectrogram
• Window analysis
– Series of short segments, analysis frames
– Short enough so that the signal is stationary
– Usually constant, 20-30 ms
– Overlaps possible
– Different types of window functions (wm[n]):
• Rectangular (equal to no window function)
• Hamming
• Hanning

X m e j  
 


n  
xm [n]e jn   w[ m
n  
 n ] x[ n ]e j n
Short-Time Fourier Analysis
• Window analysis
– Window size must be long enough
• Rectangular: N ≥ M
• Hamming, Hanning: N ≥ 2M
– Pitch period not known in advance →
– Prepare for lowest pitch period →
– At least 20ms for rectangular or 40ms for
Hamming/Hanning (50Hz)
– But longer windows give a more average spectrum
instead of distinct spectra →
– Rectangular window has better time resolution
Short-Time Fourier Analysis
Short-Time Fourier Analysis
Short-Time Fourier Analysis
Short-Time Fourier Analysis
Speech Signal Representations I

Short-Time Fourier Analysis

Speech Signal Representations I

Short-Time Fourier Analysis

Speech Signal Representations I

Short-Time Fourier Analysis

Speech Signal Representations I

Short-Time Fourier Analysis

• Window analysis
– Frequency response not completely zero outside main
lobe → Spectral leakage
– Second lobe of a Hamming window is approx. 43dB
below main lobe → less spectral leakage
– Hamming, Hanning, triangular windows offer less
spectral leakage →
– Rectangular windows are rarely used despite their
better time resolution
Speech Signal Representations I

Short-Time Fourier Analysis

Speech Signal Representations I

Short-Time Fourier Analysis

Speech Signal Representations I

Short-Time Fourier Analysis

Speech Signal Representations I

Short-Time Fourier Analysis

Speech Signal Representations I

Short-Time Fourier Analysis

Short-time spectrum of male voice speech

a) Time signal /ah/
local pitch 110Hz
b) 30ms rectangular
window
c) 15ms rectangular
window
d) 30ms Hamming
window
e) 15ms Hamming
window
Speech Signal Representations I

Short-Time Fourier Analysis

Short-time spectrum of female voice speech

a) Time signal /aa/
local pitch 200Hz
b) 30ms rectangular
window
c) 15ms rectangular
window
d) 30ms Hamming
window
e) 15ms Hamming
window
Speech Signal Representations I

Short-Time Fourier Analysis

Short-time spectrum of unvoiced speech

a) Time signal
b) 30ms rectangular
window
c) 15ms rectangular
window
d) 30ms Hamming
window
e) 15ms Hamming
window
 Spectrogram (cont.)
 Each vertical line of the spectrogram describes the frequency-
dependent power distribution of the signal over a short segment (or
window) of the speech signal, i.e. PSD of the segment.
 The width of the window is N, and the gap between consecutive
windows is the hop size  .
 The horizontal line of the spectrogram represents the power
distribution within a particular frequency band as a function of time.
 The spectrogram shows the time-frequency spectral distribution of
power within the signal.
 The spectrogram is much better suited than the waveform to
displaying speech structures, e.g. harmonics, the energy balance of
frequency components, formants, etc.
 The time and frequency resolution of the spectrogram are inversely
proportional.

31 31
 Spectrogram – resolution issues
 The STFT has a fixed resolution that depends on the selection of the
window size.
 A wider window gives better frequency resolution (frequency
components close together can be separated) but poorer time
resolution (the time at which frequencies change), and vice versa.
 We use the example from Wikipedia to demonstrate this: a signal is
composed of 4 sinusoidal components, whose frequencies are 10, 25,
50, 100Hz respectively, with the same length of 5 seconds. The
sampling frequency of the signal is 400Hz.
 Multi-resolution analysis tools exist that do not suffer from this
problem, such as wavelet transform.

32 32
Spectrogram resolution issues (cont.)

Different time-frequency resolutions for the same signal due to different

33 window 33
sizes were used in generating the STFT. (Resource: from Wikipedia.)
Spectrogram resolution issues (cont.)
 Although a long window can give higher frequency resolution, it would
be misleading if we use too long a window, as the spectral
characteristics would change over the duration of the windowed
segment.
 How long window should we choose such that the spectral
characteristics does not change (dramatically)? This question relates to
the concept of “stationarity”.
 In practice, speech segment with a length of around 20-30ms is usually
regarded as “quasi-stationary” (very littler change in spectral
characteristics). This is because the speech units (phonemes) occur at
a rate of 4-5 per second for average speech, although more rapid
changes can occur from one steady state to another.
 To ensure smooth transitions of the energy distribution from frame to
frame, the windows are usually chosen to be overlapping, with a typical
hop size of 5ms.

34 34
 Windowing and overlapping in
spectrogram
 By choosing the window length appropriately, the assumption of
stationarity (quasi-stationarity) within the windowed speech is almost true.
 However, when appending copies of the segment one after another, there
may still be sharp discontinuities in the waveform at the boundaries (see
the figure below).
 The discontinuity results in the high-frequency noise spread across the
spectrum, known as spectral leakage.

Spectral leakage:
(a) a sinusoidal audio segment
(b) its periodic extension.

35 35
 Windowing and overlapping in
spectrogram (cont.)
 To reduce spectral leakage, we can multiply the segment with a window
function that approaches zero at its ends, such as Hann window shown
below. This effectively attenuates discontiuities between two boundairies
of the window, and therefore reduces the leakage.

The waveform and amplitude spectrum of the Hann window function

 In practice, short segments can be appended with zeros to the required

length, known as zero-padding. 36 36
Various window functions

Source: Kondoz (2001)

37
Time plots of various window
functions

Source: Kondoz (2001)

38
Frequency response of various
window functions

Source: Kondoz (2001) 39

 Short-time spectral analysis
using DFT
 Effect of window types on voiced speech with 220 samples window length.
(a) and (b) are time and frequency plots of speech using a rectangular
window, and (c) and (d) are time and frequency plots of speech using
Hamming window.

Source: Kondoz (2001) 40

 Short-time spectral analysis
using DFT
 Effect of window types on unvoiced speech with 220 samples window
length. (a) and (b) are time and frequency plots of speech using a
rectangular window, and (c) and (d) are time and frequency plots of speech
using Hamming window.

Source: Kondoz (2001) 41

 Short-time spectral analysis
using DFT
 Effect of window types on voiced speech with 40 samples window length.
(a) and (b) are time and frequency plots of speech using a rectangular
window, and (c) and (d) are time and frequency plots of speech using
Hamming window.

Source: Kondoz (2001) 42

 Discrete Cosine Transform (DCT)
 A definition of DCT transform (DCT II) is shown below:

Source: wikipedia

43
 MDCT
 An advantage of Modified DCT (MDCT) is that it allows for a 50% overlap
between blocks without increasing the data rate.
 The MDCT is an example of a class of transforms called Time Domain
Aliasing Cancellation (TDAC). In particular, MDCT is sometimes referred to
as oddly-stacked TDAC (OTDAC).
 These transforms do not invert like the DFT to recover the original signal
but rather invert to recover a signal that has adjacent blocks’ signal mixed
into it so that the effect of “time-domain aliasing”, i.e. the mixing of adjacent
blocks of data, is removed. As a result, the input signal is perfectly
reconstructed.

44
 MDCT (cont.)
Analysis: from time to frequency

Synthesis: from frequency to time

For the signal to be perfectly reconstructed from after synthesis

process, the windows should satisfy the following condition:

where i is the index of blocks (or short-time frames), subscript a means

analysis, and s means synthesis. n0 = (N/2+1)/2.
45
 MDCT (cont.)
 Responses of the MDCT filter bank (cosine window function):

Source: Bosi & Goldberg (2002)

46
 Subband analysis of audio signals
 General subband analysis framework for audio coding:

Source: Bosi & Goldberg (2002)

47