Time-Domain Methods for
Speech Processing
Contents
Introduction
Time-Dependent Processing of Speech
Short-Time Energy and Average Magnitude
Short-Time Average Zero Crossing Rate
Speech vs. Silence Discrimination Using Energy and
Zero-Crossing
The Short-Time Autocorrelation Function
The Short-Time Average Magnitude Difference
Function
Time-Domain Methods for
Speech Processing
Introduction
Speech Processing Methods
Time-Domain Method:
Involving the waveform of speech signal
directly.
Frequency-Domain Method:
Involving some form of spectrum
representation.
Time-Domain Measurements
Average zero-crossing rate, energy, and the
autocorrelation function.
Very simple to implement.
Provide a useful basis for estimating
important features of the speech signal, e.g.,
Voiced/unvoiced classification
Pitch estimation
Time-Domain Methods for
Speech Processing
Time-Dependent
Processing of Speech
Time Dependent Natural of Speech
This is a test.
Time Dependent Natural of Speech
Short-Time Behavior of Speech
Assumption
The properties of speech signal change
slowly with time.
Analysis Frames
Short segment of speech signal.
Overlap one another usually.
Time-Dependent Analyses
Analyzing each frame may produce either a
single number, or a set of numbers, e.g.,
Energy (a single number)
Vocal tract parameters (a set of numbers)
Thiswill produce a new time-dependent
sequence.
General Form
Qnn
Q TT[[xx((mm)])]ww((nnmm))
mm
n: Frame index
x(m): Speech signal
T[ ]: A linear or nonlinear transformation.
w(m): A window function (finite of infinite).
General Form
Qnn
Q TT[[xx((mm)])]ww((nnmm))
mm
Qn is a sequence of local weighted
average values of the sequence T[x(m)].
Example
Energy E x
m
2
( m)
Short-Time n
En x (m) 2
Energy m n N 1
Example
Short-Time n
En x (m) 2
Energy m n N 1
Example
T [ x(m)] x (m)
2
1 0 m N 1
w(m)
0 otherwise
n
Short-Time
Ennn
E 2
TT[[xxx((m
m()]
)]w
m ) ((nnm
w m))
Energy mm
m N 1
n
General Short-Time-Analysis Scheme
Linear
Linear Lowpass
Lowpass
TT[[ ]]
Filter
Filter Filter
Filter
Depending on the
choice of window
Time-Domain Methods for
Speech Processing
Short-Time Energy and
Average Magnitude
Applications
Silence Detection
Segmentation
Lip Sync
Short-Time Energy
En [ x(m)w(n m)]
m
2
x
m
2
( m) w ( n m)
2
x
m
2
( m ) h ( n m)
x 2 ( m) * h ( m)
Short-Time Average Magnitude
Mn | x(m) | w(n m)
m
| x(m) | *w(m)
Block Diagram Representation
x(n) x2(n)
[[ ]]22 h(n)
h(n) En
h( n) w ( m)
2
x(n) |x(n)|
|| || w(n)
w(n) Mn
What
What isis the
the effect
effect of
of windows?
windows?
Block Diagram Representation
x(n) x2(n)
[[ ]]22 h(n)
h(n) En
h( n) w ( m)
2
x(n) |x(n)|
|| || w(n)
w(n) Mn
The Effects of Windows
Window length
Window function
Rectangular Window
1 0 n N 1
h( n)
0 otherwise
j
j((NN11))//22
sin(
sin(N
N // 22))
H((ee )) ee
j
H j
sin(
sin(// 22))
j
j((NN11))/ /22
sin(
sin(N
N //22))
H((ee )) ee
j
H j
sin(
sin(
//22))
Rectangular Window
8 | H (e j ) |
N=8
Peak sidelobe
2 2 2 2
N N
Mainlobe
m
width
j
j((NN11))/ /22
sin(
sin(N
N //22))
H((ee )) ee
j
What
What is
is this?
this? H j
sin(
sin(
//22))
Rectangular Window
Discuss
Discuss the
the effect
effect of
of window
window duration.
duration.
Discuss
Discuss the
the effect
effect of
of mainlobe
mainlobe width
width and
and sid
sid
8 | H (e j ) |
N=8
Peak sidelobe
2 2 2 2
N N
Mainlobe
m
width
Commonly Used Windows
Rectangular
1
0.8
0.6 Hamming
Bartlett
0.4
Hanning
0.2
Blackman
0
0 5 10 15 20
Commonly Used Windows
11 00nnNN11
Rectangular ww((nn))
00 otherwise
otherwise
22nn/(/(NN11)) 00nn((NN11))/ /22
Bartlett
ww((nn)) 2222nn/(/(NN11)) ((NN11))/ /22nnNN11
(Triangular) 0 otherwise
0 otherwise
0.5 0.5 cos[2n /( N 1)] 00nnNN11
Hanning ww((nn)) 0.5 0.5 cos[2n /( N 1)]
00 otherwise
otherwise
00.54
. 5400.46
.46cos[
cos[22n
n/(/(NN11)])] 00nnNN11
Hamming ww((nn))
00 otherwise
otherwise
0.42 0.5 cos[2n /( N 1)] 0.8 cos[4n /( N 1)] 00nnNN11
Blackman ww((nn)) 0.42 0.5 cos[2n /( N 1)] 0.8 cos[4n /( N 1)]
00 otherwise
otherwise
Commonly Used Windows
Hanning
Least mainlobe width
Rectangular Hamming
Bartlett
Blackman
Examples: Short-Time Energy
Rectangular Window Hamming Window
Examples: Average Magnitude
Rectangular Window Hamming Window
The Effects of Window Length
Increasing the window length N, decreases
the bandwidth.
If N is too small, e.g., less than one pitch
period, En and Mn will fluctuate very rapidly.
IfN is too large, e.g., on the order of several
pitch periods, En and Mn will change very
slowly.
The Choice of Window Length
No signal value of N is entirely satisfactory.
Thisis because the duration of a pitch period
varies from about 2 ms for a high pitch
female or a child, up to 25 ms for a very low
pitch male.
Sampling Rate
The bandwidth of both En and Mn is just that
of the lowpass filter.
So, they need not be sampled as frequently as
speech signals.
For example
Frame size = 20 ms
Sample period = 10 ms
Main Applications of En and Mn
To provide
the basis for distinguishing
voiced speech segments from unvoiced
segments.
Silence detection.
Differences of En and Mn
EEnn
[[xx((mm))ww((nnmm)])]
22 Emphasizing large sample-to-
sample variations in x(n).
mm
The dynamic range (max/min)
M
Mnn ||xx((m
mm
m))||ww((nnm
m)) is approximately the square
root of En.
The differences in level between voiced and unvoiced
regions are not as pronounced as En.
FIR and IIR
Allthe windows that we discussed
are FIRs.
Each of them is a lowpass filter.
It can also be an IIR.
IIR Example
aann nn00 11
hh((nn)) H((zz))
H 11
00 nn00 11az
az
Recursive formulas:
Short-Time Energy: EEnn aE 22
aEnn11 x ((nn))
x
Short-Time
Average magnitude: Mnn aM
M aMnn11||xx((nn))||
Time-Domain Methods for
Speech Processing
Short-Time Average
Zero-Crossing Rate
Voiced and Unvoiced Signals
Th/i/s
Thi/s/
The Short-Time Average Zero-Crossing Rate
ZZnn
||sgn[ m)])]sgn[
sgn[xx((m m11)])]||ww((nnm
sgn[xx((m m))
mm
1 x ( m) 0 1
sgn[ x(m)] w(m) 0 m N 1
1 x ( m) 0 2N
x(n) First Lowpass Zn
Difference
| |
Filter
Distribution of Zero-Crossings
Example
Time-Domain Methods for
Speech Processing
Speech vs. Silence
Discrimination Using
Energy and Zero-Crossing
Speech vs. Silence Discrimination
Locating the beginning and end of a speech
utterance in the environment with background
of noise.
Applications:
Segmentation of isolated word
Automatic speech recognition
Save bandwidth for speech transmission
Examples:
In some cases, we
can locate the
beginning and end of
a speech utterance
using energy alone.
Examples:
In other cases, we
can locate the
beginning and end of
a speech utterance
using zero-crossing
rate alone.
Examples:
Sometimes, we
cannot do it using
Actual beginning
one criterion alone.
Difficulties
In general, it is difficult to locate the boundaries
if we encounter the following cases:
Weak fricatives (/f/, /th/, /h/) at the beginning or end.
Weak plosive bursts (/p/, /t/, /k/) at the beginning or
end.
Nasals at the end.
Voiced fricatives which become devoiced at the end
of words.
Trailing off of vowel sounds at the end of an
utterance.
Rabiner and Sambur
10 msec frame with sampling rate 100 time/sec
is used.
The algorithm assumes that the first 100 msec
of the interval contains no speech.
The means and standard deviations of the
average magnitude and zero-crossing rate of
this interval are computed to characterize the
background noise.
The Algorithm
The Algorithm
1
2
No more than 25 frames
Examples
Examples
Time-Domain Methods for
Speech Processing
The Short-Time
Autocorrelation Function
Autocorrelation Functions
((kk))
xx((mm))xx((mmkk))
mm
x(m)
k
x(m+k)
((kk))
x
x ((mm))xx((mmkk))
mm
Properties
1. Even: (k) = (k).
2. (k) (0) for all k.
3. (0) is equal to the energy of x(m).
x(m)
k
x(m+k)
((kk))
x
x ((mm))xx((mmkk))
mm
Properties
4. If x(m) has period P, i.e. x(m)= x(m+P), then
((kk)) ((kk PP))
x(m)
k
x(m+k)
((kk))
x
x ((mm))xx((mmkk))
mm
Properties
4. If x(m) has period P, i.e. x(m)= x(m+P), then
((kk)) ((kk PP))
This motivates us to use autocorrelation for pitch detection.
((kk))
x
x ((mm))xx((mmkk))
mm
Short-Time Version
RRnn((kk))
x
x ((mm))ww((nnmm))xx((mmkk))ww((nnkk
m
m ))
mm
x(m)
n
x(m)w(nm)
k
x(m+k)w(nkm)
RRnn((kk))
xx((mm))ww((nnmm))xx((mmkk))ww((nnkkmm))
mm
Property
RRnn((kk)) RRnn((kk))
x(mk)w(n+km)
Rn(k) k
x(m)w(nm)
k Rn(k)
x(m+k)w(nkm)
RRnn((kk))
xx((mm))ww((nnmm))xx((mmkk))ww((nnkkmm))
mm
Property
RRnn((kk)) RRnn((kk))
Rn (k ) x(m)w(n m) x(m k )w(n k m)
m
x(m) x(m k )[w(n m)w(n m k )]
m
y k(m) h k(n m)
yk (n) * hk (n)
yykk((nn)) xx((nn))xx((nnkk))
hhkk((nn))ww((nn))ww((nnkk))
Property
RRnn((kk)) RRnn((kk)) yk (n) * hk (n)
Rn (k ) x(m)w(n m) x(m k )w(n k m)
m
x(m) x(m k )[w(n m)w(n m k )]
m
y k(m) h k(n m)
yk (n) * hk (n)
yykk((nn)) xx((nn))xx((nnkk))
hhkk((nn))ww((nn))ww((nnkk))
Property
RRnn((kk)) RRnn((kk)) yk (n) * hk (n)
x(n) hhkk(n)
(n) Rn(k)
zzkk
RRnn((kk))
xx((mm))ww((nnmm))xx((mmkk))ww((nnkkmm))
mm
Another Formulation
ww''((nn)) ww((nn))
Rn (k ) x(m)w[(m n)]x(m k )w[(m n k )]
m
x(m)w' (m n) x(m k )w' (m n k )
m
x(m n)w' (m) x(m n k )w' (m k )
m
ww''((nn)) 00 for
for 00 nn NN 11
Another Formulation
ww''((nn)) ww((nn))
Rn (k ) x(m)w[(m n)]x(m k )w[(m n k )]
m
x(m)w' (m n) x(m k )w' (m n k )
m
x(m n)w' (m) x(m n k )w' (m k )
m
N 1 k
[ x(n m)w' (m)][ x(n m k )w' (m k )]
m 0
AA noncausal
noncausal formulation
formulation
N=401
N=401
Examples
voiced
voiced
Unvoiced
Unvoiced
Rectangular Window Hamming Window
NN11kk
RRnn((kk))
[[xx((nnmm))ww''((mm)][
mm00
)][xx((nnmmkk))ww' '((mmkk))]]
ExamplesLess data will be involved for
Less data will be involved for
larger
larger lag
lag k.k.
N=401
N=401 R (k ) 1 k / N , | k | N
N=251
N=251
N=125
N=125
Modified Short-Time
Autocorrelation Function
Original Version:
RRnn((kk))
[[xx((nnmm))ww''((mm)][
)][xx((nnmmkk))ww' '((mmkk))]]
mm
Modified Version:
RRnn((kk))
[[xx((nnmm))ww11((mm)][
)][xx((nnmmkk))ww22((mmkk))]]
mm
RRnn((kk))
[[xx((nnmm))ww11((mm)][
)][xx((nnmmkk))ww22((mmkk))]]
Modified Short-Time mm
Autocorrelation Function
ww11((mm))
ww22((mm)) K Max. lag
NN11
RRnn((kk))
xx((nnmm))xx((nnmmkk)),, 00kk KK
mm00
Modified Short-Time
Autocorrelation Function
ww11((mm))
11 0011 NN 11
ww11((mm))
00 otherwise
otherwise
ww22((mm)) K Max. lag
11 0011 NN 11KK
ww22((mm))
00 otherwise
otherwise
N=401
N=401
Examples
i mi
SSim rila
lar
voiced
voiced
Unvoiced
Unvoiced
Rectangular Window Modified Version
Examples
N=401
N=401
N=251
N=251
N=125
N=125
Rectangular Window Modified Version
Time-Domain Methods for
Speech Processing
The Short-Time
Average Magnitude
Difference Function
The AMDF
nn((kk))
||xx((nnmm))ww11((mm))xx((nnmmkk))ww22((mm))||
mm
If x(n) is periodic with period P, then
n (kP) 0, k 0,1,2,
Computationally
Computationally more
more effective
effective than
than au
au
Example
voiced
voiced
Unvoiced
Unvoiced
Exercise
Recordinga piece of yours speech to perform
voice/unvoice segmentation.
Designa effective algorithm to perform
autocorrelation.