DEEP LEARNING PROJECT REPORT
Deep Voice: Real-time Neural Text-to-Speech
1. Introduction
Text-to-speech (TTS) technology, which synthesizes artificial human speech from text, is a vital
component in numerous applications, including speech-enabled devices, navigation systems, and
accessibility tools for individuals with visual impairments. By enabling human-technology interaction
without the need for visual interfaces, TTS enhances user experience across various domains.
However, modern TTS systems are often characterized by complex, multi-stage processing pipelines
that rely on hand-crafted features and heuristics, making the development of new systems both
labor-intensive and challenging.
Deep Voice offers a novel approach by drawing inspiration from traditional TTS architectures while
replacing all components with neural networks and utilizing simpler features. The process begins with
converting text into phonemes, followed by an audio synthesis model that transforms these linguistic
features into speech (Taylor, 2009). Unlike previous methods that depend on intricate hand-engineered
features, our system focuses solely on phonemes with stress annotations, phoneme durations, and
fundamental frequency (F0). This streamlined approach allows for greater adaptability to new
datasets, voices, and domains without the need for extensive manual data annotation or feature
engineering. We demonstrate the effectiveness of this method by retraining our entire pipeline on a
new dataset consisting solely of audio and unaligned textual transcriptions, achieving high-quality
speech generation in a fraction of the time required by conventional TTS systems. Furthermore, we
address the critical requirement of real-time inference for production-quality TTS systems,
showcasing efficient WaveNet inference kernels that produce high-quality audio at a remarkable
speedup compared to previous implementations.
2. Objective
The objective of Deep Voice is to create a production-quality text-to-speech (TTS) system entirely
powered by deep neural networks, enabling real-time or faster-than-real-time speech synthesis. It
replaces traditional, complex TTS pipelines with a fully trainable, end-to-end architecture that
minimizes reliance on hand-engineered features. By utilizing neural networks for all components, it
simplifies adaptation to new datasets, languages, and voices without manual tuning. The system aims
to deliver high-quality, natural-sounding speech, making it suitable for a wide range of applications,
from assistive technologies to virtual assistants. Its efficient design also reduces training time and
computational demands, ensuring scalability and usability in production environments.
3. Applications of Deep Voice
1. Voice Assistants: Enhances real-time, natural interactions in systems like Alexa, Google
Assistant, and Siri.
2. Assistive Technologies: Provides speech synthesis for visually impaired individuals and
supports those with speech impairments.
3. Language Learning: Offers pronunciation feedback and voice synthesis for educational
tools.
4. Customer Service Automation: Powers interactive voice response (IVR) systems for call
centers with natural-sounding voices.
5. Entertainment: Generates character voices for games, animations, and virtual environments.
6. Audiobook Narration: Produces high-quality, human-like narration for e-books and
podcasts.
7. Accessibility in Education: Converts text-based content into audio for enhanced inclusivity.
8. Navigation Systems: Provides real-time directions with natural voice outputs in vehicles and
GPS devices.
4. Related Work
Previous research has utilized neural networks to replace various components of text-to-speech (TTS)
systems, such as grapheme-to-phoneme conversion, phoneme duration prediction, fundamental
frequency prediction, and audio synthesis. However, unlike Deep Voice, these systems often do not
address the entire TTS problem and rely on specialized hand-engineered features tailored to specific
domains. Recent advancements in parametric audio synthesis, particularly with models like WaveNet,
SampleRNN, and Char2Wav, have made significant strides in audio generation, but they typically
require access to pre-existing TTS systems for training and inference. Deep Voice stands out by
functioning as a standalone system that can be trained from scratch using only a dataset of short audio
clips and their corresponding textual transcripts, minimizing reliance on complex hand-engineered
features. It employs one-hot encoded characters for grapheme-to-phoneme conversion, phoneme
durations, and normalized log fundamental frequency, all easily extracted from audio and transcripts.
Additionally, Deep Voice is designed for production readiness, enabling real-time audio synthesis in
fractions of a second while allowing for a tunable balance between synthesis speed and audio quality,
in stark contrast to previous models like WaveNet, which can take several minutes to synthesize just
one second of audio.
5. Methodology
TTS System Components
1. Grapheme-to-Phoneme Model:
Converts text into phonemes using a phonemic alphabet (e.g., ARPABET).
2. Segmentation Model:
Identifies phoneme boundaries in audio data, annotating the training set with start and end
points for each phoneme.
3. Phoneme Duration Model:
Predicts the duration of each phoneme in a sequence.
4. Fundamental Frequency (F0) Model:
Predicts whether a phoneme is voiced and estimates its fundamental frequency (F0)
throughout its duration.
5. Audio Synthesis Model:
Combines phoneme, duration, and F0 information to synthesize the final speech output at a
high sampling rate.
Inference Process:
● During inference, text is converted to phonemes, durations and F0 are predicted, and the
audio synthesis model generates the final speech output.
Training Process:
● The segmentation model annotates training data with phoneme boundaries, which are used to
train the phoneme duration and audio synthesis models.
Grapheme-to-Phoneme Model:
Our grapheme-to-phoneme model uses an encoder-decoder architecture with a bidirectional GRU
encoder and a unidirectional GRU decoder. The encoder has 3 bidirectional layers of 1024 units,
while the decoder has 3 unidirectional layers of the same size. The decoder's initial state is set to the
final hidden state of the encoder. We train the model using teacher forcing and use beam search with a
width of 5 during decoding. Dropout (0.95) is applied after each recurrent layer during training to
prevent overfitting.
Segmentation Model:
Our segmentation model aligns an utterance with a sequence of target phonemes, similar to speech
recognition tasks. We adapt the convolutional recurrent neural network (CRNN) architecture from a
state-of-the-art speech recognition system (Amodei et al., 2015) for phoneme boundary detection.
Trained with the CTC loss function, the network outputs phoneme pairs instead of single phonemes to
detect precise phoneme boundaries. We encode labels as consecutive phoneme pairs, like "sil HH, HH
EH, EH L," etc. Input audio is processed into 20 MFCC features, and the network consists of
convolutional layers, bidirectional GRU layers, and a softmax output. During decoding, we use beam
search with a width of 50 and ensure that neighboring phoneme pairs overlap. Dropout (0.95) is
applied after the last convolution and recurrent layers.
Phoneme Duration and Fundamental Frequency Model:
The model predicts phoneme duration and time-dependent fundamental frequency (F0) using
a single architecture. It takes a sequence of phonemes with stress information, encoded as
one-hot vectors, as input. The model consists of two fully connected layers with 256 units,
followed by two unidirectional GRU layers with 128 cells each. The output layer generates
three predictions for each phoneme: duration, voicing probability, and 20 sampled F0 values.
Dropout is applied after the fully connected and recurrent layers. The model is optimized by
minimizing a joint loss that includes duration error, F0 error, voicing probability loss, and a
smoothness penalty for F0. This approach ensures natural timing and pitch in speech
synthesis.
Audio Synthesis Model:
The audio synthesis model is a variant of WaveNet, consisting of a conditioning network and
an autoregressive network. The conditioning network upsamples linguistic features, while the
autoregressive network generates a probability distribution over discretized audio samples.
The model's architecture is flexible, varying the number of layers, residual channels, and skip
channels. It includes an upsampling and conditioning network, followed by two 1D
convolution layers with residual output channels and gated tanh nonlinearities. The hidden
states are concatenated and projected to skip channels, followed by additional convolution
layers with ReLU activations. For improved performance, faster training, and fewer
parameters, the model uses bidirectional quasi-RNN layers for encoding and upsampling by
repetition instead of transposed convolutions.
6. Results
The implemented code focuses on developing a Grapheme-to-Phoneme (G2P) conversion model
using deep learning techniques. It begins by setting up a character table that maps characters to their
respective indices, which is crucial for encoding and decoding the input data. The model architecture
features a bidirectional Gated Recurrent Unit (GRU) network, designed to process sequences of
characters and predict their corresponding phonemes. The training utilizes a dataset from the CMU
Pronouncing Dictionary, which contains words and their phonetic transcriptions. During the training
process, the labels are converted into a sparse format, and the model is optimized using the Nadam
optimizer. The results indicate that the model achieved an accuracy of approximately 81.47% after
two epochs, demonstrating its effectiveness in learning the mapping from graphemes to phonemes.
In the evaluation phase, the model generated predictions for several words, comparing the predicted
phoneme sequences with the true phonemes. For example, the word "catalogs" was predicted as "K
AE1 L AH0 AH0 Z," while the actual phoneme representation was "K AE1 T AH0 L AO0 G Z." This
illustrates the model's capability to produce phonetic representations that closely align with actual
pronunciations, although some discrepancies were observed, such as with the words "burchall" and
"figment." Overall, the results highlight the model's potential for accurately converting written text
into phonetic forms, which can be particularly beneficial for applications in speech recognition and
synthesis.
7. Optimizing Inference
WaveNet has shown impressive capabilities in generating high-quality synthesized speech, but its
inference presents significant computational challenges. Due to its autoregressive nature, WaveNet
requires many more timesteps than traditional recurrent neural networks, leading to long generation
times. For example, generating one second of audio with a 40-layer model requires approximately 55
billion floating point operations (FLOPs). With such high computational demands, a single layer of
the model, which involves several matrix multiplies and nonlinearities, must complete in about 1.5
seconds to meet real-time requirements. The model, which has around 16 million parameters, takes up
about 6.4 MB of memory when stored in single precision. In practice, this means that inference needs
to be optimized to avoid redundant calculations and to utilize the processor cache efficiently, as even
accessing memory on a CPU can take several tenths of a second.
Optimizing inference on both CPUs and GPUs involves addressing the high computational and
memory requirements. For CPUs, a single core must operate close to its peak single-precision
throughput of 77 GFLOPs and utilize 70% of the available memory bandwidth to achieve real-time
inference. However, this can be improved by distributing the workload across multiple cores, although
synchronization remains challenging. On GPUs, which offer higher memory bandwidth and peak
FLOPs, a naive implementation that runs a single kernel for every layer or timestep is inefficient.
Instead, a persistent RNN technique can better leverage GPU throughput. Through the implementation
of optimized inference kernels for both CPU and GPU, faster-than-real-time WaveNet inference is
achievable, with the CPU kernels outperforming the GPU models in terms of speed in some cases.
These benchmarks focus on autoregressive audio generation and do not include the parallel generation
of linguistic features.
CPU Implementation
We achieve real-time CPU inference by optimizing memory access, parallelizing tasks with
multithreading, minimizing nonlinearity FLOPs, avoiding cache thrashing, and using custom
hardware-optimized routines. The CPU implementation breaks the computation into steps, beginning
with sample embedding, where two embeddings for the current and previous timesteps are computed
and summed with a bias. Efficient synchronization and thread pinning ensure optimal performance.
Sample Embedding:
The sample embedding step involves computing the WaveNet input causal convolution by creating
two sample embeddings: one for the current timestep and one for the previous timestep. These
embeddings are then summed together with a bias to form the final input for the model.
2. LayerInference:
For every layer j from j=1 to with dilation width:
(a)Compute the left half of the width-two dilated convolution via matrix-vector multiply:
(b)Compute The Right Half Of The Dilated Convolution
(c)Compute the hidden state h(j) given conditioning vector L(j) h :
Where v0:r denotes the first elements of the vec to rand vr:2r denotes then extra [Link],
computetheinputtothenextlayervia matrix vector multiply:
(d)Compute the contribution to the skip-channel matrix multiply from this layer, accumulating
overall players,with q(0)=Bskip:
[Link]:Compute The Two Output 1 1 convolutions:
We parallelize the computation across two thread groups: one for main threads that handle tasks like
computing x(0), a(j)cur, h(j), x(j), za, and p, and another for auxiliary threads calculating a(j)prev, q(j),
and zs. The main threads compute zaand p, while auxiliary threads precompute values for the next
timestep. Threads can be single or multi-threaded, with each thread handling a block of matrix-vector
multiplication or other operations, synchronized as needed. Pinning threads to physical cores
improves performance by 30%, preventing thread contention and cache thrashing.
We replace nonlinearities with high-accuracy approximations, achieving a 30% performance boost
with minimal error. Quantizing weight matrices to int16 speeds up larger models, though thread
synchronization overhead limits its effectiveness with more threads. Custom AVX assembly kernels,
specialized for our matrix sizes using PeachPy, accelerate inference, providing up to 1.5x faster
performance than Intel MKL and 3.5x faster than OpenBLAS for float32 operations.
GPU Implementation
GPUs are often used for neural models due to their high computational throughput, but for memory
bandwidth and FLOP-bound models like ours, there are challenges. Typically, each matrix multiply or
vector operation is handled by a separate kernel, but the latency of launching CUDA kernels and
loading the model into GPU memory makes this approach about 1000x slower than real-time.
To achieve near-real-time inference, we use persistent RNNs, which generate all output samples in a
single kernel launch. The model weights are loaded into registers once and used throughout inference.
However, this approach requires highly specialized kernels for specific model sizes, making
development labor-intensive. While GPU inference isn’t yet real-time, we believe that further
optimization can enable real-time WaveNet inference on both GPUs and CPUs.
8. Conclusion
In this work, we demonstrate that current deep learning approaches are effective for building a
high-quality, fully neural text-to-speech (TTS) system. We optimize inference to achieve
faster-than-real-time speeds, enabling real-time, streaming audio generation. Our system is trainable
without human intervention, significantly simplifying TTS system creation. The work opens new
avenues for improvement, such as further optimization of inference performance, GPU model
quantization, and int8 quantization on CPUs. Exploring architectures like Xeon Phi and merging
separate TTS stages into a unified sequence-to-sequence model could lead to a fully end-to-end
trainable system. Additionally, enhancing duration and frequency models with larger datasets or
generative techniques may further improve voice naturalness.
9. References
Mehri, Soroush, Kumar, Kundan, Gulrajani, Ishaan, Ku mar, Rithesh, Jain, Shubham, Sotelo, Jose,
Courville, Aaron, and Bengio, Yoshua. Samplernn: An uncondi tional end-to-end neural audio
generation model. arXiv preprint arXiv:1612.07837, 2016.
Morise, Masanori, Yokomori, Fumiya, and Ozawa, Kenji. World: a vocoder-based high-quality
speech synthesis system for real-time applications. IEICE TRANSAC TIONS on Information and
Systems, 99(7):1877–1884, 2016.
Oord, Aaron van den, Kalchbrenner, Nal, and Kavukcuoglu, Koray. Pixel recurrent neural networks.
arXiv preprint arXiv:1601.06759, 2016.
Paine, Tom Le, Khorrami, Pooya, Chang, Shiyu, Zhang, Yang, Ramachandran, Prajit,
Hasegawa-Johnson, Mark A, and Huang, Thomas S. Fast wavenet gen eration algorithm. arXiv
preprint arXiv:1611.09482, 2016.
Pascual, Santiago and Bonafonte, Antonio. Multi-output rnn-lstm for multiple speaker speech
synthesis with interpolation model. way, 1000:2, 2016.
Group Members:
Pavan Kumar 2110110035
Mohammad Adil 2110110924
Mohammad Danish 2110110334