Speaker Fluency Level Classification Using Machine Learning Techniques

Alan Preciado-Grijalva∗, and Ramon F. Brena

Grupo de Investigación en Sistemas Inteligentes, Tecnológico de Monterrey, Monterrey, NL, Mexico

Level assessment for foreign language students is necessary for putting them in the right level group, furthermore,
interviewing students is a very time-consuming task, so we propose to automate the evaluation of speaker fluency level
by implementing machine learning techniques. This work presents an audio processing system capable of classifying
the level of fluency of non-native English speakers using five different machine learning models. As a first step, we
arXiv:1808.10556v1 [[Link]] 31 Aug 2018

have built our own dataset, which consists of labeled audio conversations in English between people ranging in different
fluency domains/classes (low, intermediate, high). We segment the audio conversations into 5s non-overlapped audio
clips to perform feature extraction on them. We start by extracting Mel cepstral coefficients from the audios, selecting
20 coefficients is an appropriate quantity for our data. We thereafter extracted zero-crossing rate, root mean square
energy and spectral flux features, proving that this improves model performance. Out of a total of 1424 audio segments,
with 70% training data and 30% test data, one of our trained models (support vector machine) achieved a classification
accuracy of 94.39%, whereas the other four models passed an 89% classification accuracy threshold.

toring during a conversation between two or more persons.

Our approach to this problem is based on audio analysis,
1. Introduction
starting with audio feature extraction and afterwards train-
ing machine learning (ML) models to perform classification
The development of artificial intelligence (AI) - powered ap-
of audio segments provided labeled target classes. Previ-
plications has been growing remarkably over the last decade
ously, there have been advancements in environmental sound
[1][2]. With regards to language learning apps, there are cur-
classification [4] and real-time speech recognition based on
rently several software language companies that are employ-
neural networks [5]. In these cases, the audio sets have been
ing AI techniques to improve user engagement and learn-
environmental sounds (rain, cars, birds, etc.) and recorded
ing experience. The main promise of AI-powered language
speech, music and noise sounds, respectively. In our case,
learning apps is that users will achieve basic proficiency in
to approach the general problem of fluency level monitoring
a foreign language as they progress through their lessons
of each individual during a conversation, we have first pro-
within a few months and with a small amount of time study-
ceeded to build our own audio set (Avalinguo audio set), the
ing per day, all being guided by AI.
details of the audio set are presented in section 2. There-
A slightly different language learning scenario is the one after, we have split each conversation in 5s non-overlapped
involving two or more (known or unknown) persons who segments, these segments have had some features extracted
are actively looking for tandem groups to improve their lan- (mel coefficients + zero crossing rate + root-mean-square-
guage skills. Currently, our group at our university, the energy + spectral flux). Later on, the feature vectors are
Tecnológico de Monterrey (ITESM), is working on the con- fed into a classification model to train it and evaluate its
struction of an AI-powered mobile app for language learn- performance using accuracy metrics. Our defined fluency
ing called Avalinguo. Avalinguo is an internet-based sys- classes are three: low fluency, intermediate fluency, and high
tem, and it merges virtual reality with AI to create “digital fluency. We have compared five ML models, namely, multi-
classrooms” in which people, each one with a corresponding layer perceptron (MLP), support vector machines (SVM),
avatar, can practice a language [3]. random forest (RF), convolutional neural networks (CNN)
Avalinguo has many benefits such as 1) users are not at- and recurrent neural networks (RNN). The workflow de-
tached to fixed schedules, 2) it is portable and can be used scribed previously is the standard ML approach for the audio
anywhere (internet provided), 3) real-time real-person inter- analysis of sound events [6].
action, 4) user privacy is kept because of the use of avatars,
5) it clusters users based on profiles (target language, inter- The main question that we are trying to answer here is:
ests, etc.), 6) due to clustering, each login presents new pos- Given a labeled balanced audio set fulfilling predefined flu-
sible matches with other users (recommendation system), 7) ency metrics, can we construct a model capable of classifying
it contributes to a relaxed and casual participation by im- the level of fluency of an audio segment?
plementing fluency monitoring and topic recommendation If so, this would allow us to determine whether a group
during conversations and 8) its cost is inferior to particular in a conversation needs a recommended topic to keep it flow-
online courses. ing. Also, we would be able to tell if Person A has a lower
In this paper, we report work done related to point 7 by fluency than Person B and needs to be re-assigned to a lower
presenting the advancements corresponding fluency moni- fluency level group (same for the inverse case).
∗ Corresponding author: apreciado42@[Link]

1
 Our final classification results have achieved accura- have sample rates ranging from 22050 Hz to 48000 Hz, are
cies higher than 90% (except for one model), being the mono and multi-channel and were converted to MP3 format.
highest of up to 94.39% for an SVM. As a first step, we Figure 1 shows the class distribution of the audio set.
have determined the appropriate number of Mel coefficients The intermediate class has a higher percentage because,
(MFCCs) extracted to ensure high accuracies. Thereafter, technically, it is easier to collect audios of people ranging
we’ve proved that adding features to the baseline MFCCs in the intermediate level rather than the low level. Despite
such as zero-crossing-rate (ZCR), root-mean-squared-energy this percentage differences, we have kept all audio files to
(RMSE) and spectral flux onset strength envelope (SF) in- avoid reducing the size of the data set. Note that this can
creased overall model performance. be considered a small audio set (ca. 2 hours) if it is com-
pared with some public sets (such as Google AudioSet).

2. Avalinguo audio set

Having a clean-high quality data set is a must for any ma-
chine learning project. The main challenge for a model’s
architecture is to be able to grasp patterns among data so
that it can be complex enough to perform accurate predic-
tions/classifications. This can be affected if mislabeled or
missing data is contained in the data set. On the other side,
the technical limitations of a model have to do with the com-
putational power available and the amount of labeled data
required for appropriate training.
Unfortunately for us, there are no publicly available data
sets that fulfill our research purposes. Whilst the commu-
nity has been working extensively on the construction and
Figure 1: Avalinguo audio set class distribution. The set
support of audio sets [8], it has not been possible for us to
contains a total of 118.65 minutes of recorded audio.
address an audio set for audio-speech analysis composed of
conversations by non-native English speakers. There are in-
deed audio corpora of people speaking English such as the 2.1 Fluency metrics
UCSB and MUSAN [9-10], but no audio set of people who
are actually learning the language (by this we mean people Previously, we mentioned that each audio segment was la-
who hesitate when speaking, people who take long pauses beled manually. In order to do this, we have defined baseline
when speaking, or also people who just speak too slowly). fluency levels definitions:
We are mostly interested in recordings of this kind be- Low 0: Person uses very simple expressions and talks
cause the Avalinguo system will deal mainly with non-native about things in a basic way. Speaks with unnatural pauses.
English speakers who will present the speaking characteris- Needs the other person to talk slowly to understand.
tics mentioned before. Due to this reason, we have decided Low 1: Person can understand frequently used expres-
to build our own audio set; Avalinguo audio set [11]. The sions and give basic personal information. Person can talk
Avalinguo audio set is a collection of audio recordings of peo- about simple things on familiar topics but still speaks with
ple whose language fluency ranges from low to high. The unnatural pauses.
audio recordings have the next common characteristics: Intermediate 2: Can deal with common situations, for
example, traveling and restaurant ordering. Describes expe-
• Spontaneous (non-scripted) conversations riences and events and is capable of giving reasons, opinions
or plans. Can still make some unnatural pauses.
• Random conversation topics chosen by speakers
Intermediate 3: Feels comfortable in most situations.
Can interact spontaneously with native speakers but still
• Audios recorded with low-to-no background noise
makes prolonged pauses or incorrect use of some words. Peo-
The sources who provided the audio recordings are three: ple can understand the person without putting too much
Friends/Family, Language Center (ITESM), and Youtube effort.
Audios. Each audio recording comprises a conversation that High 4: Can speak without unnatural pauses (no hesi-
lasts around 10 minutes and has a wide range of topics from tation), doesn’t pause long to find expressions. Can use the
athletes speaking to physicists talking about science to sis- language in a flexible way for social, academic, and profes-
ters talking about their daily activities. All of these 10 min- sional purposes.
utes conversations were cut in 5s equally-sized segments and High 5: Native-level speaker. Understands everything
were thereafter manually labeled and assigned into one of that reads and hears. Understand humor and subtle differ-
the three fluency classes. Summarizing, the audio set con- ences.
sists of 1420 (5s duration) non-overlapped audio segments There is no single-universal definition for fluency. Actu-
comprising three fluency classes (low, intermediate and high ally, each language institution establishes a fluency metric
fluency). This is about 2 hours of recordings. The audios for scoring based on their internal parameters. In our case,

2
to score speaker fluency, we have taken the baseline defini- speaks more in it. After this, we proceed to label frames ac-
tions described above and have made specific emphasis on cording to their fluency level. The segments were cut from
the next points regarding the concept: the audios using a python module called Pydub 1 .
1) Our metric is mainly sound based. With "fluent" Due to the way we proceeded, we didn’t work with over-
meaning speaking without unnatural pauses. lapped segments; since we are interested in differentiating
persons among a conversation, this approach would have
2) If there is hesitation (slowness or pauses) when speak- had ended as non-overlapped segments but with shorter time
ing, then that affects the fluency score of the speaker. durations. The other approach that we explored was using
voice activity detection (VAD), this approach creates audio
3) There is a distinction between fluency and proficiency. segments when it detects a different voice or a pause during
Meaning that fluency is someone able to feel comfortable, a conversation. However, the VAD interface [15] suppresses
sound natural, and with the ability to manipulate all the any possible silence within the conversation, that is, it only
parts of a sentence at will. creates segments when people are speaking. Since we are
interested in detecting possible silences and pauses in each
audio segment, we have discarded this approach.

3. Experimental Framework 3.2 Feature Extraction

This is the step where features are extracted. We have writ-
Our experimental procedure is the standard ML approach ten a python script to perform feature extraction for the
for analysis of sound events [12]. Our system consists of created audio segments using a python package called Li-
three main steps (see Figure 2 ): Feature extraction, classi- brosa 2 (among many functionalities, Librosa is commonly
fication and output of the predicted label with higher prob- used for feature extraction, allowing to compute more than
ability for individual segments (frames). First, we perform thirty audio features). The audio frames f0 , ..., fn , have thus
feature extraction on each audio frame. Thereafter, the fea- their corresponding feature vectors p0 , ..., pn .
ture vector is fed into a classifier which outputs the most For the results presented in this paper, we have first var-
probable class based on previous training. ied the number of MFCCs extracted, this with the purpose
to estimate the appropriate number of MFFCs to extract.
Later on, we have added the ZCR, RMSE and SF to deter-
Input Audio

mine if this improves model performance.

Besides feature extraction, Librosa allows to plot audio
spectrograms using its [Link] function. For the
sake of completeness, we show some commonly spectrograms
Feature extraction for a single audio frame (see Figure 3 ).

Classification

Low
System Output

Medium

High

Figure 2: Pipeline of an audio classification system.

3.1 Audio segmentation Figure 3: Spectral feature plots (single frame).

Audio segmentation is one of the most important preprocess-
ing steps in most audio applications [13]. Research groups
3.3 Audio Classification
often propose novel segmentation frameworks in order to
improve audio applications such as speech recognition [14]. In this work we are interested in comparing different classi-
In our case, our segmentation method simply consists on fication frameworks, namely, multilayer perceptrons, convo-
cutting the audio files into 5s non-overlapped segments and lutional neural networks, recurrent neural networks, support
manually assign a frame to a single person (for example, vector machines and random forest. The main goal here is
Frame 1 belongs to person A). If a frame contains more to test each model classification accuracies as we vary the
than one person speaking, we assign it to the person that number of features extracted.
1 Pydub: High level python interface for audio manipulation
2 LibROSA: python package for music and audio analysis.

3
 The proposed multilayer perceptron architecture has two it is our duty to find out the trade-off between the appropri-
hidden layers comprising 512 x 512 neurons followed by an ate Nmel and the maximum achievable accuracy. There is a
output layer consisting of three neurons (one representing point in which adding more Nmel doesn’t translate into con-
each fluency class). Each neuron within the hidden layers siderable improvements (this can either increase accuracy by
uses the relu function as the activation function. For the a small percentage or decrease it). With this experiment, we
output layer, we implement the softmax function to convert are able to get the appropriate Nmel for our data set.
the output into class probabilities. Finally, the predicted As a second step, we show how adding features such as
label is the class with the highest probability. ZCR, RMSE and SF to the baseline chosen Nmel boosts
The convolutional neural network architecture has four accuracy in most of the cases.
hidden layers, the first two have 64 convolution filters and
the following two have 32 convolution filters. The output 4.1 Classification Experiments
layer consists of three neurons corresponding to our three
classes and similarly to the multilayer perceptron, the acti- We have chosen SVM and RF as our models based on re-
vation function of the hidden layers is the relu and for the search regarding the most appropriate ML approaches for
output layer is the softmax function. audio classification. Both of these models have proven to be
The recurrent neural network architecture is a long- good candidates for the classification of sound events, such
short-therm-memory (LSTM) [18]. It has two hidden layers as the ECS-50 audio set, as proposed by Piczak [17].
comprising 256 x 32 neurons and three neurons correspond- We are also comparing three different neural network
ing to the output layer. This architecture also implements models; the MLP architecture has proven to classify accu-
the same activation functions as the other two networks. rately speech, audio and noise audio of the MUSAN audio
In table 1 we summarize the architectures described set as reported by Wetzel et al. [5], CNNs have also been
above. used to classify the ECS-50 audio set [4] and have also been
used to classify audio without performing prior feature ex-
Neural Network Hidden layers Neurons Activation traction, in the sense that the network itself extracts corre-
MLP 2 512x512x3 relu, softmax sponding features from the waveform sample [12], lastly, we
CNN 4 64x64x32x32x3 relu, softmax have also employed RNNs because according to Huy Phan
RNN 2 256x32x3 relu, softmax et al., this type of neural networks achieved an accuracy of
97% in the classification of sound scenes from the LITIS
Table 1: Neural networks architectures. Rouen dataset [19].
All our models were randomly sampled with 70% train-
The other two models are traditional machine learning ing data and 30% test data. Given the 1424 total audio
models. One is a support vector machine. This model has a frames of the Avalinguo audio set, this corresponds to 926
basic construction (similar to the one proposed in the scikit- audio frames for training and 498 audio frames for testing.
learn documentation). The main hyper-parameter of the es- In our first experiment, we have trained and evaluated
timator that we varied was the regularization parameter C. accuracy performance with increasing values of Nmel . Ta-
3 ble 2 contains the achieved accuracies as we set Nmel =
The other model uses a random forest classifier. This 5, 10, 12, 20. In each case, the accuracy improved consider-
model also has a basic construction in the sense that our ably as the value Nmel increased. We obtained an accuracy
python script only initializes the model, trains it and then as high as 94.39% with the SVM for Nmel = 20. We also
evaluates its performance. The single parameter we varied trained our models with Nmel = 30, 40 but this only in-
here was the number of estimators (number of trees).4 creased feature space dimensionality but not accuracy. From
this experiment, we see that the commonly number of Mel
coefficients used for audio-analysis (12 to 20) applies to our
4. Experimental Results data as well.

In this section we present our feature extraction and classi- Model 5 10 12 20

fication results. The main data analysis tools that we have SVM 86.00% 89.49% 92.06% 94.39%
used for our experiments are Anaconda (under Python 2.7), RF 84.80% 89.00% 90.42% 92.29%
Keras (backend Tensorflow), Scikit-learn, Librosa, Pydub MLP 78.00% 88.78% 89.01% 92.05%
and Pandas. CNN 80.00% 85.04% 87.61% 93.69%
We have evaluated different model constructions, here, RNN 78.90% 85.04% 86.44% 87.00%
we report the model architectures (from Table 1 ) and hyper-
parameters that achieved the highest classification accura- Table 2: Accuracy performance of the classification models
cies. for different Nmel values.
As a first step, we explore the effect of varying the num-
ber of Mel coefficients (Nmel ) on the final accuracy. Increas- The second stage consists in taking the Nmel = 20 as
ing the value Nmel increases the complexity of a model, thus, baseline features and test extra spectral features to see the
3 Link to SVM scikit-learn documentation.
4 Link to RF scikit-learn documentation.

4
outcome. After doing feature exploration in our runs, we The neural networks have slightly underperformed com-
have ended up adding ZCR (as proposed in [5]), as well as paring them with the other two models. But they have
RMSE and SF (as proposed in [7]). This translates to a 23- achieved decent accuracies as well. The main difference be-
dimensional feature space (20 MFCCS + 1 ZCR + 1 RMSE tween the deep learning and the traditional models has to
+ 1 SF). do with the time it takes them to train, requiring the latter
The architecture of our models combined with the total way less computational time. The under-performance of the
final features yields a runtime of about 1 ms per classifica- neural networks does not exclude them from this analysis
tion on a 2.4 GHz single core CPU for the neural networks. at all, it can be that a different architecture can boost their
For the SF and RF, it takes about two seconds to train accuracies.
completely. In order to evaluate the quality of the output of the clas-
Table 3 shows the obtained accuracies with the extra fea- sifiers we have used a confusion matrix. The corresponding
tures. Once again, the SVM achieved the highest accuracy map can be seen in Figure 5 and it belongs to the SVM
followed by the RF. The MLP and CNN obtained similar re- results, which trained to classify among our three fluency
sults, the one big difference is that the MLP outperformed levels, has achieved a classification accuracy of 94.39%. The
the CNN in computing time; the former took about 1min matrix was plotted using the [Link] module from
to train completely, whereas the latter took about 5 mins scikit-learn [20]. From the plot, we see that the classifier
to train completely. The RNN obtained the lowest accuracy actually predicted all the high labels (classes) correctly. For
and it took about 7 mins to completely train. the intermediate label, it misclassified 15 audio frames ei-
In contrast with the results from table 2 (case Nmel = ther as a low or high label, this is understandable since, in-
20), the results in table 3 (Nmel = 20 + extra features) tuitively, it is harder to discriminate if a frame lies in the in-
show that the performance of the SVM remained equal, the termediate level or if it belongs to any of its “neighbors”. For
CNN performance decreased by around 1% and the other the low label results, the SVM predicted two audio frames
three models increased their accuracy. We present graphi- as highly fluent whilst they belong to the low fluency class,
cally this comparison with a bar plot in Figure 4. this could be the “most critical” mistake our model has done
by predicting that two audios whose fluency is low, actually
Features SVM RF MLP CNN RNN
has an almost native fluency. However, in the overall, the
Nmel +ZCR+RMSE+SF 94.39% 93.45% 92.52% 92.75% 89.01%
model has performed remarkably.
Table 3: Accuracy performance of the classification models
20 MFCCs + extra features.

Figure 4: Accuracy comparison when using Nmel = 20 (blue

bars) and with extra features (red bars). For sake of visual-
ization, the accuracy of the plots starts at 80%.

With these results, we have that the SVM followed by Figure 5: Confusion matrix for the SVM trained and tested
the RF are the models that best classify our data set. The with the Avalinguo audio set.
reason why the SVM couldn’t improve further with the extra
features can have to do with the design of the model itself.
In this case, we varied the choice of the regularization pa- The link to the repository with the Python script to repli-
rameter C but couldn’t obtain any better performance. In cate this paper can be found in [16]. All the technical re-
the case of the RF, we increased the number of trees when quirements to run the code are documented in the attached
we added the other features, gaining more than 1% accuracy. repository.

5
5. Conclusions [3] Avalinguo Project Link

In this work, we have presented an audio processing system [4] Karol J. Piczak. Environmental Sound Classification
capable of determining the level of fluency of non-native En- with Convolutional Neural Networks. IEEE Interna-
glish speakers, taken 5s non-overlapped audio segments from tional Workshop On Machine Learning For Signal Pro-
the Avalinguo audio set. We have used five different ML cessing (2015). Boston, USA.
models to classify audio segments into low, intermediate or [5] Micha Wetzel, Matthias Sperber and Alexander
high fluency levels. Each model was capable of classifying Waibel. Audio Segmentation for Robust Real-Time
audio frames with an accuracy of more than 90% (except Speech Recognition Based on Neural Networks (2016).
one classifier that reached 89%).
As a first step, we have determined that the appropri- [6] Toni Heittola, Emre Çakır and Tuomas Virtanen. The
ate number of Mel cepstral coefficients for our data set is Machine Learning Approach for Analysis of Sound
20. Thereafter, with these baseline features, we have added Scenes and Events. Springer International Publishing
zero-crossing rate, root mean square energy and spectral flux AG (2018).
features to improve the accuracy of our models. The high-
est accuracies were reached by SVM and RF, with 94.39% [7] Ashutosh Kulkarni, Deepak Iyer and Srinivasa Rangan
and 93.45%, respectively. The neural networks achieved also Sridharan. Audio Segmentation. Stanford University.
remarkable accuracies (MLP 92.52%, CNN 92.75%, RNN [8] Iulian Vlad Serban, Ryan Lowe, Peter Henderson, Lau-
89.01%). rent Charlin and Joelle Pineau. A Survey of Available
We have also reported the construction and details of the Corpora for Building Data-Driven Dialogue Systems.
Avalinguo audio set, whose main characteristic is that it is arXiv preprint arXiv:1512.05742v3 (2017).
composed by conversations of people who are learning the
English language. [9] UC Santa Barbara Corpus of Spoken American English
The accuracies that we have achieved can be considered
[10] David Snyder, Guoguo Chen and Daniel Povey. MU-
high but nonetheless there is room for improvement. For
SAN: A Music, Speech, and Noise Corpus. In: Com-
example, tunning more precisely the hyper-parameters of
puting Research Repository (2015).
the SVM and RF estimators by running grid searches. In
the case of the neural networks architecture, we can still ex- [11] Github repository of the Avalinguo Audio Set
plore adding specific hidden layers and modifying the num-
ber of neurons per layer. Added to this, as other works [12] Maxime Jumelle and Taqiyeddine Sakmeche. Speaker
have proposed, exploring with other audio features such as Clustering With Neural Networks And Audio Process-
chromagram, Mel spectrograms or spectral contrast can im- ing. arXiv preprint arXiv: 1803.08276v1 (2018).
prove accuracy. We must take into account that we are only
[13] David Rybach, Christian Gollan, Ralf Schlüter and
defining three fluency classes, in order to make fluency levels
Hermann Ney. Audio Segmentation for Speech Recog-
more specific, we would have to define more fluency classes
nition Using Segment Features. ICASSP (2009).
in between. This poses a challenge for the accuracy perfor-
mance. [14] Manpreet Kaur and Amanpreet Kaur. A Review: Dif-
The main technical limitation that we have right now is ferent methods of segmenting a continuous speech signal
that the Avalinguo audio set (with about 2 hrs of record- into basic units. International Journal Of Engineering
ings) can be considered a small set. Part of our future work And Computer Science (2013).
consists in maintaining and increasing the size of the audio
set. Another technical challenge consists in automatically [15] Link to Python interface to the WebRTC Voice Activity
identifying persons within a conversation and at the same Detector.
time, capturing the silences and pauses they can make.
[16] Link to code for paper replication (github).
This project will be integrated to the Avalinguo system.
Here, we will have to deal with the classification of live con- [17] Karol J. Piczak. ESC: Dataset for Environmental
versations, for example. Sound Classification. Proceedings of the 23rd ACM in-
ternational conference on Multimedia (2015).
[18] Sepp Hochreiter and Jurgen Schmidhuber. LONG
References SHORT-TERM MEMORY. Neural Computation
[1] David W. Cearley, Brian Burke, Samantha Searle and 9(8):1735-1780 (1997).
Mike J. Walker. Top 10 Strategic Technology Trends for
[19] Huy Phan, Philipp Koch, Fabrice Katzberg, Marco
2018 (2017). Gartner, Inc. Maass, Radoslaw Mazur and Alfred Mertins. Audio
Scene Classification with Deep Recurrent Neural Net-
[2] Robert Spangenberg, Daniel Goehring and Raúl Rojas. works. arXiv preprint arXiv:1703.04770v2 (2017).
Pole-based Localization for Autonomous Vehicles in Ur-
ban Scenarios. IEEE/RSJ International Conference on [20] Pedregosa et al., Scikit-learn: Machine Learning in
Intelligent Robots and Systems (IROS) (2016). Python, JMLR 12, pp. 2825-2830 (2011).

6
Acknowledgments tributed with the construction of the dataset.
We gratefully acknowledge the support by the Language
Center at ITESM and the individual persons who con-