CRNN
Algorithm
P R E S E N T E D BY: M O H A M E D A M I N E C H A D L I .
S U P E RV I S O R : D R. B A C H I R B O U I A D J R A R O C H D I
C O - S U P E RV I S O R : D R. F E K I R A B D E L K A D E R
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 1
Summary
Problematic
Introduction
State of the art
The proposed network architecture
Experiments
Implementation details
Comparative study
Results
Conclusion and Perspective
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 2
Problematic
Image-based sequence recognition is one of the major topic research in the
computer vision. In this work, we investigate the problem of text recognition
which is among the most challenging tasks in the field of pattern recognition
because of it’s sequential nature. In this work, we introduced a novel deep
learning approach, which has four distinctive features over the state of the art
pre-existing systems that we are going to illustrate further in this presentation.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 3
Introduction
In the recent years computer vision community has drastically focused their
efforts on deep learning models. And one specific model has proven itself as an
efficient model to automatically extract features from images which is the
Convolutional Neural Network (CNN) model. But the CNN is designed to handle
fixed length input and fixed length output predictions. Therefore, CNN cannot
naturally handle variable length inputs and outputs like the task of text
recognition. What makes the task of text recognition even harder than classic
recognition problems, is that we do not have insight on the output length of the
prediction. Therefore, we need to bring unusual solutions to solve unusual
problems.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 4
State of the art
Some attempts have been made to tackle this issue of variable length input and
output that characterize image-based sequence recognition:
Some algorithms first try to detect the individual characters and apply a CNN
recognition on each detected character. Therefore, we need a dataset that have
character segmentation in it’s core. (We know that segmenting an entire dataset
on the character level is a hard work and it needs a lot of effort to be made).
Developing such kind of systems will require dividing the system into two parts;
the segmentation part and the recognition part and this two parts are trained
separately.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 5
State of the art
Other algorithms treats this task like as it is an image classification problem. It
gives a class to each unique word in English vocabulary. These types of systems
will rise a serious problem of the size of the deep learning model. English
vocabulary consists of 90k words. Therefore we need 90k class in our model to be
able to represent all English words. These systems are best suited for domain
specific recognition systems where the number of words in that vocabulary is
limited.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 6
The proposed
network
architecture
Our proposed network architecture is
inspired by speech recognition
systems. We spotted some
similarities between speech
recognition and image-based
sequence recognition. They are both
a sequence-based tasks. They are
both variable length problems. They
are both ordered type of
classification (The output predictions
are ordered and corresponds to
some inputs, respectively).
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 7
1-Feature
sequence
extraction
In this part, we use a CNN
designed specifically to extract
sequence features from the
input images where the input is
an image and the output of the
CNN is a sequence of features
maps where each features map
frame in the sequence
represents a region in the
corresponding input image as
follow:
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 8
2-Sequence
labeling
In this part, we used a deep bi-
directional recurrent neural
network, specifically LSTM
network to predict the character
label for each features map frame
given by the previous CNN part.
RNN networks are best known for
their ability to learn from the
context of the training examples.
We used bidirectional-LSTM to
learn also from the future context
not only from the past one.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 9
3-Transcription
In this part, we use a technique called Connectionist Temporal Classification
(CTC), most know in speech recognition systems to align the per-frame
predictions made by the Bi-LSTM network from the previous step to the ground
truth labels of the training examples. Technically it’s a loss function for our model
that can handle temporal classification.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 10
Experiments
We experimented our approach on a famous synthetic dataset of English words.
We used this dataset to train our model. We used several famous benchmarks to
evaluate our model such as: ICDAR 2003 dataset and ICDAR 2013 dataset.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 11
Implementatio
n details
The architecture of our
model is as follow:
The architecture of the
convolutional layers is based
on VGG deep network.
We designed this CNN to
output values corresponding
to regions in the input
images.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 12
Implementatio
n details
The output of this CNN
architecture is a 25 units of
features corresponding to
regions in the input image
and this is done if we
consider the input image
size of 32 pixels height and
100 pixels width.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 13
Implementatio
n details
These 25 units of features will be
fed the next part of the model
(Bi-LSTM). This LSTM network will
do the actual predictions of
characters. Therefore in this
case, it will predict 25 characters.
Most English words won’t exceed
25 characters. Therefore this
architecture will be ready to be
used in predicting any English
word.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 14
Implementatio
n details
The last step is the transcription
layer. In this step we fed the
predicted characters from the
previous step to the CTC loss
function or CTC decoder; it
depends on the context if it’s
training or evaluating.
The CTC will take care of aligning
the sequence to a valid English
word by using a dictionary or free
prediction without a dictionary.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 15
Comparative
study
Our system has a four distinctive
features over other systems developed
in the literature.
1. End-to-end trainable
2. It uses convolutional features.
3. Requiring no ground truth bounding
boxes for characters during training
4. Not confined to a pre-defined
dictionary (Unconstrained)
5. Model size is very small
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 16
Results
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 17
Conclusion
We illustrated that our system has a significant advantages than other systems.
We illustrated the superiority of our system in recognizing unseen words from a
totally new lexicon. We also illustrated the uses cases of our system and we
showed that it can be used in real time applications because of it’s small size
feature. We showed the huge benefits of our system by it’s ability to train on
unsegmented data.
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 18
Perspective
We aim to develop a novel CNN
architecture for our system. This new
system will have the ability to extract
features from other regions of the
input images. Where we want to
represent four regions of each column
instead of representing the whole
column like the current system. This
enhancement is necessary when trying
to develop a system that handle better
Arabic words. Because of the
distinctive characteristics of Arabic
words. An illustration of the novel CNN
architecture is shown:
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 19
Thank your for your time
UNIVERSITY MUSTAPHA STAMBOULI MASCARA 2019-2020 20