UNIVERSITY OF INFORMATION TECHNOLOGY

FACULTY OF COMPUTER NETWORK AND COMMUNICATION

REPORT
Subject: Cryptography
Semester II (2021 – 2022)

PROTECT THE PRIVACY FOR SMART DEVICE

USING FACE AUTHENTICATION

Student: Võ Anh Kiệt

Student ID Number: 20520605
Student: Nguyễn Bảo Phương
Student ID Number: 20520704
Class: [Link]
University of Information Technology
Lecturer: Nguyễn Ngọc Tự

Hồ Chí Minh City, March 2022

 UNIVERSITY OF INFORMATION TECHNOLOGY
FACULTY OF COMPUTER NETWORK AND COMMUNICATION

REPORT
Subject: Cryptography
Semester II (2021 – 2022)

PROTECT THE PRIVACY FOR SMART DEVICE

USING FACE AUTHENTICATION

Student: Võ Anh Kiệt

Student ID Number: 20520605
Student: Nguyễn Bảo Phương
Student ID Number: 20520704
Class: [Link]
University of Information Technology
Lecturer: Nguyễn Ngọc Tự

Hồ Chí Minh City, March 2022

2
 Acknowledgement

To begin, we would like to thank our advisor, PhD Nguyễn Ngọc Tự, for his direction
and consistent monitoring, as well as for providing important project information and
for their help in finishing the research.

My gratitude and appreciation also extend to our colleagues and lecturers who assisted
us in the development of the project, as well as to those who have volunteered their
time and skills to assist us.

Võ Anh Kiệt – 20520605 – ANTN.2020

Nguyễn Bảo Phương – 20520704 – ANTN.2020

3
 Contents
Acknowledgement..........................................................................................................3

Part 1: Introduction.........................................................................................................6

1.1. Overview...........................................................................................................6

1.2. Problem Statement............................................................................................7

1.3. Scope.................................................................................................................8

1.4. Objective...........................................................................................................8

Part 2: Background.........................................................................................................9

2.1. Fundamental of face recognize.........................................................................9

2.2. OpenCV..........................................................................................................10

2.3. OpenCV – HaarCascade Classification..........................................................10

2.4. OpenCV – Single Shot Detector base ResNet network – Caffe Model..........11

Part 3: Design and Code...............................................................................................13

3.1. Design.............................................................................................................13

3.1.1. Register.....................................................................................................13

3.1.2. Recognize and verify................................................................................14

3.2. HaarCascade....................................................................................................15

3.2.1. Register.....................................................................................................15

3.2.2. Recognize and Verify...............................................................................16

3.3. Caffe................................................................................................................18

3.3.1. Register.....................................................................................................18

3.3.2. Recognize and Verify...............................................................................20

Part 4: Implement and Test...........................................................................................22

4.1. Implemet.........................................................................................................22

4.1.1. HaarCascade.............................................................................................22

4
 4.1.2. Caffe.........................................................................................................23

4.2. Test..................................................................................................................25

4.3. Review............................................................................................................25

Part 5: Conclusion and Future Work............................................................................26

5.1. Conclusion......................................................................................................26

5.2. Future work.....................................................................................................26

Reference......................................................................................................................27

5
 Part 1: Introduction

1.1. Overview

Biometrics, a method of automatically identifying a person based on

physiological or behavioral characteristics instead of traditional authentication
methods by granting them access to physical and virtual domains based on a
password, PIN code, smart card, plastic card, token, key, and so on., has
emerged as the most promising option for recognizing individuals in recent
years. There are several biometric systems such as signature, fingerprint, voice,
iris, retina, hand geometry, ear, and face geometry. Among these systems,
facial recognition seems to be one of the most popular, collectible, and
accessible. Facial recognition has several advantages over other biometric
methods, as demonstrated here: most other biometric-based techniques require
the user to voluntarily take an action, for example such as placing your palm up
for fingerprints or holding it in a fixed position in front of the camera to
identify iris or retina. On the contrary, face recognition can be done passively
without any specific action as face images can be obtained by remote camera.
Furthermore, there are many possible problems with other techniques during
data collection in general such as the uselessness of fingerprints for a
hypothetical person or the high cost of equipment during the data collection
process. using iris and retina etc. However, facial images can be easily obtained
with a few inexpensive still cameras. Good face recognition algorithms and
proper image pre-processing can compensate for noise and small changes in
orientation, scale, and lighting. Finally, while many techniques use the same
device to capture biological characteristics that can lead to the transmission of
germs and impurities by other users. In contrast, facial recognition is
completely non-intrusive and poses no danger to the user's health.

Face recognition has a long history dating back to the 1960s with manual
measurement of the coordinate position of various facial features such as eyes,
ears, nose and mouth, increasing accuracy by adding more markers are
subjective but always have to be calculated manually.

6
 Until 1991, the first version of automatic face recognition was released but was
not ready until 2006, the performance of the latest facial recognition algorithms
was evaluated using using the latest facial recognition algorithms available.
Facebook is the first social media company to begin rolling out facial
recognition that helps identify people whose faces may appear in photos
Facebook users update daily. airport, financial authentication, etc. Originally a
form of computer application, it has recently been used in a variety of
platforms, from mobile devices to in-vehicle devices. But he still hasn't fully
overcome the ongoing challenges with the quality of his delivery such as
lighting, backdrops, naming a few, and importantly, the intricacies of facial
recognition. relies on a 'too deep' architecture of complex neural networks
(CNNs) that are complex and unsuitable for real-time performance on
embedded devices.

1.2. Problem Statement

Technology has accelerated the speed of data access, acquisition and creation,
users' devices are now rich repositories of their life memories and content.
Many users use their devices for most browsing activities, the importance of
improving security and privacy on smartphones is becoming more and more
important.

Despite the development of technology, there are more and more cases of
information disclosure, theft, data information, invasion, data loss on the device
because some sneaky people try to access the phone and put the data on public
place on the internet.

7
 Sneaky people

Users Access the device

1.3. Scope

Face recognition

Users

Sneaky people

1.4. Objective

Understand the core concepts and algorithm using face recognition

Build a simple face authentication mechanism to verify the identity quickly and
protect the privacy of the users.

8
 Part 2: Background

2.1.1. Fundamental of face recognize

Facial recognition essentially works the same way as other forms of

"biometric" identification such as voice, iris, or fingerprint recognition. Given a
specific photo or other biometric data as input, the computer analyzes and
looks for a very specific set of markers within it. Comparing facets in this way
is conceptually like comparing fingerprints, although much more complex. If
the computer finds a significant similarity threshold between the sample and
the example samples, a match is declared.

The process of recognizing a face in an image has three phases:

 Face detection: Determines whether a human face appears in each image

and detects pixels in the image that represent a face.
 Preprocessing: A variety of preprocessing techniques are used to
minimize the effect of factors that can negatively affect the facial
recognition algorithm.
 Feature extraction: the features used in the recognition phase are
computed. These features vary depending on the automatic facial
recognition system used. For example, the first and simplest features
used in face recognition are the geometrical and distance relationships
between key points on the face and the recognition "algorithm"
corresponding to those intervals, this way.
 Face recognition: It consists of two separate steps:

9
 o Training: Create a database of faces. For each person, several images
are taken, and their characteristics are extracted and stored in a
database.
o Evaluation: When a face image is available, face detection and
feature extraction are used so that facial features can be obtained to
compare with each face class available in the database The more
appropriate the data and model, the more determined. If they are
close enough, a recognition event is fired. In general, facial
recognition is used for two main tasks:
Verification (1-1 match): When presented with an image of the
face of an unidentified individual along with identity
confirmation, verifying whether the individual is who he claims
to be.
Identity (1-many match): given an image of an unknown
individual, identify that person by comparing (possibly after
encoding) that image with a database (can be encrypted) images
of known individuals.

2.2. OpenCV

OpenCV is a library of programming functions primarily for real-time

computer vision. Free and cross-platform library under the open-source BSD
license. OpenCV supports TensorFlow, Torch/PyTorch, and Caffe deep
learning frameworks. OpenCV launches with a pre-trained engine that can be
used for face detection.

To detect face from camera stream, we use both the the suppliment from
both the HaarCascade and Caffe to build the register and verifier. Finally, make
the comparision.

2.3. OpenCV – HaarCascade Classification

The Haar Cascades classifier, which uses Haar's feature-based cascading

classifier, is an efficient object detection method proposed by Paul Viola and

10
 Michael Jones in their paper, "Object Detection rapidly using simple feature
enhancement cascaders" in 2001. It is a machine learning-based approach in
which a cascade function is formed from a large number of positive and
negative images. It is then used to detect objects in other images. HaarFeatures
has very good edge and line detection. This makes it especially good at
detecting faces.

Different stages in visualization. Source: [Link]

However, since Haar Features must be defined manually, there is some limit to
the types of things they can detect. If you feed the classifier (a network or any
algorithm that detects faces) edge and line features, it will only be able to
detect objects with obvious edges and lines. Even as a face detector, if we
manipulate the face a little (e.g. cover our eyes with sunglasses or tilt our head
to one side), the Haar-based classifier may not recognize it. face.

2.4. OpenCV – Single Shot Detector base ResNet network – Caffe Model

OpenCV's deep learning face detector is based on the Single Shot Detector
(SSD) framework with the ResNet base framework.

The SSD approach is based on a forward agglomeration network that generates

a fixed-size set of bounding boxes and scores for the presence of feature class
instances in those boxes, then zero-decimal deletion step to generate the final
detection. The first network layers are based on a standard architecture used for
high-quality image classification (truncated before any classifiers), which we
will call the underlying ResNet network.
11
ResNet, the so-called Residual Neural Network by Kaiming He et al.,
introduced a new architecture with "jump connections" and provided heavy
batch normalization at ILSVRC 2015. These hopping connections also known
as 'closed unit' or closed repeat unit and bears strong resemblance to recent
success factors applied in RNN. Thanks to this technique, they were able to
generate a NN with 152 layers while having less complexity than VGGNet. It
achieves a top5 error rate of 3.57%, which exceeds human performance on this
dataset.

12
 Part 3: Design and Code

3.1. Design

3.1.1. Register

13
3.1.2. Recognize and verify

14
3.2. HaarCascade

3.2.1. Register

Get the data of the users from Images

Process the image

Save the data to pickle file

15
3.2.2. Recognize and Verify

Load the model and pickle file

Load the video stream, start recognizing and verifying

16
17
3.3. Caffe

3.3.1. Register

Load the model, database and declare variable

Process the image

18
Save the data to pickle file

19
3.3.2. Recognize and Verify

Load the model and data

20
Start recognizing and verify

21
 Part 4: Implement and Test

4.1. Implemet

4.1.1. HaarCascade

Before registering

Register

22
Recognize and verify

4.1.2. Caffe

Before registering

23
Register

Recognize and verify

24
4.2. Test

HaarCascade Caffe Paper1

Recognize in Pass Pass Pass
normal light
Recognize a part of Pass Pass Not mentioned
face
Detect fake face Not Pass Not Pass Pass
Recognize in low Not Pass Not Pass Not mentioned
light
Respone time 2.0052 second 2.2138 second 0.3496 second
counter
Accuracy 46.8547% 71.8954% 99.95%

4.3. Review

This project provides the simple functions and features to register and verify
the people with ficial recognition. In this project, people can regist with the pictures
and verify with the video stream. Despite the verified testcase, it still faces a security
problem and user experience.

25
 Part 5: Conclusion and Future Work

5.1. Conclusion

In this project, we demonstrated another authentication method using facial

recognition and authentication with HaarCascade and Caffe (powered by
ResNet). The program works well with a basic function for users to register and
verify.

Due to the latest technology, we faced many problems due to our lack of
knowledge about face recognition as well as multithreaded programming in
Python. Besides, facing difficulties in developing and deploying Azure
Function is also a challenge for us.

Although our project is not excellent enough for real-world application because
of security issues with facial recognition, it is the foundation for development
in the next steps. So, we will continue to develop it after this project.

During the implementation of this project, we had the opportunity to review

and update the knowledge that we had acquired during our 2 years of study at
the university. Furthermore, we have also gained valuable insights into new
technology trends.

5.2. Future work

Various adaptations, test, and experiments were left for the future due to lack
of time. Future work issues extend to more advanced features

 Develop the mobile app and web app allowing user to have more
methods in registering and verifying
 Implement the program in the IOT device such as Raspberry Pi
 Build the training data system using the larger data to improve the
register and verify system
 Improve the register and verify system algorithm to pop up the accuracy
of the system

26
 Reference

1. Jong-Hyuk Im , Seong-Yun Jeon, & Mun-Kyu Lee (2020). Practical Privacy-

Preserving Face Authentication for Smartphones Secure Against Malicious
Clients. IEEE TRANSACTIONS ON INFORMATION FORENSICS AND
SECURITY, VOL. 15, 2020 (2386-2401)
2. Adrian Rosebrock, Face detection with OpenCV and deep learning, 2018,
[Link]
deep-learning/
3. Chi-Feng Wang, What’s the Difference Between Haar-Feature Classifiers and
Convolutional Neural Networks? 2018, [Link]
the-difference-between-haar-feature-classifiers-and-convolutional-neural-
networks-ce6828343aeb
4. OpenCV library documentation, [Link]
5. Ramiz Raja, Face Detection Using Opencv And Python: A Beginner’s Guide,
2017 [Link]
6. Renan Dias, The 5 Factors of Authentication, 2017,
[Link]
bcb79d354c13

27