UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

Visual face scanning and emotion

perception analysis between Autistic
and Typically Developing children

Uzma Haque Syeda Syed Mahir Tazwar Abstract

University of Dhaka University of Dhaka Autism is a neuro-developmental condition that is
Dhaka, Bangladesh Dhaka, Bangladesh characterized by a number of unconventional behaviors
[Link]@[Link] [Link]@[Link] such as restricted and repetitive activities. It is often
largely attributed to deficiency in communication and
Ziaul Zafar Miftahul Jannat Rasna social interaction. Therefore, it is difficult to make
University of Dhaka University of Dhaka autistic individuals, especially children, to comply with
Dhaka, Bangladesh Dhaka, Bangladesh researches that aim at comprehending this condition.
zafarziaul@[Link] rasna@[Link] However, with the availability of non-invasive eye-
tracking technology, this problem has become easier to
Zishan Zahidul Islam Koichi Kise deal with. The following research probes into the visual
University of Dhaka Osaka Prefecture University face scanning patterns and emotion recognition
Dhaka, Bangladesh Osaka, Japan between 21 autistic and 21 control or TD (typically
[Link]@[Link] kise@[Link] developing) children when displayed pictures of 6 basic
emotions (happy, sad, angry, disgusted, fearful and
Md. Atiqur Rahman Ahad
surprised). Tobii EyeX Controller was used to attain the
University of Dhaka
gaze data and the data was processed and analyzed in
Dhaka 1000, Bangladesh
MATLAB. The results revealed that children with autism
atiqahad@[Link]
look less at the core features of the face (eyes, nose
Permission to make digital or hard copies of all or part of this work for and mouth) while scanning faces and have more
personal or classroom use is granted without fee provided that copies difficulty in perceiving the correct emotion compared to
are not made or distributed for profit or commercial advantage and
that copies bear this notice and the full citation on the first page. the typically developing children. This atypical face
Copyrights for components of this work owned by others than ACM scanning and lack of preference to the core features of
must be honored. Abstracting with credit is permitted. To copy
otherwise, or republish, to post on servers or to redistribute to lists, the face can be the reason why autistic individuals have
requires prior specific permission and/or a fee. Request permissions trouble understanding others’ emotions and an overall
from Permissions@[Link].
UbiComp/ISWC'17 Adjunct, September 11–15, 2017, Maui, HI, USA
incompetency in communication and social interaction.
© 2017 Association for Computing Machinery. To delve more into this, further eye-tracking,
ACM ISBN 978-1-4503-5190-4/17/09...$15.00
[Link]

844
UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

neuroimaging and behavioral studies should be done in expressions by looking at the core features of the face,
integration. which includes the eyes, nose and the mouth [2]. The
understanding of emotions by correctly scanning the
Author Keywords major features of the face plays a vital role in the
Autism; eye-tracking; face scanning; emotion efficiency of one’s social and communication abilities.
perception; typically developing; Tobii EyeX Controller. The face offers notable stimuli whose social significance
is exhibited at a very young age in humans. Infants are
ACM Classification Keywords able to orient their gaze at faces from birth [3]. This
I.4 Image Processing and Computer Vision; visual behavior allows the early communicative ability
which is essential in the establishment and
H.5.m. Information interfaces and presentation (e.g., development of social and emotional relationships. The
HCI): User-centered design; Miscellaneous; understanding of facial expressions mainly stems from
looking at the eye region of the face and can influence
I.2.9 Robotics: Sensors the ability to understand the mental state of others [4].
Due to their informative nature, the eyes and the
I.4.8Scene Analysis mouth are the favorable visual targets in the
observation or inspection of the face and typically
Introduction developing individuals spend more time looking at
Autism is a developmental disorder that includes a these features [5]. Neurobiological studies show that
range of abnormal behaviors. The main trait of this face perception is atypical in individuals with autism
disorder consists of social and communication and this is thought as one of the likely reasons for their
incompetency, repeated activity, sensory problems and social incompetency [6]. In many behavioral studies,
delay in learning. Autism is often attributed to children showed flawed perception of faces and a lack
abnormal visual behavior such as avoiding eye contact of interest to look especially at the eye area [7], [8].
[1]. With the prevalence of eye-tracking devices in the However, very few studies have been conducted in this
market, visual behavior studies of autistic individuals regard using eye-trackers.
have become a research field with great potential.
The purpose of this research is to carry out visual face
The ability of perceiving facial expressions and scanning pattern and emotion perception analysis
deducing other important social cues from looking at between typically developing children and children with
the faces of others is a vital prerequisite for normal autism. This is done by studying visual data provided
social interactions and communication. To perceive and by these two groups when they are shown the pictures
to judge facial expressions means to understand the of 6 basic emotions (happy, sad, angry, surprised,
emotional state of another by looking at the major disgusted and fearful). The atypical visual behaviour of
features of their faces. Typically developing children children with autism is likely to spring from their
usually perceive emotions and process facial inability to effectively process information that they

845
 SESSION: MENTAL HEALTH AND WELL-BEING

see. An understanding of the difficulties that these concluded that they have diminished fixation towards
children experience by those around them can make a the eye region [19], [20].
big difference in their ability to integrate effectively in
the society. The novel contribution of this paper is that, Another research reported discrepancies in face
in perspective of Bangladesh, this is a complete new scanning patterns between ASD and TD children [14].
contribution. In global perspective, the methodology The study compared the visual scanning patterns in a
and approach is different from previous related works. group of 3- to 6 year-old children with ASD (n=11) and
A moderate amount of samples has been used. Most a 5-year-old control group during exploration of faces
importantly, the device used for the research is very showing positive and negative emotions. Areas of
inexpensive, user-friendly and readily available in the interest (AOI) included the eyes, nose and mouth. The
market. ASD group looked less at the AOIs compared to non-
AOI areas on the screen, and spent less time scanning
Related Works the mouth [14]. It is a notion that directing gaze to the
Face processing and recognition shortcomings are usual eyes of other people is crucial for extracting emotional
among individuals with autism [9]. Aversion to making information [21]. Furthermore, brain-imaging
eye contact could lead to poor face and emotion techniques such as fMRI (functional magnetic
processing [10]. A study found reduced fixation on the resonance imaging) have offered insight into the face
eye area of the face in people with Autism Spectrum perception and processing deficit in ASD.
Disorder (ASD) as compared to age-matched typical
controls in a face recognition task [11]. The two groups Method
showed a similar trend for fixation on the mouth region A. Participants
and the face region in general [11]. Reduced fixation to
the eye region was also found in other researches on A total of 42 children participated in the research
face recognition in autism [12], [13]. There are also experiment. 21 children with autism were taken (14
findings that contradicted these results [14], [15], male and 7 female). The participants were aged
[16]. A study indicates that deficits in face processing between 5 to 17 years. They were recruited from an
might be present in individuals with autism as early as Autism habilitation school. All of them were previously
the first year of life [17]. diagnosed with autism by doctors. The participants
were all able to speak and understand instructions.
There are discrepancies in findings with regards to
longer fixation on the mouth compared to eye for For the control group, a total of 21 typically developing
children with autism [11], [18]. One possible (TD) children were selected from the local community.
explanation for this might be the difference in stimuli as The participants were gender and age matched with the
one study used dynamic scenes as stimuli [18] and the autistic group i.e., 14 male and 7 female between the
other used static face photographs [11]. There are ages 5 to 17. These children had no previous records
studies using adults and adolescents with ASD, which of neurological or mental illness.

846
UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

The research was conducted with utmost regards to all concentrated mostly on the display screen. The Tobii
ethical issues since it involved dealing with clinical EyeX Controller was mounted on the laptop screen and
subjects who have been diagnosed with autism was kept an approximate of 60 cm away from the chair
previously. Ethical clearance was received from the where the participants sat. Guardians/ teachers were
Director and the Head of the Pediatrics Department of allowed to accompany the children throughout the
Dhaka Medical College Hospital. Individual parental experiment. The researcher explained the procedure of
permission was also received for each participant. It the study to the guardian and the participant. Once it
was made sure that all the guardians and the was clear that they completely understood the
participants understood the aim and procedure of the procedure and were willing to continue, the experiment
research. Participants and guardians were also assured began. At the beginning, a 9-point calibration took
that they can withdraw from the study anytime if they place where if 6 points were correctly looked at, it was
wished to, without any penalty. considered as a successful calibration. After each
successful calibration, the images started to appear,
B. Stimuli each for 7 seconds. The participants were instructed to
look at the image and answer the question of the
The participants were required to look at 6 images experimenter (or their guardian if necessary). For each
displaying different basic emotions (happy, sad, angry, image, the experimenter or the guardian gave the
disgusted, fearful and surprised) and correctly identify participant two options verbally and they had to pick
the emotion. For their convenience, they were verbally the one that correctly matched the emotion on the
given two options for each image. This was done in screen.
order not to unsettle the children (especially, the ones
with autism), as more than two options were highly Data Analysis
likely to confuse them or make it hard for them to The Tobii EyeX Controller was used to record the gaze
follow and respond within only 7 seconds. The images data of the participants. This eye-tracking device can
were collected from a database that is comprehensive operate at a maximum range of 90cm and has a data
and established [22], [23]. Each image had a resolution rate of ~60 Hertz. However, it does not support
of 640x490 pixels and had duration of 7seconds. The internal memory and cannot store raw data. Hence, an
images were in grayscale so as to make them less open source MATLAB toolkit was used to interface and
distracting and more accurate for the study. store the data acquired from the Tobii EyeX Controller
[24]. A code was written in MATLAB to display the
images and to simultaneously acquire and save the
C. Experimental Setup and Procedure gaze data received. After the data was acquired, it was
processed and analyzed using MATLAB.
The research experiment was carried out in a room with
lights that were not too bright. This was done to make Region of interest (ROI) and scanning pattern analysis
sure that the children do not get distracted and was used to process the data received. In ROI method,

847
 SESSION: MENTAL HEALTH AND WELL-BEING

a region of interest is determined beforehand and the Scanning pattern analysis, on the other hand, means
numbers of data that fall in that particular defined area analyzing the data to find the areas of the stimuli on
are calculated. The data analysis of face scanning which the participant fixated and studying his/ her
mainly involved the parameter of duration of fixation at visual pathways. For the scanning pattern analysis, a
different regions of the face. The core features of the code was written in MATLAB to find the gaze points
face as mentioned are the eyes, nose and the mouth. numbered with respect to time. For instance, the first
These 3 areas of the face were considered as the ROI gaze point becomes number 1; the second gaze point
for the data analysis. Firstly, the total duration that a becomes number 2 and so on. This information can be
particular participant looked at the screen was a valuable measure of the saccade made by the
calculated for each image. This divided by the total participants and is expected to be more irregular and
duration of the image gave the percentage of time the higher in participants with autism. The images showing
participant was engaged in looking at the image. These the results are not in grayscale in order to understand
calculations were made for all 6 images for each the right and left eye gazes clearly.
participant.

The next step was to separately calculate the duration

a participant looked at the eyes, nose, and the mouth
respectively. These duration data were then converted
to fixation percentage with respect to the total fixation
duration of that particular participant, i.e., fixation
percentage for the eyes, mouth and nose separately. In
addition to these, the data for looking at the non-
feature areas was calculated and designated as ‘rest’.

For a particular image, the number of autistic

participants that correctly recognized the emotion was
referred to as ‘successful identifiers’. Then the fixation
percentages for all the core features and ‘rest’ for these
participants were averaged to observe where exactly
they looked to correctly identify the emotion. For the
‘unsuccessful identifiers’ too, the same process was Figure 1: The red and blue patterns indicate the right and left
eye gaze respectively, and the numbers represent the order of
carried out to look for differences in the gaze pattern
fixation.
from the ones who correctly recognized the emotion.
For the TD group, a total average of the fixation for Result
each area was found. The two datasets for autistic and TD group were found
to have significant dissimilarities. The duration values

848
UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

Average fixation %
for TD children

Rest
13%
Nose Eyes
18% 45%
Mou
th
24%

Figure 3: Average fixation

percentage for the Typically
Developing group

Average fixation %
for Autistic children

Eyes
14%
Rest Mou Figure 2: Average fixation percentage for the Autistic group (Successful and unsuccessful identifiers are indicated by the first and
41% th
second bar respectively for each emotion. The gaze percentage for eyes, mouth, nose and rest are given for each case).

24% differed largely from each other. The autistic group showed that the key features (eyes, nose and mouth)
Nose showed less concentration level than the TD group and got much less importance from autistic children
21% had lower affinity towards the key features. compared to the control children and this is in
agreement with previous literature. Figure 2 shows the
The average looking duration of autistic children was
average fixation percentages for the TD children. It can
68.29% of the total test time. This was substantially
be seen that they tend to focus more on the core
Figure 4: Average fixation lower than the 87.33% that was determined for TD
features of the face in order to perceive emotions. On
percentage for the Autistic group children. The comparison of the two datasets also
the other hand, the average fixation percentages for

849
 SESSION: MENTAL HEALTH AND WELL-BEING

the autistic group in Figure 3 indicate that they do not Performance was satisfactory for both ‘sad’ and
show any preference towards the key features of the ‘surprised’ emotions where almost half of them could
face. Their value for the fixation towards the eyes is recognize the emotion correctly. Here too, it is evident
particularly low, which is very likely to indicate that that they looked mostly at the mouth region. However,
they have an aversion towards looking at the eyes. It is for the other three emotions, the children performed
seen that they tend to spend more time looking at the very poorly and only a few could name the emotions
non-feature parts of the face (hair, ears, neck, etc.) or accurately. Overall, this is a significantly poorer
at the background of the images. performance in comparison to the TD group. All the TD
participants identified the emotions correctly except for
In Fig. 4, the average fixation percentages for the
only 2 and 4 participants who failed to identify the
autistic group are shown. For each emotion, two bars
‘surprised’ and ‘disgusted’ emotions respectively.
are shown; the first one represents the individual
fixation percentages for the ones who correctly
Figure 5 shows the scan pattern analysis samples for
identified the emotion and the second bar represents
both the autistic and the TD group. Here it can be again
the fixation percentages for those who failed to identify
seen that the children with autism tend to look less at
the emotion. The fixation or gaze percentages for the
the core features of the face and concentrate more on
eye region, mouth, nose and rest are shown in each
the other aspects of the picture. In particular, they are
bar. It can be seen at a glance that the values for the
very keen to avoid looking at the eye region of the
fixation towards the non-feature areas of the face is
image in most cases. A lot of gaze patterns are seen at
significantly high in almost all the bars.
the background of the images which are pretty plain.
However, for the control or the TD group, it can be
On the other hand, the value for looking at the eye
seen that they mostly concentrate on the key features
region is very low in all the bars, indicating that
of the face, i.e., towards the eyes, nose and mouth and
children with autism is most likely to have an aversion
their gaze pattern is quite clustered in these regions. It
towards looking at the eyes. The values for fixation at
can also be seen that the number of fixation points for
the mouth are higher for those who successfully
the TD children is much more than that for the autistic
identified the correct emotions than those who could
children. This can be because of the fact that autistic
not. For the nose area, the values are very high for the
children’s overall screen looking time is much less than
‘disgusted’ emotion image, but in other cases it is
that of the TD children. The gaze pattern within the
pretty low and shows no pattern. The number of
autistic group was rather random and had no specific
autistic children who could correctly perceive the
pattern or preference towards any special point. But
emotions is also not very high. For the ‘happy’ emotion,
the gaze pattern for TD children did not exhibit much
they performed pretty well and except four, all of them
variation and focused mainly on the core features.
could understand that happiness was being expressed
in the image. Here, it is seen that they looked mostly
In summary, whether the children with autism
towards the mouth to understand the ‘happy’ emotion.
perceived the emotions correctly or not, no particular

850
UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

preference towards the core features of the face was compared to their peers. Their total screen time was
evident from their gaze behavior. This was in contrast much less than the controls and they preferred to look
with the TD participants, who showed clear preference at the non-feature areas of the face (ears, neck, hair,
towards looking at the core features of the face in order background, etc.) rather than the core features (eyes,
to perceive its emotion. nose and mouth).

Typically developing group Autistic group

Figure 5: Scan pattern analysis samples for the TD and the Autistic group. Here the red and blue patterns indicate the right and the left
eye gaze respectively. The numbers represent the gaze order, for example, number 1 is the first fixation point, number 2 the second,
etc. (Note that, the images were obtained from the MATLAB display and their contrasts were adjusted for better understandability.)

Conclusion In many neuro-imaging, behavioral and few eye-

The face scanning and emotion perception analyses tracking studies, the results found were similar. This
between children with autism and typically developing type of aversion towards faces and the lack of
children exhibited results that are open to understanding of other people’s emotions can be a
interpretations and are parallel with previous findings. result of the abnormal face processing mechanism of
Participants with autism were found to have an the individuals with autism. One of the main markers of
aversion towards looking at the faces in general autism is the incompetency in social interaction which is

851
 SESSION: MENTAL HEALTH AND WELL-BEING

often linked to the inability to properly process facial mechanisms", Journal of Experimental Child
emotions and perceive them [25]. This means that they Psychology, 90(3), 255-273, 2005.
scan the essential parts of the face (eyes, mouth and 4. S. Baron-Cohen, Mindblindness, 1st ed.
nose) less than that which is typical to normal human Cambridge, Mass.: MIT Press, 1997.
beings. The results found in our research also showed a 5. G. Walker-Smith, A. Gale, and J. Findlay, “Eye
general aversion to the core features of the face which movement strategies involved in face
is keys to understanding the emotion of a person. It perception”, Perception, 6(3), 313-326, 1977.
can also be the reason why autism is greatly associated 6. K. Pierce, R. Muller, J. Ambrose, G. Allen, E.
with deficiency in social interaction and communication Courchesne, “Face processing occurs outside
[6]. Further studies by collaborating eye-tracking and the fusiform ‘face area’ in autism: Evidence
neuroimaging should be carried out to delve deeper from functional MRI”, Brain, 124, 2059-2073,
into this. 2001.
7. T. Langdell, "Recognition of Faces: an approach
Acknowledgements to the study of autism", Journal of Child
The authors would like to pass their honest gratitude to Psychology and Psychiatry, vol. 19, no. 3, pp.
255-268, 1978.
the Director and the Head of Pediatrics Department of
Dhaka Medical College Hospital for providing ethical 8. R. Joseph and J. Tanaka, "Holistic and part-
clearance to carry out the research. Special thanks to based face recognition in children with autism",
the Autism Spectrum School that allowed us to Journal of Child Psychology and Psychiatry, vol.
44, no. 4, pp. 529-542, 2003.
approach their students and also to each and every
parents/guardians of the participants for letting their 9. S. Weigelt, K. Koldewyn and N. Kanwisher,
children take part in the research. "Face identity recognition in autism spectrum
disorders: A review of behavioral
studies", Neuroscience & Biobehavioral
References Reviews, vol. 36, no. 3, pp. 1060-1084, 2012.
1. J. Woods and A. Wetherby, "Early Identification
of and Intervention for Infants and Toddlers 10. R. Schultz, "Developmental deficits in social
Who Are at Risk for Autism Spectrum perception in autism: the role of the amygdala
Disorder", Language Speech and Hearing and fusiform face area", International Journal
Services in Schools, vol. 34, no. 3, p. 180, of Developmental Neuroscience, vol. 23, no. 2-
2003. 3, pp. 125-141, 2005.

2. Diamond, R., & Carey, S. "Why faces are and 11. K. Dalton, B. Nacewicz, T. Johnstone, H.
are not special: an effect of expertise". Journal Schaefer, M. Gernsbacher, H. Goldsmith, A.
of Experimental Psychology, 115, 107-117, Alexander and R. Davidson, "Gaze fixation and
1986. the neural circuitry of face processing in
autism", Nature Neuroscience, 2005.
3. Turati, C., Valenza, E., Leo, I., & Simion, F.
"Three-month-olds’ visual preference for faces 12. K.A. Pelphrey, N.J. Sasson, J. Reznick, G. Paul,
and its underlying visual processing B.D. Goldman, and J. Piven, "Visual Scanning

852
UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

of Faces in Autism", Journal of Autism and 19. B. Corden, R. Chilvers, and D. Skuse,
Developmental Disorders, Vol. 32, No. 4, "Avoidance of emotionally arousing stimuli
August 2002. predicts social–perceptual impairment in
13. M. Spezio, R. Adolphs, R. Hurley, and J. Piven, Asperger's syndrome", Neuropsychologia, vol.
"Analysis of face gaze in autism using 46, no. 1, pp. 137-147, 2008.
“Bubbles”", Neuropsychologia, vol. 45, no. 1, 20. L. Speer, A. Cook, W. McMahon, and E. Clark,
pp. 144-151, 2007. "Face processing in children with
14. T. de Wit, T. Falck-Ytter, and C. von Hofsten, autism", Autism, vol. 11, no. 3, pp. 265-277,
"Young children with Autism Spectrum Disorder 2007.
look differently at positive versus negative 21. R. Adolphs, F. Gosselin, T. Buchanan, D.
emotional faces", Research in Autism Spectrum Tranel, P. Schyns, and A. Damasio, "A
Disorders, vol. 2, no. 4, pp. 651-659, 2008. mechanism for impaired fear recognition after
15. T. Nakano, K. Tanaka, Y. Endo, Y. Yamane, T. amygdala damage", Nature, vol. 433, no.
Yamamoto, Y. Nakano, H. Ohta, N. Kato, and 7021, pp. 68-72, 2005.
S. Kitazawa, "Atypical gaze patterns in children 22. T. Kanade, J. F. Cohn, and Y. Tian,
and adults with autism spectrum disorders "Comprehensive database for facial expression
dissociated from developmental changes in analysis", Proc. of the Fourth IEEE
gaze behaviour", Proceedings of the Royal International Conference on Automatic Face
Society B: Biological Sciences, vol. 277, no. and Gesture Recognition, 2000.
1696, pp. 2935-2943, 2010.
23. P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z.
16. K. Chawarska and F. Shic, "Looking but not Ambadar, and I. Matthews, "The Extended
seeing: Atypical visual scanning and Cohn-Kanade Dataset (CK+): A complete
recognition of faces in 2 and 4-year-old expression dataset for action unit and emotion-
children with autism spectrum disorder". specified expression", Proc. of the Third
Journal of Autism and Developmental International Workshop on CVPR for Human
Disorders, 39(12), 1663-1672, 2009. Communicative Behavior Analysis (CVPR4HB
17. A. Klin and W. Jones, "Altered face scanning 2010), 94-101, 2010.
and impaired recognition of biological motion in 24. Online "MATLAB Toolbox
a 15-month-old infant with EyeX", [Link], 2017.
autism", Developmental Science, vol. 11, no. [Link]
1, pp. 40-46, 2008. wiki/Home/ Accessed on: 29 Mar. 2017.
18. W. Jones, K. Carr, and A. Klin, "Absence of 25. U. Rosenhall, E. Johansson, and C. Gillberg,
Preferential Looking to the Eyes of Approaching "Oculomotor findings in autistic children", The
Adults Predicts Level of Social Disability in 2- Journal of Laryngology & Otology, vol. 102, no.
Year-Old Toddlers With Autism Spectrum 05, pp. 435-439, 1988.
Disorder", Archives of General Psychiatry, vol.
65, no. 8, p. 946, 2008.

853