RECENT ADVANCES in ARTIFICIAL INTELLIGENCE, KNOWLEDGE ENGINEERING and DATA BASES

Intelligent Emotion-Oriented eCommerce Systems

SIMONE LEON, ALEXANDER NIKOV
Department of Mathematics and Computer Science, The University of the West Indies
St. Augustine, TRINIDAD AND TOBAGO (W.I.)
simoneleon2008@[Link], [Link]@[Link], [Link]

Abstract: - Emotion-oriented eCommerce is a new research field. Intelligent emotion recognition

techniques based on modern mathematical models using computational intelligence are presented.
Progression within this field of study is highlighted. The requirements for designing of intelligent
emotion-oriented eCommerce systems are defined. A model for simulation of intelligent emotion-oriented
eCommerce systems is proposed. It is an important tool supporting the experimental study and design of
emotion-oriented eCommerce systems.
Key-Words: - eCommerce, Emotion recognition, Computational intelligence, Modelling, Affect

1 Introduction simulation of emotion-oriented eCommerce

Electronic commerce is increasingly popular in systems will be very useful for their study and
today’s businesses [5] and is becoming an integral design.
facet in the online shopping experience [1].
Emotions are influencing customer cognitive
processes and eShopping behaviour [28]. ‘Affect’ 2 Intelligent Emotion Recognition
is an umbrella term to describe human moods and Emotion recognition involves assigning
emotions [18]. It is an important variable in computers with the ability to observe, interpret
cognition and decision behaviour. Future research and generate affect features. Its intent is to
should investigate affective eCommerce as online improve the quality of communication between
decision behaviour. Affect, emotion, and the customer and eCommerce system. The
subjective experience should be factored into the capturing and processing of emotions by the
design of eCommerce systems [23]. computer is a new study. The standard procedure
Computational Intelligence (CI) techniques are of affective interaction consists of affect
more suitable than standard techniques for information capture and modelling, affect
emotion recognition [9]. CI can be used also in understanding and expression.
simulation of emotion-oriented eCommerce Researchers are utilizing various methods of
systems. capturing emotions of the user. Such data should
Emotions play a significant role in an be processed to generate an adequate emotional
electronic commerce environment. Emotion- response that would satisfy the customer’s needs.
oriented eCommerce is a new research field. Bianchi-Berthouze [16] developed a technology
During recent years, the use of intelligent systems that improves the sense of engagement or
in eCommerce has increased significantly, immersion of its customers (positive usability) by
providing a new perspective on the overall taking into account their affective states. Their
shopping experience. The aim of the paper is to body postures were used as an indicator of human
propose and discuss the application of affective states. Picard [22] utilized a number of
computational intelligence models involved in techniques for capturing emotive behaviour,
emotion recognition, simulation and design of namely eye-tracker, face reader, affective learning
emotion-oriented eCommerce systems. companion, conductive chat, emote mail, interface
As a first step towards creation of intelligent tailor, mouse behaviour analysis and relational
emotion-oriented eCommerce systems should be agents.
defined its design requirements. Then a model for

ISSN: 1790-5109 202 ISBN: 978-960-474-154-0

 RECENT ADVANCES in ARTIFICIAL INTELLIGENCE, KNOWLEDGE ENGINEERING and DATA BASES

INPUT DATA PROCESSING EMOTION RECOGNITION OUTPUT

Emotion Signals Data Collection Techniques Intelligent Data Processing Intelligent Emotion Recognition Emotion Expression

Speech Speech Recognition Natural Language Processor

Facial Expression Face Capture Face Reader

Motor Behavioural Patterns Log Files Haptic Cues processor

Emotion Recognition
Using Computational
Intelligence Models

Body Gestures and  Hand Tracking Expressive Gesture

Movements Processing

Bio‐physiological Sensor System Physiological Signal 

Signals Processor

 
Fig. 1: Intelligent Emotion Recognition.

People express emotion through facial 2.3 Motor Behavioural Patterns

expression, body movements and gestures, voice The form of communication can be easily
behaviour and other bio patterns. Fig. 1 conveyed by using the standard mouse and
summarizes the key technologies in emotion keyboard provided by any computer system. The
recognition starting with input signals, data mouse and keyboard can identify motor-
collection and processing, emotion recognition behavioural parameters from log-files. The
based on computational intelligence modelling analysis of mouse clicks, mouse speed, or
and emotion expression. keystroke speed provides an uncomplicated
method of measuring emotions without any
additional devices.
2.1 Speech Processing
A powerful method for communicating and
expressing emotion is through speech. Research 2.4 Body Gestures and Movements
on emotional speech focuses on typical acoustic Body gesture and movement refer to the different
features [31]. Such elements like pitch (level, positions of the body and its changes over a
range and variability) and speaking rate have been period of time [31]. At present the study of body
analysed by researchers in addition to tempo and gestures and movements is focused on hand
loudness [27]. People express their feelings by the tracking. It conveys many meanings, for example
acoustic features and by the content of what they a clenched first may indicate an aroused emotion
want to say (words, phrases and syntactic of fear, anger, or excitement. There are two
structures) [31]. The use of speech recognition methods of analysing the movement in the hand:
techniques is useful as the user’s tone and the apparentness or the 2-D method [7] which
intonation depicts the emotional state of the user. analyses the apparent features of hand movement
using 2-D images and the 3-D method which does
2.2 Facial Expression the same as the former in a 3-D environment [32].
Facial expressions are one of the principal
methods that are used to detect human emotions
[30]. Ekman [8] outlined some typical features 2.5 Biosignals
associated with the six basic emotions namely joy, Detecting and recognising biological signals is an
anger, surprise, disgust, fear and sadness. The essential aspect of emotion recognition [22].
Facial Action Coding System (FACS) details a set Picard [24] developed a sensor system measuring
of muscular movements for each of the basic biosignals such as heart rate, skin conductivity,
emotions [7]. respiration, temperature, pulse, electrical activity

ISSN: 1790-5109 203 ISBN: 978-960-474-154-0

 RECENT ADVANCES in ARTIFICIAL INTELLIGENCE, KNOWLEDGE ENGINEERING and DATA BASES

in the muscles, etc. that allows the detection of respiration, pulse, skin conductance level and
various emotional states in a user [22]. corrugator activity were measured. Experiments
proved that emotive behaviour can be successfully
analysed through the use of the mouse and
2.6 Intelligent Data Processing keyboard [20].
Most of the research does not combine Intelligent emotion-oriented eCommerce
different modalities into a single system for the system can be seen as a system that can effectively
analysis of human emotional behaviour: different identify, interpret and respond to a customer’s
channels of information (mainly facial emotions in eCommerce environment.
expressions and speech) are considered The idea of developing emotion-oriented
independent to each other [3]. As data is collected eCommerce systems may be difficult to conceive,
it must be processed and analysed and the however, this type of eCommerce systems provide
appropriate intelligent data processing techniques many advantages. Despite the many capabilities
like computational intelligence must be applied. the computer system may possess, e.g. storage of
large volumes of data, processing speed and
performing complex calculations, computers are
2.7 Emotion Recognition based on unable to determine the way the user may feel or
Computational Intelligence Models think. An eCommerce system which is able to
Computational intelligence (CI) has been defined detect the emotional state of the user should be
as “the study of adaptive mechanisms to enable or able to take the necessary steps to produce a
facilitate intelligent behaviour in complex and positive emotional environment for the user.
changing environments” [9]. Appropriate for
emotion recognition are CI methods like Fuzzy
Systems, Genetic Algorithms, Neural Networks, 3.1 Requirements for Design of Intelligent
and especially Swarm Intelligence which includes Emotion-oriented eCommerce System
Particle Swarm Optimization, Ant Colony An intelligent emotion-oriented eCommerce
Optimization, Bee Colony Optimization and Wasp system should be designed to effectively
Colony Optimization. accommodate techniques that would identify
affective behaviour in the customer. The
3 Intelligent emotion-oriented identification of the emotional state of a potential
customer should be followed by the appropriate
eCommerce Systems response from the system which will encourage
Studies on intelligent emotion-oriented the customer to make a favourable purchase
eCommerce systems are very limited up to this decision.
point [14]. Any affective eCommerce system The design of such a system should allow
would need to assess the affective state of the mixed interaction where control is shared between
customer through affective sensing and the customer and the system. There are two
recognition in order to determine an appropriate scenarios that could be investigated in relation to
reaction [12]. Affective responses to web intelligent emotion-oriented eCommerce systems:
interfaces play a significant role in the acceptance (1) Customer - to - the intelligent emotion-
of banking websites [26]. In [22] is investigated oriented eCommerce systems – The customer
the possibility of designing emotive interfaces communicates with the system via the chosen
which involves customers emotionally and input medium.
enhance the quality of decisions made during the (2) Intelligent emotion-oriented eCommerce
interaction process. Experiments on recording the systems - to - Customer - The system would
emotive reactions of the participants to interface request input from the customer and in return
design of cyber banking sites are carried out. The provide the necessary feedback.
influence of induced emotions on motor- The intelligent emotion-oriented eCommerce
behaviour parameters while shopping on an system is best suited for a business-to-customer
eCommerce website for office supplies was eCommerce environment as the aim is to gain the
investigated. Bio physiological parameters like customer’s trust and loyalty. The ultimate goal is

ISSN: 1790-5109 204 ISBN: 978-960-474-154-0

 RECENT ADVANCES in ARTIFICIAL INTELLIGENCE, KNOWLEDGE ENGINEERING and DATA BASES

customer satisfaction and appealing to the

emotions of customers provides a pathway for
solidifying a positive purchase decision. The
intelligent emotion-oriented eCommerce system
should therefore adapt to the following
requirements:
• Follow a social model that takes into
consideration the affective behaviour of the
customer; Fig. 2: Moving from Non-Purchasing Decisions to
• Follow a framework that would address the Purchasing Decisions.
aesthetic dimensions of an interface,
interaction and information delivery which There are a number of factors that can be
would facilitate easy communication between considered in achieving the above purchase 
the system and the customer; decision:
• Making provision for rational and reasoning • Design elements – e.g. typography, graphics,
requirements in which the system would audio, use of colours, frames, navigation, etc.
know how to appropriately respond to the • Customer’s initial mood
customer’s state and requests. • Internal and external disturbances that affects
the customer’s experience – e.g. noise,
It is important that an intelligent emotion- computer’s speed, etc.
oriented eCommerce system is able to seamlessly Taking into account above requirements for
adapt to a real world environment. It must be able design of emotion-oriented eCommerce systems
to response intelligently to unorthodox situations very useful will be a simulation model of this
posed by the customer. Therefore, the design system.
consideration for this system should involve but
not limited to emotion models, various interactive
techniques, adaptability and knowledge of
3.2 Simulation Model
perception, reasoning and customer’s profile and
A model of emotion should be able to identify
their general environment.
events and objects involved in the interaction
Many dimensions of website design have been
process. Emotion models should demonstrate the
outlined by authors dealing with aesthetics,
computer’s ability to express emotions at the right
customers’ trust and web usability. However,
intensity and appropriate time. Appraisal theories
there are few articles that answer the question -
have influenced the development of
What emotions influence the customer’s purchase
computational models of emotion. Essentially, our
decision and how these emotions can be
appraisal of a situation causes an emotional or
encouraged by the use of proficient website
affective response.
design? In essence, research studies lack a
The model most often used to incorporate
comprehensive framework explicitly geared
emotion into eCommerce system which is based
towards emotion-oriented eCommerce systems.
upon the appraisal theory is the OCC model [30].
Design requirements should include the
It addresses the problem of representing emotions
identification of non-purchasing emotions verses
not by a set of basic emotions but by grouping
purchasing emotions. This determination can be
emotions according to cognitive eliciting
made by conducting simulations and experiments
conditions. It believes that emotions arise from
designed to identify those emotions that are
valenced (positive or negative) reactions to
involved in the purchasing process. Companies
situations involving events, agents and objects
want their products sold, therefore changing a
[22]. Another computational model of emotions
negative emotion to a positive emotion which
based upon the appraisal theory is the EMA
leads to a favourable purchase decision, should be
(EMotion and Adaptation [11]. The agent’s
of paramount importance as illustrated in Fig 2.

ISSN: 1790-5109 205 ISBN: 978-960-474-154-0

 RECENT ADVANCES in ARTIFICIAL INTELLIGENCE, KNOWLEDGE ENGINEERING and DATA BASES

Disturbances
(noise, computer speed, etc)

eCommerce
System Design
Elements

Creating and eCommerce eCommerce Intelligent

maintaining a good System Customer Emotion Customer’s Emotions
emotion Targets Recognition Purchase Decision

Encouraging Negative Emotions Positive Emotions

purchasing decision

Fig. 3: Simulation model of intelligent emotion-oriented eCommerce system.

interpretation of its “agent-environment the purchasing decision of the customer and the
relationship” is considered to be a representation creation of a good mood.
of beliefs, desires, intentions, plans and A control-oriented method for simulation of
probabilities, which is referred to as causal eCommerce environment using MATLAB
interpretation to emphasize the importance of SIMULINK is suggested. SIMULINK allows
causal reasoning as well as the interpretative for the visual understanding how a dynamic
(subjective) character of the appraisal process system like eCommerce system operates.
[11]. Detailed dynamic models are required for
A model for simulation of intelligent performing ephemeral stability studies. An
emotion-oriented eCommerce systems following eCommerce system presents a dynamic system.
the basic control structure shown in Fig. 3 is SIMULINK applications can be applied to
proposed. The target variables identify what the represent this dynamic environment.
system should do and the input variables are
appropriately linked to the eCommerce system.
The input variable refers to the design elements 4 Conclusions
required for the design of an emotion-oriented Much thought and analysis should be invested in
eCommerce System. The customer would designing an interface that would elicit positive
interact with the system through a designed emotions. The business community would
interface. The output variables will affect the benefit significantly if they understand and
eCommerce customer and the decision they apply the right tools when considering the
make. There is also the opportunity to identify emotions of the customer. Consumers are
and collect the necessary bio-physiological data emotional beings and the emotions which they
emitted from the customer. Positive feedback, exhibit will ultimately affect their purchase
may lead to the eventual purchasing of a decision. It is imperative that studies and
product, whilst negative feedback will lead to research in the area of intelligent emotion-
product evaluation and system design oriented eCommerce systems has a scientific
modification. The decision made by the and empirical foundation.
customer, will establish if further emphasis need This paper raises issues of studies in
to be placed on the input variables. In addition, intelligent use of computers in an emotion-
designing such an environment must take into oriented eCommerce environment. Intelligent
consideration the disturbances which may affect emotion recognition techniques based on
modern mathematical models using

ISSN: 1790-5109 206 ISBN: 978-960-474-154-0

 RECENT ADVANCES in ARTIFICIAL INTELLIGENCE, KNOWLEDGE ENGINEERING and DATA BASES

computational intelligence are presented. The Intelligent Information and Engineering Systems,
requirements for designing of intelligent Springer, 2008, pp. 202-209.
[15] Kim, J., Moon, J.Y., Designing Emotional Usability in
emotion-oriented eCommerce systems are Customer Interfaces - Trustworthiness of Cyber
defined. A model for simulation of intelligent banking System Interfaces, Proc. International
emotion-oriented eCommerce systems is Conference on Human Factors in Computing Systems,
proposed. It is an important tool supporting the 1997.
[16] Kleinsmith, A., Bianchi-Berthouze N., Recognizing
experimental study and design of emotion- Affective Dimensions from Body Posture, Proc.
oriented eCommerce systems. International Conference of Affective Computing and
Emotion-oriented eCommerce research field Intelligent Interaction, 2007, pp.48-58.
is relatively new which presents many [17] Lockerd, A., et al., Tracking Mouse Movement
opportunities to research subject matters that Activity on Websites, a Tool for User Modeling, Proc.
Conference on Human Factors in Computing Systems,
show computers beginning to recognise and 2001.
respond to certain expressions of customer [18] Loiacono, E. T., et. al., Online Mood Induction, Proc.
emotion in an online shopping environment. 11th Americas Conference on Information Systems,
2005, pp. 2357-2360.
[19] Ortony, A., et al., The Cognitive Structure of
Emotions, Cambridge University Press, 1988.
References [20] Pavlovic, V., et. al., Visual Interpretation of Hand
[1] Anumba, C.J., Ruikar, K., Electronic commerce in Gestures for Human-Computer Interaction, University
construction—trends and prospects, Automation in of Central Florida Press, 1995.
Construction, Vol. 11, No. 3, 2002, pp. 265-275. [21] Petrushin, V. A., Emotion Recognition in Speech
[2] Barreto, A., Zai, J., Adjouadi M., Non-intrusive Signal: Experimental Study, Proc. Artificial Neural
Physiological Monitoring for Automated Stress Networks in Engineering, 2000.
Detection in Human-Computer Interaction, Workshop [22] Picard, R., Affective Computing, MIT Press, 1997.
on Human-Computer Interaction, 2007.
[23] Picard, R., Affective computing: challenges,
[3] Castellano, G., et. al., Multimodal emotion recognition
International Journal of Human Computer Studies,
from expressive faces, body gestures and speech,
Vol. 59, No. 1, 2003, pp.55-64.
Proc. 2nd International Conference on Affective
[24] Picard, R., Toward Agents that Recognize Emotion.
Computing and Intelligent Interaction, 2007.
Vivek, Vol. 13, 2000, pp. 3-13.
[4] Dormann, C., Seducing consumers, evaluating
[25] Pivk, A., et al., Intelligent Agents in E-commerce,
emotions, Proc. Conference on Human Factors in
(Online), [Link]
Computing Systems, 2001, pp. 10 -14.
Accessed: 24.11.2009, 2000.
[5] Du, T.C., et al., Dynamic vehicle routing for online
[26] Schleiffer, R., An intelligent agent model, European
B2C delivery, Omega, Vol. 33, No.2, 2005, pp. 33-45.
Journal of Operational Research, Vol. 166, No. 3,
[6] Ekman, P., An argument for basic emotions,
2005, pp. 666-693.
Cognition and Emotion, Vol. 6, No. 1, 1992, pp. 169-
[27] Tao, J., Tan, T., Affective Computing: A review,
200.
Proc. First International Conference on Affective
[7] Ekman, P., Friesen, W.V., Facial Action Coding
Computing & Intelligent Interaction, 2005.
System, Consulting Psychologists Press, 1977.
[28] Tucker, M.L., et al., Training tomorrow's leaders:
[8] Ekman, P., Friesen, W.V., Unmasking the Face,
Enhancing the emotional intelligence of business
Prentice Hall, 1975.
graduates, Journal of Education for Business, Vol. 75,
[9] Engelbrecht, A. P., Computational Intelligence: An
No. 6, 2000, pp. 331-341.
Introduction, Wiley, 2007.
[29] Turner, C. W., Studying Affective Responses to
[10] Frijda, N., The Nature of Emotion: Fundamental
Complex Technology, Workshop on Beyond the
Questions, Oxford University Press, 1994.
Usability Test, 2002.
[11] Gratch, J., Marsella S., Evaluating a computational
[30] Wang, H., et. al., Communicating Emotions in Online
model of emotion, Journal of Autonomous Agents and
Chat Using Physiological Sensors and Animated Text,
Multi-agent Systems, Vol. 11, No. 1, 2005, pp. 23-43.
Proc. ACM SIGCHI Conference on Human Factors
[12] Hudlicka, E., To feel or not to feel: The role of affect
in Computing Systems, 2004, pp. 1171-1174.
in human-computer interaction, International Journal
[31] Westerman, S.J., et al., Usability Testing Emotion-
of Human-Computer Interaction, Vol. 59, No.1, 2003,
Oriented Computing Systems - Psychometric
pp.1–32.
Assessment, (Online) [Link]
[13] Isen, A.M., Positive Affect and Decision Making. In
Accessed: 11.05.2009, 2006.
Lewis, M., Haviland, J.M., (eds.), Handbook of
[32] Zimmermann, P., et al., Affective computing - a
Emotions, Guilford Press, 2000, pp. 417-435.
rationale for measuring mood with mouse and
[14] Jascanu, V., Bumbaru S., Toward Emotional E-
keyboard, International Journal of Occupational
Commerce - The Customer Agent. In: Lovrek, J.,
Safety and Ergonomics, Vol. 9, No. 4, 2003, pp. 539-
Howlett, R. J., & Jain, L.C. (eds), Knowledge-Based
551.

ISSN: 1790-5109 207 ISBN: 978-960-474-154-0