Cognition

Cognition refers to the mental activities that let people think, understand, remember, and use
knowledge. In HCI, the main function of cognition is to help users control the computer so they
can do what they want. To perform tasks, users must attend to interface elements, remember their
properties, understand labels and icons, and reason about what the system will do.

The chapter adopts an information-processing view of cognition, where people receive information,
process it in their mind, store some of it, and use it to act. It emphasizes that cognitive abilities are
powerful but limited, and they are influenced by the task and the design of the user interface. The
chapter focuses on five elementary cognitive capabilities that are especially relevant to computer use:
cognitive control, memory, attention, reasoning, and decision-making.

General findings about cognition

5.1.1 Cognition helps us set goals and keep focused on them

Cognition allows users to form goals, such as “send this email” or “change my privacy settings,” and
then maintain focus on these goals during interaction. Human memory does not store everything;
instead, it keeps information that is useful for achieving current and future goals, and these goals
guide what people attend to and remember.

5.1.2 Cognition is limited

People have strict limits in working memory, attention, and planning. Working memory can only
maintain a few items at once; visual attention is restricted to a narrow region; and people often need
external aids like notes or calculators to compensate. These limitations strongly shape how well users
can interact with complex systems.

5.1.3 Cognition is based on internal models of reality

Users build internal mental models of how systems work, which they use to reason, plan, and predict
what will happen. Interface metaphors such as the desktop—folders, documents, trash—help users
construct such models by connecting digital actions to familiar real‑world concepts.

5.1.4 Cognition is necessary for learning and adaptation

People’s cognitive, motor, and perceptual processes constantly adapt to their environments. Instead
of simply following rigid scripts, users experiment, form beliefs, try new tactics, and refine their
behavior when technologies or usage contexts change.
5.1.5 Cognition requires energy and effort.
Mental effort is the energy used to control our thinking to achieve goals, with task effort (extra energy
when tasks are demanding or new) and state effort (energy needed to protect performance when we
are tired or affected by other factors); even though effort feels negative, it also protects us from wasting
energy on less important activities.

5.2 Cognitive control – English mini-lecture

Cognitive control is the ability to direct thinking and action toward a goal while managing limited
cognitive resources. When interacting with computers, users face many competing stimuli, so
cognitive control must decide which representations to activate, which to suppress, and when to switch
tasks.

5.2.1 Activating goals

In complex tasks with multiple steps, cognition activates the right subgoals at the right time. The goal
activation model explains how a main task is decomposed into subgoals that are turned on and off
as the task progresses. Priming refers to subconscious activation by cues, previous goals can
interfere with current goals, and postcompletion errors occur when actions that should happen after
goal completion are forgotten.

5.2.2 Activating task representations

To perform a task, cognition must make relevant task representations available and inhibit
irrelevant ones. When searching for an icon, you should avoid revisiting already checked locations;
this phenomenon is called inhibition of return. In information-rich environments, inhibition helps
users resist attention capture from every flashing or blinking element.

5.2.3 Choosing actions

Cognition is needed to choose actions when rewards are delayed. Through experience, users learn
value estimates of actions based on recurring contextual cues, forming associations that often
operate subconsciously. Users can also reason deliberately, simulating possible routes in their mind
before acting.

5.2.4 Multitasking
Multitasking is a resource-sharing problem in which tasks compete for limited resources such as
attention, the motor system, and working memory. Some resources are all-or-nothing (for
example, one dominant hand), while others can be shared more flexibly. Multiple Resource Theory
(MRT) explains conflicts between tasks in terms of perceptual modality, processing stage, and
cognitive coding: tasks that occupy the same slots in this cube produce more interference (i.e., Figure
5.2).

5.2.5 Predictive control

In dynamic environments, cognitive control must be proactive instead of merely reactive. The
Bayesian brain hypothesis views the brain as a statistical estimator that updates probabilities about
the state of the world when new sensory information arrives. The less expected an event is, the greater
the surprise, so the brain constantly adjusts its internal model to minimize uncertainty.

5.2.6 Cognitive workload

Cognitive workload is the mental effort required to maintain cognitive control in a task. Learning a
new system can feel demanding because of increased task effort and state effort. Workload can be
measured with instruments such as the NASA-TLX, and over time people develop minimum-effort
strategies; as skills become automatic, perceived workload decreases.
5.3 Memory and learning
Memory is critical for computer use: without it, users could not complete tasks like filling forms,
changing settings, or navigating pages. According to dual-process models, there are separate
processes for conscious and unconscious remembering, and several distinct memory systems with
different functions and neural bases.

5.3.1 Working memory

Working memory (WM) is the temporary maintenance and manipulation of representations needed
for action. It is separate from long-term memory because it is limited in time and capacity: we can only
keep a few items active at once, probably around three to five. Items in WM fade quickly unless actively
maintained. Tasks like the n-back experiment show how difficult it is to keep several items updated
simultaneously, revealing our radically limited ability to maintain multiple ideas. WM is needed in many
interactive tasks, such as copying text between applications or dealing with interruptions.

5.3.2 Long-term memory

Long-term memory (LTM) is the set of memory systems that store what we know and what we have
experienced over a long time. Declarative memory is the part we can consciously recall and talk
about. It has two subtypes: semantic memory (facts and meanings) and episodic memory (events in
our life). Non-declarative memory is the part that affects our behavior without us needing to
consciously recall anything. It includes procedural skills and habits, priming, conditioning, and
non-associative learning.

When you remember a password, you may use both systems. If you can say or spell the password,
that uses declarative memory. If your fingers can type the password automatically on a familiar
keyboard even when you are not thinking about each character, that uses procedural memory, which
is non-declarative. Figure 5.3 organizes memory into working memory and long-term memory, and
then further into declarative and non-declarative types and their subtypes.

5.3.3 A three-stage view of memory functioning

Memory can be understood in terms of encoding, storage, and retrieval. Encoding forms memory
traces during interaction; storage maintains them between encoding and retrieval; retrieval accesses
them later, for example when using the same user interface again. The levels of processing effect
states that deeper mental processing during encoding produces stronger memory traces.
5.3.4 Forgetting
Between encoding and retrieval, some memories are lost or become harder to access. Decay theory
says memory traces lose activation over time, whereas interference theory says memories become
confused with each other. Not all memories are forgotten in the same way; forgetting also reflects
adaptation to the statistical structure of environments, helping us prioritize what is important for
survival and function.

5.3.5 Learning over time

Every time we practice a task, small neural changes occur that prepare us to perform similar tasks
better in the future. Performance usually improves a lot at the beginning of practice and then the gains
become smaller and smaller. This pattern is called diminishing returns. In motor learning, this
relationship between reaction time and number of practices is captured by the power law of practice:

𝑅𝑇 = 𝑎𝑃 −𝑏 + 𝑐

Here, 𝑅𝑇 is the reaction time, 𝑃 is the number of completed practices, and 𝑎, 𝑏, and 𝑐 are free
parameters that describe the starting level, the speed of improvement, and the lower limit of
performance. As practice continues, not only does performance become faster; it also becomes more
stable and automatic. People make fewer errors, feel more confident, and rely less on conscious
control.

5.3.6 Designing practices

Designing practice means deciding how people should practice a skill so that learning is efficient. The
text highlights three main parameters:
 1. Scheduling of practice – when practice trials happen over time.
• Massed practice puts trials close together (cramming) and is worst for long-term
learning.
• Equi-spaced practice spreads trials evenly.
• Expanding practice gradually increases the spacing between trials and is best for
consolidation.
2. Selection of practices – which specific parts of the skill are practiced.
• Focused practice uses isolated tasks with clear objectives and corrective feedback,
especially on weak areas.
3. Feedback – how practice results are reported and used to change strategy.
• Feedback can indicate performance (scores, speed), correct performance (point out
errors and fixes), or enhance strategy (suggest better ways of doing the task).

5.3.7 Stages of skill development

Skill development in HCI is usually described in three stages: novice, intermediate, and expert.
1. Novice
• Novices struggle even with simple tasks and their performance is highly variable
(sometimes they succeed, sometimes they fail).
• They rely heavily on visible interface elements such as menus, icons, and step-by-step
instructions.
• When the interaction strategy or interface changes, their performance can drop
sharply.
2. Intermediate
• Intermediate users reach an acceptable level of performance for everyday tasks.
• Many computer users are self-taught and stay at this level; they know many tricks but
their mental models of the system may be shallow or fragmented.
• Their skills may not be robust: they may not transfer well to new devices or new tasks.
3. Expert
• Experts achieve the highest, most stable performance, but only after very large
amounts of deliberate practice, typically thousands of hours.
• They have deep conceptual representations of how technology works and can adapt
their skills to new situations.
• Often, their practice has been guided by coaches, mentors, or highly structured
training.

5.3.8 Knowledge-in-the-world, knowledge-in-the-head

In HCI, we distinguish between two places where knowledge can be stored:
1. Knowledge-in-the-head
This is knowledge stored in the user’s own memory. Examples include remembering a
password, knowing keyboard shortcuts, or knowing the steps of a procedure without looking
at instructions. The user must recall this information internally.
2. Knowledge-in-the-world
This is knowledge stored outside the user, in the environment or in the interface. Examples
include written passwords on paper, labels on buttons, icons, checklists, or on-screen hints.
Instead of remembering everything, users read or perceive this information directly from the
world.
Users often externalize knowledge to reduce the load on their memory, but this can also make them
dependent on having access to those external aids. In complex environments like airplane cockpits,
knowledge is distributed across people, instruments, and procedures; cognition is not just inside one
head, but spread across the whole system.

5.4 Reasoning and decision-making

Reasoning is needed when users do not have direct access to all the information they need and must
draw conclusions from what they know. It involves transforming existing beliefs and rules into new
beliefs—for example, inferring that an icon in another operating system has the same function as an
icon in Windows. Prediction is a special form of reasoning about the future; users must infer what will
or might happen from what they observe now.

Reasoning is especially important for complex systems. Users may need to reason about what a piece
of software can do based on advertisements and word of mouth, what will happen when they issue a
new command, or what risks are involved in a decision.
5.4.1 Mental models
Mental models are memory-based representations of interactive systems that users employ for
reasoning, inference, and prediction. For example, a mental model of a thermostat tells the user how
temperature affects warmth in a room. Mayer and Gallini’s study used parts-and-steps illustrations of
a pump system and found that these helped students both recall and solve problems, showing that
good mental models support understanding of internal device states.

Users’ mental models are often incomplete and fragmented, built from episodes of use rather than
from full system understanding. In practical problem-solving, users may prefer to avoid effortful
reasoning and instead try things out to see what happens.

5.4.2 Decision-making
Decision-making refers to situations where multiple options are given and one or a subset must be
chosen. This is illustrated using a virus-removal example: users must choose between options with
different probabilities of success and different costs (e.g., slowing down the computer). In the general
case, each option has an expected gain and loss with certain probabilities, and users may not always
pick the objectively best option.

Prospect theory claims that people evaluate choices relative to a reference point, are often
risk-averse for gains and risk-seeking for losses, and use reference points such as the status quo or
the most recent choice. In HCI, decisions like choosing privacy settings or password length can be
analysed through expected gains, losses, and probabilities.

Multi-attribute choice arises when options differ on multiple attributes (price, type, size, location,
etc.). In such tasks, users must move their limited visual attention between options and attributes, and
working memory must keep track of which attributes have been considered. Because this is hard,
users often satisfice—they pick the first option that seems “good enough”—or rely on external aids
such as lists, spreadsheets, or advice.

5.4.3 Decision heuristics

Kahneman’s dual-system view distinguishes a fast, intuitive System 1 and a slower, reflective
System 2. When full analysis is not possible, users rely on cognitive heuristics, rules of thumb that
give quick but possibly biased solutions.

These are several heuristics:

• Anchoring – choices are biased toward a known reference solution (e.g., starting with a
familiar application when selecting software).
• Decoying – a reference option makes another option less visible or attractive.
• Availability – events that come to mind easily (recent or vivid) are judged more probable.
• Status quo bias – preference for the current or default option.
• Bandwagon bias – preference for options that others are choosing.

Heuristics are useful because they allow quick decisions, but they can also lead to systematic biases
in HCI decisions.