Thinking About Thinking: Exploring Bias in Cybersecurity With Insights From Cognitive Science
Thinking About Thinking: Exploring Bias in Cybersecurity With Insights From Cognitive Science
Table of Contents
Exploring Bias in Cybersecurity 3
Aggregate Bias 6
Anchoring Bias 7
Availability Bias 8
Confirmation Bias 9
[Link] 2
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
System 1 System 2
Intuition Reasoning
Automatic Effortful
Implicit Explicit
Fast Slow
Metaphorical Exact
1 For a full overview of Dual Processing Theory (System 1/System 2) and behavioral economics (including bias), refer to
the work of Daniel Kahneman (academic articles, or for an accessible book, refer to “Thinking, Fast & Slow”)
[Link] 3
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
Both System 1 and System 2 are required to keep humans running smoothly through
their lives. Just as our automatic processes allow us to tie our shoelaces without
thinking about it, our effortful processes allow us to systematically think through
various pros and cons associated with difficult career or financial decisions.
An exceptional human trait is that we are able to think about thinking, which
means that we have the ability to consciously switch from System 1 thinking
to System 2 thinking.
Take a look at Figure 1. On the top, the two horizontal lines look like they are different
lengths.2 On the bottom, the image appears to show equilateral triangles.3 System 1
is responsible for your initial perception; your ability to automatically use contextual
cues to estimate the sizes of objects, and your ability to “fill in the blanks” by
establishing patterns.
Image A
However, we can also engage System 2 when looking at these images. If you
measure the two horizontal lines from Image A, you’ll see and logically understand
that the lines are the same length, but this won’t necessarily stop you from
perceiving them as two different lengths. When you take a closer look at Image
B, you’ll notice that none of the shapes are actually triangles, but you will continue
to see triangles in the image. Ultimately, we are not able to block these perceptual
illusions from occurring. This is not a problem in situations where the illusion has no
impact on our performance, or on our decisions. However, when faced with a critical
decision, depending on faulty impressions or gut feelings can result in errors in
reasoning and poor decision-making.
The concept that people engage in different types of thinking is not new. The
System 1 and System 2 paradigm aligns with psychological theories that pre-date
our current knowledge of cognitive and neuropsychology. Sigmund Freud, for
instance, believed that all human behavior was driven by unconscious urges and Image B
People spend the vast majority of their life immersed in System 1 thinking because
brains are built for efficiency. Brains require approximately 20% of the human body’s
energy,4 even at rest, which creates a need to prioritize saving mental time and
energy over engaging in resource-heavy analytic thought. Psychologists often refer
to our natural inclination towards conserving mental energy as being “cognitive
misers.” Misers avoid spending their assets, and similarly, humans avoid spending
mental effort. The major difference is that financial misers conserve resources on
purpose, but cognitive misers conserve resources subconsciously. In most cases,
[Link] 4
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
Responses to this question vary, but up to 80% of respondents will select “C.”
However, the correct answer is A.
When taking a deeper look at the options, you can see that it does not matter
whether or not we know if Anne is married. If she isn’t married, then Jack, a married
person, is looking at Anne, an unmarried person. If she is married, then Anne, a
married person, is looking at George, an unmarried person.
Anne
Unmarried
Jack George
Married Unmarried
Anne
Married
Thinking through the possible options on marital status and directional gaze
takes more effort than quickly identifying that Anne does not have a marital
status. The missing information about Anne’s marital status quickly registers as
“missing information” for many readers, and the miserly mind connects the missing
information to the “cannot be determined” answer. If you answered this question
correctly, it is possible that due to the context of this paper you assumed that the
question would be tricky, and therefore engaged in purposeful critical thinking. Bias
lurks within System 1, as snap judgments, stereotyping, and rules of thumb allow
us to take shortcuts to conserve mental energy—just like the shortcut that many
readers made when answering the question above.
5 Hector Levasque, as cited by Keith Stanovich, “Rational and Irrational Thought: The thinking that IQ Tests Miss”
[Link] 5
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
6 [Link]
pass%E2%80%9D-nist%E2%80%99s-digital-identity-guidelines
[Link] 6
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
While it is not possible to know the actual rate of password reuse, especially for
cross-sectional domains such as streaming services versus banking services,
the willingness of younger adults to share their passwords at a much higher rate
illustrates a potential misconception about which users of technology are riskiest.
The trope of making sure our grandparents do not send money to a Nigerian prince
may be far less important to our overarching security than identifying ways to
decrease credential sharing among a younger generation that perceives account
details and privacy through a different (and seemingly more lenient) lens.
Aggregate bias can also impact security investigations, in which an analyst wrongly
focuses on an individual due to the individual’s group membership (e.g., highly
technical person with a lot of access) rather than the facts or forensic information
that accurately describes the individual and their behavior. Focusing on an individual
due to a misapplication of characteristics can prompt analysts to fish for answers
and reasons to support their assumptions, which can delay identification of the true Overcoming
source of security issues.
aggregate bias
Overcoming aggregate bias through understanding of individual human behavior through understanding
is critical to security solutions that want to address human error and/or human risk
factors in protecting data. To achieve this goal, and to move beyond attributing of individual human
or misattributing behavioral characteristics to individuals, advanced behavioral
analytics that allow for self-to-self, self-to-peer, and self-to-global comparisons can
behavior is critical
help provide context for understanding complex individual behaviors. to security solutions
Anchoring Bias
that want to address
Anchoring occurs when a person locks onto a specific salient feature or set of human error and/or
features of information early in the decision-making process. This frequently
occurs with numbers, such as in sales, when one party in a negotiation proposes a
human risk factors in
price point. Once a price point is set, the number serves as an anchor for additional protecting data.
negotiations (which may be too high, too low, or even accurate).
Ask the first group, how many phishing attempts do you think we get every
week, 3,000?
Ask the second group, how many phishing attempts do you think we get every
week, 300,000?
You will likely find that the estimates for the group with the lower anchor (3,000) are
much lower than the estimates from the group with the higher anchor (300,000).
Of course, if your groups are comprised of people who deal with phishing in a
professional capacity, their answers may be anchored by their real-life experience!
[Link] 7
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
discount other influential information associated with the threat. At a broader level,
if a high-level person within an organization such as a CISO provides information
about potential threats or quantifies the potential impact of a threat, the CISO’s News cycles that
words prime and anchor employees lower on the organizational chart to focus
(sometimes incorrectly) on specific threats. focus on ransomware
or specific types of
Overcoming anchoring is particularly challenging, as awareness of the anchoring
phenomenon does not necessarily negate its effects. Recalculating estimates is not threats can influence
something that humans are particularly good at, especially when there are multiple
(or complex) factors at play. This is one specific type of bias where humans can, or analysts’ perceptions
should, depend more heavily on statistical analysis techniques that can decrease
of what is risky.
the impact of overly weighted early judgments in favor of balancing the impact of
new and critical information into their decision paradigm.
Availability Bias
Memory plays a large role in availability bias. The more frequently a person
encounters specific types of information, the more readily accessible the
information is in their memory. The availability of information, where certain types
of information are encountered more frequently, can impact how humans perceive
how likely an event is to occur (Figure 3).
# of people killed by
Luckily, analysts are in the business of thoroughly exploring data. While they are, at
cows per year: 20
times, susceptible to overestimating the probability of an event occurring, their job
is to consistently challenge their reasoning strategies and to consistently seek the
unexpected. Organizationally, leadership—who may be more or less technical, and
more or less privy to in-depth information that analysts see on a daily basis—is more # of people killed by
likely to be swayed by availability bias. This means that organizational cultures that sharks per year: 1
undervalue, or ignore, data that accurately represents the probability of specific
types of threat events may seek out or invest in solutions that are built to cope with
problems that carry very low likelihood of occurring. Figure 3. While shark attacks may garner more press coverage, cows
actually cause more deaths per year.
[Link] 8
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
Confirmation Bias
In the age of big data, we can almost always find data to support our opinions and
ideas. It is often possible to support multiple theories regarding why an incident
occurred or what type of risk is present on a network. For example, if you had to
argue that the earth is flat, you could find plenty of information to support that claim
online. Alternatively, you can find plenty of information to support the claim that
the earth is round. When people have a theory to explore when trying to answer a
question or support their opinion, they are highly susceptible to confirming their
beliefs by searching for (and often finding) support for their hunch. Confirming
our own beliefs by searching for and building information around our arguments,
while excluding or deemphasizing opposing viewpoints, is called confirmation bias.
Confirmation bias not only affects our reasoning strategies, but it also impacts our
memory of information. People tend to focus on and remember information that
confirms or aligns with their beliefs, while discounting or forgetting information that
opposes their viewpoint.
Analysts, with the best of intentions, may find themselves spending a lot of time
looking for causes or issues associated with an adverse event by only searching
for causes or issues that align with their personal theories or insight. This is
particularly relevant for experienced analysts who may “decide” what happened
prior to investigating an event. Their expertise and experience, while extraordinarily
valuable, can be a weakness if they investigate incidents in a way that only supports
their existing belief.
7 For more information on mental toolkits and psychological factors associated with intelligence analysis, see Richards J.
Heuer, Jr. (1999). Psychology of Intelligence Analysis.
[Link] 9
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
While there are individual differences in how people answer this question (we’ve all
seen the game show participants who are willing to “risk it all” for one more chance
to win big), the “sure thing” is often perceived as the best option when it comes
Buyers of security
to choices associated with gains. When we think about what people choose, and
we’re talking about a positive outcome, people tend to make the less risky choice by solutions can
choosing the sure thing.
overcome the impact
But what happens when the decision, and the choices associated with the decision,
of framing effects
aren’t associated with something positive like winning money? What if the choices
are presented in a way that highlights a loss, or the chance for a loss? Let’s revisit by slowing down
the money question: would you rather definitely lose $100, or have a 15% chance of
losing $1,000? and thinking more
analytically about the
What we see (of course, not perfectly reversed due to those pesky individual
differences) is that people are more willing to take the probabilistic (riskier) option problems they are
when they are faced with a loss.
trying to solve, and the
Security problems are often aggressively worded, and use negative framing
suggested efficacy of
strategies to emphasize the potential for loss. This strategy prompts security
decision makers to, at times, invest in security solutions that are expensive (or the solutions offered.
overkill!) to address overly specific and low-probability risk factors. Consider the
vendor who promotes that “one out of five small companies never got their data
back after a ransomware attack!” The focus on the one company that didn’t get data
back versus the four that did over-emphasizes the risk for companies.
[Link] 10
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
the fundamental attribution error is called the self-serving bias, where the individual
making the mistake places the “blame” on environmental or contextual factors
rather than internalizing the mistake as an internal trait.
Coping with fundamental attribution errors, and the self-serving bias, requires
personal insight and empathy. It can be extraordinarily difficult to engage in
consistent self-assessment to determine when we may be placing blame on a
person rather than blame on environmental factors that impacted a person’s
behavior. It is also difficult to acknowledge when we are responsible, due to our own
shortcomings, for adverse events or outcomes. What we can do is practice empathy
and build our capacity for giving others the benefit of the doubt. For supervisors and
leaders, acknowledging imperfections/failures can help create a more resilient and
dynamic culture. For the people designing complex software architectures, consider
that your perspective is highly security focused—while your users’ motivations may
not be—and that their failures are not because they are stupid, but because
they’re human.
[Link] 11
Thinking About Thinking: Exploring Bias in
Cybersecurity with Insights from Cognitive Science
Of the biases outlined in this paper, several of them can be addressed directly
through the use of improved advanced analytics. The prime example is aggregate
bias. As we develop the capabilities to understand individual human behavior, rather
than group human behavior, we can get much better at applying policies, rules,
and constraints on those individuals who push the boundaries of risky behavior
or on those individuals who have the greatest negative security impacts on an
organization. The ability to do this, without the application of broad or inflexible rules
and restrictions generated for a specific group (say, for older adults or for engineers
who create and edit source code), can promote a more resilient workforce that is
able to work efficiently and effectively with fewer security-induced roadblocks.
Decreasing frustration and friction associated with security protocols is critical, and
by understanding individual behavior through advanced behavioral analytics, we are
getting closer to an adaptive security framework that benefits users, organizations,
and security professionals.
However, there are other biases that require a far more human approach or that do
not have an obvious technology-based strategy. One bias that requires human effort
is overcoming the impact of the fundamental attribution error. While organizations
can raise awareness of this phenomenon, individuals within an organization must
take on the responsibility for challenging their own assumptions about themselves
and about others. That said, when creating new technologies, use of design thinking
techniques and working towards integrating human-centered design methods
can help.
As a security professional, take a few moments to walk through the six biases
described in this paper:
1. Do you or your colleagues make assumptions about individuals but use group
characteristics to form your assumptions?
2. Have you ever been hung up on a forensic detail that you struggled to move
away from to identify a new path for exploration?
4. When you run into the same problem over and over again, do you slow down
to think about other possible solutions or answers?
5. When offered new services and products, do you assess the risk (and your
risk tolerance) in a balanced way? From multiple perspectives?
6. And finally, does your team take steps to recognize your own responsibility
for errors or for engaging in risky behaviors, and give credit to others who
may have made an error due to environmental factors?
After taking the time to review our experiences, professional environment, and
decision-making habits, we’ll all likely find that some of these biases impact us,
our teams, or our companies more heavily than others. It’s critical, even in today’s
environment of never-ending alerts and dangers, that cybersecurity teams and
professionals slow down and think more deeply and strategically in order to combat
these biases. If not, we may find that biases are blinding us to the real threats.
[Link] 12
About the Author
Dr. Margaret Cunningham is a Principal Research Scientist for Human Behavior in
Forcepoint’s X-Labs. Her current passion is identifying critical interactions between
humans and technology through a socio-technical systems lens, with a goal of establishing
a better human-centric model for improving cybersecurity.
About Forcepoint
Forcepoint is transforming cybersecurity by focusing on what matters most: people’s behavior
as they interact with critical data and systems. This human-centric approach to cybersecurity
frees employees to innovate by understanding the normal rhythm of user behavior and the flow
of data in and out of an organization. Forcepoint behavior-based solutions adapt to risk in real
time and are delivered via a converged security platform to protect network users and cloud
access, prevent confidential data from leaving the corporate network, and eliminate breaches
caused by insiders. Based in Austin, Texas, Forcepoint protects the human point for thousands
of enterprise and government customers in more than 150 countries.
[Link]/contact
© 2019 Forcepoint. Forcepoint and the FORCEPOINT logo are trademarks of Forcepoint. All other trademarks
used in this document are the property of their respective owners.
[BIAS IN CYBERSECURITY-GLOBAL-REPORT-US-EN] 200079.052119