Algorithm Appreciation: People

Prefer Algorithmic To Human

Judgment
Jennifer M. Logg
Julia A. Minson
Don A. Moore

Working Paper 17-086

 Algorithm Appreciation: People Prefer
Algorithmic To Human Judgment
Jennifer M. Logg
Harvard Business School

Julia A. Minson
Harvard University

Don A. Moore
University of California, Berkeley

Working Paper 17-086

Copyright © 2018 by Jennifer M. Logg, Julia A. Minson, and Don A. Moore

Working papers are in draft form. This working paper is distributed for purposes of comment and discussion only. It may
not be reproduced without permission of the copyright holder. Copies of working papers are available from the author.
Running Head: ALGORITHM APPRECIATION

Algorithm Appreciation:
People prefer algorithmic to human judgment

Jennifer M. Logg
Harvard University

Julia A. Minson
Harvard University

Don A. Moore
University of California, Berkeley

Submitted: April 20, 2018

** Please do not circulate without authors’ permission.**

Author Note

Jennifer M. Logg, Harvard Business School, Harvard University

Julia A. Minson, Harvard Kennedy School, Harvard University
Don A. Moore, Haas School of Business, University of California, Berkeley

Correspondence concerning this article should be addressed to Jennifer M. Logg, Harvard

Business School, Harvard University, Baker Library, Bloomberg Center 433, Harvard Business
School, Boston, MA 02163. E-mail: jlogg@[Link]

1
ALGORITHM APPRECIATION

Abstract

Even though computational algorithms often outperform human judgment, received wisdom

suggests that people may be skeptical of relying on them (Dawes, 1979). Counter to this notion,

results from six experiments show that lay people adhere more to advice when they think it

comes from an algorithm than from a person. People showed this sort of algorithm appreciation

when making numeric estimates about a visual stimulus (Experiment 1A) and forecasts about the

popularity of songs and romantic matches (Experiments 1B and 1C). Yet, researchers predicted

the opposite result (Experiment 1D). Algorithm appreciation persisted when advice appeared

jointly or separately (Experiment 2). However, algorithm appreciation waned when: people

chose between an algorithm’s estimate and their own (versus an external advisor’s; Experiment

3) and they had expertise in forecasting (Experiment 4). Paradoxically, experienced

professionals, who make forecasts on a regular basis, relied less on algorithmic advice than lay

people did, which hurt their accuracy. These results shed light on the important question of

when people rely on algorithmic advice over advice from people and have implications for the

use of “big data” and algorithmic advice it generates.

Keywords: algorithms; accuracy; advice-taking; forecasting; decision-making; theory of machine

2
ALGORITHM APPRECIATION

Algorithm Appreciation:
People prefer algorithmic to human judgment

Although people often receive advice from other people, the rise of “big data” has

increased both the availability and utility of a new source of advice: algorithms. The superior

accuracy of algorithmic judgment relative to human judgment (Dawes, Faust, & Meehl, 1989)

has led organizations to invest in the power of algorithms -- scripts for mathematical calculations

-- to sift through data and produce insights. Companies like Johnson & Johnson and Jet Blue

invest in complex algorithms to hire promising employees, track the satisfaction of current

employees, and predict which employees are at risk for leaving the organization. Individuals in

need of legal counsel can now use “Robo-advisors” such as DoNotPay to help them contest

parking tickets or even file for political asylum. Indeed, decision-makers now rely on algorithms

for personal use, in place of human secretaries, travel agents, headhunters, matchmakers, D.J.s,

movie critics, cosmetologists, clothing stylists, sommeliers, and financial advisors (Siri, Google

Search, and spell-check; Kayak, LinkedIn, OkCupid, Pandora, Netflix, Birchbox, Stitch Fix,

Club W, and Betterment, respectively).

Such widespread reliance on algorithmic advice seems at odds with the judgment and

decision-making literature which demonstrates human distrust of algorithmic output, sometimes

referred to as “algorithm aversion” (Dietvorst, Simmons, & Massey, 2015).1 This idea is so

prevalent that it has been adopted by popular culture and the business press. Articles advise

business leaders on how to overcome aversion to algorithms (Harrell, 2016). Companies in the

business of selling algorithmic advice often go to great lengths to present it as purely human-

1 While this influential paper is about the effect that seeing an algorithm err has on people's likelihood of choosing
it, it has been cited as being about how often people use algorithms in general.

3
ALGORITHM APPRECIATION

generated. Stitch Fix, for example, uses a combination of algorithmic and human judgment to

provide clothing recommendations to consumers but includes a personalized note from a stylist

with each shipment in order to focus consumers’ attention on the human component.

In the present paper, we trace the history of research examining human responses to

algorithmic output and highlight boundary conditions for empirical evidence supporting

algorithm aversion. We then present results showing that under conditions that apply to many

decisions, and across a variety of estimation and forecasting tasks, people actually prefer advice

from algorithms to advice from people.

Prior Research on Algorithm Aversion

The first scholarly reference to psychological distrust of algorithms may belong to Meehl

(1954). Importantly, it appears in the discussion section of a classic book outlining the

predictive superiority of algorithms over human experts. Specifically, when researchers shared

their findings with the experts in question, the conclusions were met with skepticism. It appears

that the experts in the 1950s were hesitant to believe that a linear model could outperform their

judgment. Similar anecdotes have circulated from other scholars (Dana & Thomas, 2006;

Dawes, 1979; Hastie & Dawes, 2001). Over time, these anecdotal claims have become received

wisdom in the field of judgment and decision making. In his best-selling book, Thinking Fast

and Slow, Kahneman recounts Meehl’s story: “From the very outset, clinical psychologists

responded to Meehl’s ideas [accuracy of algorithms] with hostility and disbelief” (2011, pg.

227).

Decades passed before the mistrust of algorithms was empirically tested. The results did

not always support the received wisdom. On the one hand, in subjective domains governed by

personal taste, people relied on friends over algorithmic recommender systems for book, movie,

4
ALGORITHM APPRECIATION

and joke recommendations (Sinha & Swearingen, 2001; Yeomans, Shah, Mullainathan, &

Kleinberg, 2018). Participants who imagined themselves as medical patients more frequently

followed a subjectively worded recommendation for an operation from a doctor than from a

computer (Promberger & Baron, 2006). And an influential set of papers demonstrates that after

seeing an algorithm err, people relied more on human judgment than an algorithm’s (Dietvorst,

Simmons, & Massey, 2015; Dzindolet, Pierce, Beck, & Dawe, 2002).

On the other hand, work in computer science shows that participants considering logic

problems agreed more with the same argument when it came from an “expert system” than when

it came from a “human” (Dijkstra, Liebrand, & Timminga, 1998). This preference for

algorithmic output persisted, even after they saw the algorithm err (Dijkstra, 1999). In other

studies, people outsourced their memory of information to algorithmic search engines (Sparrow,

Liu, & Wegner, 2011; Wegner & Ward, 2013). In the meantime, companies continue to worry

about aversion to their algorithms (Haak, 2017) from their own employees, other client

organizations, and individual consumers, even while many people entrust their lives to

algorithms (e.g., autopilot in aviation).

Who is relying on which advice and for what purpose?

Given the richness of the decision-making landscape, it is perhaps not surprising that

different studies have arrived at different results. One important feature that distinguishes prior

investigations is the comparison of people’s reliance on algorithmic judgment to their reliance on

their own, self-generated judgments. This approach has intuitive appeal, as managers and

consumers often choose whether to follow their own conclusions or rely on the output of an

algorithm. However, given the evidence that people routinely discount advice, such paradigms

do not answer the question of how people react to algorithms, as compared to other advisors.

5
ALGORITHM APPRECIATION

Indeed, the robust result from literature on utilization of human advice is that individuals

regularly (and inaccurately) discount the advice of others when making quantitative judgments

(Yaniv & Kleinberger, 2000). Research has attributed such underweighting of other’s input to

egocentrism (Soll & Mannes, 2011). Furthermore, an extensive literature on overconfidence

repeatedly demonstrates that individuals routinely report excessive confidence in their own

judgment relative to that of their peers (Gino & Moore, 2007; Logg, Haran, & Moore, 2018;

Moore & Healy, 2008; Moore, Tenney, & Haran, 2015). These literatures therefore raise the

question of whether individuals insufficiently trust algorithms (relative to human advisors) or

merely overly trust themselves? Thus, a direct comparison of advice utilization from an

algorithm versus a human advisor would prove especially useful.

Another important concern is the quality of the advice in question. The now classic

literature on clinical versus actuarial judgment has demonstrated the superior accuracy of even

the simplest linear models relative to individual expert judgment (Grove, Zald, Lebow, Snitz, &

Nelson, 2000). Although it is unlikely that research participants are aware of these research

findings, it could be the case that they believe that algorithmic judgment is only useful in some

domains. This issue of advice quality particularly plagues studies that compare

recommendations from an algorithm to recommendations from a human expert, such as a doctor

(Promberger & Baron, 2006). As the normative standard for how a single expert should be

weighed is unclear, it is also unclear what we should conclude from people choosing an

algorithm’s recommendation more or less frequently than a doctor’s. Thus, the accuracy of

algorithmic advice becomes important for interpreting results. In order to isolate the extent to

which judgment utilization is specifically driven by the fact that the advisor happens to be an

algorithm, the human and algorithmic advice should be of comparable quality.

6
ALGORITHM APPRECIATION

Yet another important factor is the domain of judgment. It may be perfectly reasonable

to rely on the advice of close friends rather than a “black box” algorithm when making a decision

reflecting one’s own personal taste (Yeomans et al., 2018). Conversely, individuals may feel

more comfortable with algorithmic advice in domains that feature a concrete, external standard

of accuracy, such as investment decisions or sports predictions. Relatedly, the extent to which

some domains may appear “algorithmically appropriate” may depend on the historical use of

algorithms by large numbers of consumers. For example, most people have grown comfortable

with weather forecasts from meteorological models rather than one’s neighbors because

meteorological models have enjoyed widespread use for decades. Conversely, the idea of

fashion advice from algorithms is still relatively new and may face greater resistance.

When understanding algorithmic utilization, the algorithms’ prior performance, when

made available, is useful to consider. The work of Dietvorst et al. (2015) demonstrates that

when choosing between their own (or another participant’s) estimate and an algorithm’s

estimate, participants punished the algorithm after seeing it err, while showing greater tolerance

for their own mistakes (or another participant’s). Importantly, because the algorithm is, on

average, more accurate than a single participant, choosing the human estimate decreases

judgment accuracy. However, in the control conditions of the Dietvorst et al. studies,

participants chose the algorithm’s judgment more frequently than they chose their own (or

another person’s). And although the authors consistently and clearly state that “seeing

algorithms err makes people less confident in them and less likely to choose them over an

inferior human forecaster” (pg. 10), other research cites the paper as demonstrating generalized

“algorithm aversion.”2

2For instance, Petropoulous, Fildes, & Goodwin (2016) cite Dietvorst et al. (2015) as evidence of people “rejecting
models in favor of their own (mis)judgments even when given evidence of the superior performance of models” (p.

7
ALGORITHM APPRECIATION

Measuring Algorithm Appreciation

The current research revisits the basic question of whether individuals distrust

algorithmic advice more than human advice. In our experiments, we benchmark people’s

responses to advice from an algorithmic advisor against their responses to advice from a human

advisor, to take into account that people generally discount advice.

Importantly, we also control for the quality of the advice. Participants in different

conditions receive identical numeric advice, merely labeled as produced by a human or an

algorithmic source, which differentiates our work from past work. Thus, any differences we

observe are not a function of advice accuracy, but merely the inferences participants might make

about the source. Across experiments, we describe the human and algorithmic advice in several

different ways in order to test alternative explanations based on participants’ interpretations of

our descriptions (Table 1). Finally, we use a variety of contexts to examine utilization of advice

across different judgment domains.

We employ the Judge Advisor System (JAS) paradigm to measure the extent to which

people assimilate advice from different sources (Sniezek & Buckley, 1995). The JAS paradigm

requires participants to make a judgment under uncertainty, receive input (“advice”), and then

make a second, potentially revised judgment. Where appropriate, we incentivized participants’

final judgments. The dependent variable, Weight on Advice (WOA), is the difference between

the initial and revised judgment divided by the difference between the initial judgment and the

advice. WOA of 0% occurs when a participant ignores advice and WOA of 100% occurs when a

participant abandons his or her prior judgment to match the advice.

851). And Prahl and Van Swol (2017) cite the paper as evidence that “humans are trusted more than computers” (p.
693).

8
ALGORITHM APPRECIATION

Extensive prior research has documented that when seeking to maximize judgment

accuracy, a person who receives advice from a single randomly-selected individual should

generally average their own judgment with the advice, as reflected in a WOA of 50% (Dawes &

Corrigan, 1974; Einhorn & Hogarth, 1975). Yet, people tend to only update 30-35% on average

toward advice, incurring an accuracy penalty for doing so (Liberman, Minson, Bryan & Ross,

2012; Minson, Liberman & Ross, 2011; Soll & Larrick, 2009). Various attempts to increase

advice utilization have demonstrated the effectiveness of paying for advice (Gino, 2008);

highlighting advisor expertise (Harvey & Fischer, 1997; Sniezek, Schrah, & Dalal, 2004); and

receiving advice generated by multiple individuals (Mannes 2009; Minson & Mueller, 2012).

Could providing advice from an algorithm also increase people’s willingness to adhere to

advice?

In addition to capturing advice utilization using a more precise, continuous measure,

weight on advice also taps a somewhat different psychological phenomenon than does choosing

between one’s own judgment and that of an algorithm. Prior work demonstrates that people are

quite attached to their intuitive judgments and prefer to follow them even while explicitly

recognizing that some other judgment is more likely to be objectively correct (Woolley & Risen,

2018). This could lead people to endorse an advisor but ultimately fail to act on the advice. By

contrast, weight on advice, especially under incentivized conditions, captures actual change in

the participant’s own judgment as a function of exposure to the advice.

The Present Research

In six experiments, participants made quantitative judgments under uncertainty and

received advice. In one experiment, active researchers in the field predicted how participants

responded to advice. In the majority of experiments, we manipulated the source of advice

9
ALGORITHM APPRECIATION

(human versus algorithm) and examined how much weight participants gave to the advice. In

Experiments 2 and 4, participants choose the source of advice they preferred.

Across our experiments, we find that people consistently give more weight to equivalent

advice when it is labeled as coming from an algorithmic versus human source. Yet, researchers

predict the opposite, that those participants will display algorithm aversion. Together, these

results suggest that algorithm aversion is not as straightforward as prior literature suggests, nor

as contemporary researchers predict.

In our experiments, we report how we determined sample sizes and all conditions. All

sample sizes were determined a priori by running power analyses (most at 80% power). Where

possible, we based effect size estimates on prior experiments. Where that was not possible, we

conservatively assumed small effect sizes, which led us to employ larger sample sizes. Pre-

registrations (including all exclusions), materials, and data are posted as a supplement online at

the Open Science Framework:

[Link] (The journal editor has

access to the non-anonymized pre-registrations with correct time stamps.) We pre-registered

analyses (and exclusions) for Experiments 1B, 1C, 1D, 2, 3, and 4. We ran Experiment 1A

before pre-registration became our standard practice.

Experiments 1A, 1B, and 1C:

Benchmarking Reliance on Algorithmic Advice against Reliance on Human Advice

Experiments 1A, 1B, and 1C test the extent to which people are willing to adjust their

estimates in response to identical advice, depending on whether it is labeled as coming from a

human versus an algorithmic advisor. To generate advice, we used one of the simplest

algorithms: averaging multiple independent judgments. Doing so allows us to provide high

10
ALGORITHM APPRECIATION

quality advice and truthfully describe it as coming from either people or from an algorithm. In

Experiment 1A, participants guessed an individual’s weight from a photograph. In Experiment

1B, participants forecasted the popularity of songs on the upcoming week’s Billboard Magazine

Hot 100 Music Chart. In Experiment 1C, participants predicted how another person would judge

a potential romantic partner.

Method: Experiment 1A (Visual Estimation Task)

The sample included 202 participants (90 women; 112 men; Mdn age = 28). Accurate

final responses entered participants into a raffle for a $10 bonus. Participants estimated the

weight of a person in a photograph (see Figure 1 in the Appendix) at two points in time: before

and after receiving advice, which either came from a person or an algorithm. Immediately after

each estimate, participants indicated their confidence in that estimate: “How likely is it that your

estimate is within 10 pounds of the person's actual weight?” on a scale from 0 = no chance to

100 = absolutely certain.

Everyone received the same advice (163 pounds), which was actually the average

estimate of 415 participants in another experiment (Moore & Klein, 2008). Importantly, this

estimate was nearly perfect (164 pounds was the actual weight). The advice was either labeled

as coming from other people or an algorithm. The manipulation wording for this and all other

experiments is listed in Table 1. After reporting their second estimate, their confidence in it, and

how difficult it was to determine the person’s weight, participants answered an 11-item

Numeracy Scale (Schwartz, Woloshin, Black, & Welch, 1997). Higher scores (0 to 11) reflect a

greater comfort with numbers.

We calculated Weight on Advice (WOA) by dividing the difference between the final

and initial estimate produced by each participant by the difference between the advice and the

11
ALGORITHM APPRECIATION

initial estimate. This produces a measure of advice utilization that, in most cases, ranges from

0% (full advice discounting) to 100% (full advice adherence). Following prior research, we

Winsorized any WOA values greater than 1 or less than 0.

Table 1. Manipulation wording for the source of advice across experiments.

Experiment
Description and Algorithmic Advice Description Human Advice Description
Sample Size

1A: An algorithm ran calculations based on The average estimate of participants from a
Weight estimate estimates of participants from a past study. past experiment was: 163 pounds.
The output that the algorithm computed as
N = 202 an estimate was: 163 pounds.

1B: An algorithm estimated: X. The predicted song rank based on an

Song Rank aggregation of 275 other participants is: X.
Forecasts

N = 215

1C: An algorithm estimated this (wo)man's In another study, 48 people estimated this
Attractiveness attractiveness (humor/dinner date) from (wo)man's attractiveness (humor/dinner
Estimate Mike's (Julia’s) perspective. The date) from Mike's (Julia’s) perspective.
algorithm's estimate was: X Their estimate was: X
N = 286

1 D: Same as 1C Same as 1C
Researcher
Predictions
2: The output that an algorithm computed as The estimate of another participant was:
Joint vs. Separate an estimate was: 163 pounds. 163 pounds.
Evaluation

N = 154

3: Use only the statistical model’s estimated Self condition:

Self/Other Choice rank to determine my bonus. Use only my estimated rank to determine
my bonus.

N = 403 Other condition:

Use only the other participant’s estimated
rank to determine my bonus.
4: National Security The estimate from an algorithm is: X. Weight estimate:
Experts The estimate of another person was: 163
pounds.

N = 301; 70 Forecasts:
The average estimate from forecasters in
the forecasting tournament is: X.

12
ALGORITHM APPRECIATION

General Discussion: … an algorithm, based on estimates of 314 …a randomly chosen participant from a
Normative Standard participants who took a past study. pool of 314 participants who took a past
Benchmark study.

N = 671

Results

Participants showed an appreciation of algorithms, relying more on the same advice when

they thought it came from an algorithm (M = .45, SD = .37), than when they thought it came

from other people (M = .30, SD = .35), F (1, 200) = 8.86, p = .003, d = .42; see Figure 1. Results

hold when controlling for gender, numeracy, and confidence in the initial estimate, F (1, 197) =

9.02, p = .003. There were no main effects of these other variables (Fs < .39, ps > .52).

0.5
Weight on Advice (0-1)

0.4

0.3

0.2

0.1
Weight (1A) Songs (1B) Attractiveness (1C) Researcher
Predictions (1D)
Topic

Human Algorithm

Figure 1. Weight on Advice (WOA) as a function of condition, Experiments 1A, 1B, and 1C.
The higher the WOA, the more participants revised their estimate toward the advice they
received. For Experiment 1D, means represent researcher’s predicted WOA for participants in
Experiment 1C. Error bars indicate standard errors.

13
ALGORITHM APPRECIATION

Similarly, confidence increased more from Time 1 to Time 2 in the algorithmic advice

condition (Time 1: M = 71.30, SD = 18.11; Time 2: M = 79.91, SD = 16.93) than the human

advice condition (Time 1: M = 70.62, SD = 18.15; Time 2: M = 75.10, SD = 17.81), as reflected

in the main effect of time, F(1, 200) = 77.09, p = .001, and interaction between source and time,

F(1, 200) = 5.62, p = .019. Furthermore, higher numeracy correlated with greater reliance on

algorithmic advice, r(100) = .21, p = .037; this however, was not the case in human advice

condition, r = -.12, p = .225.

Discussion

Our results suggest that people display algorithm appreciation, even despite a minimal

description of the algorithm; participants relied more on identical advice when they thought it

came from an algorithm than when they thought it came from other people. Importantly, given

that the advice was nearly perfect, greater reliance on it improved judgment accuracy. Although

participants underweighed advice in both conditions (they adjusted less than half-way toward

advice that had been generated averaging the estimates of many people), they discounted

algorithmic advice less than they discounted advice from other people. These results suggest

that one way to increase adherence to advice is to provide advice from an algorithm.

Updating to algorithmic advice without access to its equations or processes reflects

people’s willingness to listen to algorithms even with uncertainty about its inner workings. Our

operationalization of a “black box” algorithm parallels the widespread appearance of algorithms

in daily life, including Netflix, weather forecasts, population estimates, and economic

projections. Presenting a black box algorithm allowed participants the use of their own default

interpretations of algorithmic judgment. This begs the question, what are people’s default

interpretations of an algorithm?

14
ALGORITHM APPRECIATION

Defining “Algorithm”

An important consideration in examining people’s reactions to algorithms is identifying

how participants define the construct. If individuals are uncertain about what an algorithm is,

then this uncertainty might account for some of the mistrust postulated by the algorithm aversion

literature. Or perhaps the typical research participant is certain but incorrect. We conceptualize

an algorithm as a series of mathematical calculations. Similarly, the field of mathematics defines

an algorithm as “a procedure for computing a function” (Rogers, 1987).

In order to establish whether most participants share this definition, we asked participants

in Experiment 2 (N = 149), as well as participants from an MBA sample at a large West Coast

university (N = 77) to define what an algorithm is. A research assistant coded participants’ open-

ended responses using thematic coding (Pratt, 2009). Specifically, the research assistant was

instructed to create as few categories as possible without making them too general (see Table 2).

Table 2. Participant-generated definitions of an algorithm.

Category of %
Definition Example Definition N=226

Math / Equation / “An algorithm is a set of equations to find an answer. 42%

Calculation It will spit out an answer.”

Step by Step “An algorithm is a systematic way of solving problems. It looks at a 26%
Procedure problem and goes through a process to figure out the solution.”

Logic / Formula “A formula that can be used to obtain a quantity or characteristic. 14%
An algorithm should be able to yield the same result for the same
input exactly and consistently.”

Computer “A series of formulas that will generate an output with the correct .09%
inputs. Usually used on the computer.”

Other “a way to solve a problem repetitively” .06%

Predictive Data “A model to predict a result based on incomplete data.” .04%

Analysis

15
ALGORITHM APPRECIATION

Overall, the categories based on participants’ responses provide evidence for 1) a high

level of consensus among participants regarding what an algorithm is as well as 2) agreement

with the expert definition. These data suggest that our participants are mostly familiar with

algorithms and conceptually understand what they do.

Method Experiment 1B (Forecasting Task)

We pre-registered collecting data from 200 participants. The final sample included 215

participants (114 women; 101 men; Mdn age = 31) as we over-estimated the number we would

exclude based on repeat I.P. addresses. We pre-screened participants who were 18-40 years old

to ensure that they had some familiarity with current music. In this and all subsequent

experiments, we pre-registered excluding participants associated with identical I.P. addresses

(accepting the first instance and excluding all additional instances; see Supplement for any other

pre-registered exclusions).

Participants predicted the rank of ten randomly selected songs on the Billboard

Magazine’s “Hot 100” which had placed on the chart in previous weeks (see Appendix Table 1).

For each song, participants saw a graph with that song’s ranks from prior weeks and made a

forecast by entering a number from 1 to 100. For example, one forecast asked:

“What rank will ‘Perfect’ by Ed Sheeran place on the Billboard Magazine ‘Hot 100’ this

week?”

They were asked to not search the internet for information. After making the first forecast, all

participants received identical advice and a chance to make a second, incentivized forecast.

Participants were randomly assigned to receive advice described as either aggregated from 275

other participants’ estimates or from an algorithm (see Table 1). We calculated Weight on

16
ALGORITHM APPRECIATION

Advice (WOA) as we did in Experiment 1A. Each exact correct final answer entered them into a

raffle for $10.

Results

Consistent with our pre-registered analysis plan, we treated the ten forecasts as repeated

measures. We regressed WOA on source of advice (- 1: Algorithm; +1: Human) using an OLS

regression and clustering observations at the participant level. We see again that participants

relied more on identical advice when they thought it came from an algorithm than from other

people, β = -.34, t(214) = 5.39, p < .001. Results hold analyzing an averaged WOA measure for

each participant (Algorithm: M = .37, SD = .02; Other People: M = .21, SD = .02), t(213) = 5.54,

p < .001; see Figure 1.3

Discussion

Study 1B speaks to the generalizability of algorithm appreciation by testing the

phenomenon using a forecasting task (rather than an estimation task) with a randomly selected

set of stimuli. Participants relied more on identical advice when they thought it came from an

algorithm than when they thought it came from other people, even in a highly familiar and taste-

based domain, such as predicting the popularity of songs. We examine algorithm appreciation in

Experiment 1C within an even more subjective domain: predicting romantic attraction.

Method: Experiment 1C (Romantic Attraction)

The final sample included 286 participants (157 women; 129 men; Mdn age = 37).

Participants were randomly assigned to read identical descriptions about a heterosexual woman,

Julia, or man, Mike. These descriptions only differed in their use of pronouns (see Supplement

through OSF link). Then participants saw one photograph of a different man (woman) and

3
For the figure, to create an appropriate comparison with Experiments 1A, 1C, and 1D, we use the means and
standard errors obtained by averaging all observations produced by one participant into a single data point.

17
ALGORITHM APPRECIATION

predicted how Julia (Mike) would evaluate the target using scales from 1 = not at all to 100 =

Extremely:

 How attractive do you think Julia (Mike) would find this (wo)man?

 How funny do you think Julia (Mike) would find this (wo)man’s sense of humor?

 How much do you think Julia (Mike) would enjoy a dinner date with this

(wo)man?

On the following page, participants received advice about each of the three forecasts and

made their final forecasts. Again, participants received identical advice but were randomly

assigned to read that the advice came from either 48 people from another study or an algorithm

(see Table 1). All advice was actually an average of forecasts from 48 past participants. These

participants evaluated the attractiveness, humor, and dinner enjoyment respectively for Julia’s

potential date as: 54, 59, and 58; and Mike’s potential date as: 64, 57, 66.

Results

As pre-registered, we calculated WOA for attractiveness, humor, and dinner forecasts and

averaged them into a single index (α= .77). Again, participants relied more on identical advice

when they thought it came from an algorithm (M = .38, SD = .28) than when they thought it

came from other people (M = .26, SD = .27), t(284) = 3.50, p = .001, d = .44. Algorithm

appreciation is robust to whether people made predictions for Mike or Julia, as evidenced by a

non-significant interaction between source of advice (algorithm vs. people) and target of

evaluation (Julia vs. Mike’s date), F(1, 282) = .05, p = .818. There was a main effect of source,

F(1, 282) = 13.48, p < .001, but not of target, F(1, 282) = 2.32, p = .129.4

4
When we enter gender of the participant as an additional factor, the effect of source remains significant, F(1, 278)
= 12.81, p < .001, and there are no other significant main effects nor interactions, ps > .093.

18
ALGORITHM APPRECIATION

Discussion

Even when predicting interpersonal attraction, a domain where individual preferences

(and even emotions) rule, participants relied more on identical advice when they thought it came

from an algorithm than when they thought it came from other people. Like Experiments 1A and

1B, Experiment 1C merely manipulated the label of the advice, ensuring that the algorithm was

described as simply as was the advice from people (“algorithm” vs. “48 people estimated”).

Method: Experiment 1D (Researcher’s Predictions)

In experiment 1D, we recruited academic researchers to predict the results of Experiment

1C to test whether they would expect mTurk participants to display algorithm aversion.

Participants, whom we will call “researchers,” followed a link circulated to the Society for

Judgment and Decision Making email distribution list and via personal requests. The final

sample was collected over a period of two weeks (as pre-registered), and included 119

participants (55 women; 64 men; Mdn age = 32). We excluded one participant from the 120

completed surveys who provided an answer outside of the range of 0 to 100%5 (not pre-

registered). Correct predictions entered participants into a raffle for a $100 bonus.

The researchers predicted the mTurk participants’ responses from Experiment 1C after

viewing those survey materials. Researchers saw materials from both the advisor conditions for

either Mike or Julia. Researchers read that the descriptions of Julia and Mike were identical but

for the names and pronouns. Researchers read how the WOA measure is calculated, how to

interpret the different values, and how participants generally respond to advice within the

paradigm based on prior research (see Supplement). Then, they predicted how mTurk

participants weighed the advice they received. These consisted of six incentivized predictions:

5
Not pre-registered.

19
ALGORITHM APPRECIATION

WOA for each of the attraction, humor, and dinner predictions for both the human and algorithm

advisors. Lastly, they answered questions about their own academic background.

Results

We averaged researchers’ WOA predictions across attractiveness, humor, and dinner into

a “predicted people WOA” index (α = .92) and a “predicted algorithm WOA” index (α = .91).

Then we subtracted the predicted algorithm WOA index from the predicted people WOA index.

Here, a positive value represents a prediction of algorithm aversion and a negative one represents

a prediction of algorithm appreciation.

Even though mTurk participants in Experiment 1C displayed algorithm appreciation (-

.11), the researchers predicted algorithm aversion (M = .14, SD = .20), one-sample t(118) =

14.03, p < .001, d = 1.25. These results not only reflect a magnitude difference between the

researches’ predictions and mTurk participants’ actual responses but also a directional

difference. Indeed, the algorithm appreciation exhibited by participants in experiment 1C was

significantly different from equal utilization of human and algorithmic advice, as was the

algorithm aversion predicted by researchers in experiment 1D, t(118) = 7.92, p < .001, d = .80;

see Figure 1.

Discussion

Although mTurk participants displayed algorithm appreciation when predicting romantic

attraction, researchers expected them to display algorithm aversion. The idea that people are

averse to algorithmic advice is evidently pervasive. Of the researchers in our sample, 34%

identified themselves as graduate students. Researchers, both junior and senior, do not give

people the credit they deserve in their willingness to rely on algorithmic advice (graduate

20
ALGORITHM APPRECIATION

students: M = .16, SD = .17; senior researchers: M = .14, SD = .21), t(117) = .52, p = .607.6 But

how robust is algorithm appreciation? Perhaps the way in which our experiments presented

advice led people to rely on algorithmic advice more than they might when choosing directly

between two advisors.

Experiment 2: Joint versus Separate Evaluation

Experiments 1A, 1B, and 1C demonstrated that in separate evaluation of advisors, our

between-subjects manipulation of the source affected advice utilization. Yet, prior work finding

algorithm aversion asked participants to choose between sources (joint evaluation). As attributes

are easier to evaluate in a joint evaluation, due to increased information (Bazerman,

Loewenstein, & White, 1992; Bazerman, Moore, Tenbrunsel, Wade-Benzoni, & Blount, 1999;

Hsee, 1996; Hsee, Loewenstein, Blount, & Bazerman, 1999), perhaps providing participants with

a counterfactual to the algorithmic advice decreases reliance on it. Experiment 2 examines

whether joint versus separate presentation of advisors determines whether people display

algorithm appreciation or algorithm aversion.

Method

Participants

We set out to collect data from 150 participants to detect a medium sized effect (Cohen’s

d = .4) at 80% power. The final sample included 154 participants (104 women; 50 men; Mdn

age = 21) from a West Coast U. S. university’s credit and paid subject pools.

Design

6Not pre-registered. We had pre-registered excluding graduate students, if we failed to find an effect when they
were included. However, after finding an effect, we wanted to compare the samples to make sure they did not
differ.

21
ALGORITHM APPRECIATION

The experiment followed the design of Experiment 1A but additionally included a third

condition where participants chose between an algorithmic and human advisor. Thus, it featured

a 3-cell (person (separate) vs. algorithm (separate) vs. choice between a person and algorithm

(joint) between-subjects design that manipulated the presentation of the advisors.

Procedure and Materials

After participants estimated the weight of the man in the same photograph as Experiment

1A, they learned that they were about to receive advice. In each of the separate conditions,

participants read simple descriptions of their advisors, either another participant, or an algorithm,

and then received the advice on the next page. In the joint condition, participants read the same

descriptions of both advisors: “you will choose whether you will see advice regarding the same

person’s weight from another study participant or from an algorithm” and chose their advisor.

On the next page, participants received advice either based on random assignment

(separate) or on their choice (joint) from another participant or an algorithm (see Table 1). We

again measured Weight on Advice (WOA). In the choice condition, we additionally measured

whether participants chose to receive advice from the person or algorithm, which was our main

variable of interest.

Results

As in our prior experiments, participants who evaluated advisors separately relied more

on the advice when they thought it came from an algorithm (M = .50, SD = .37) than from

another participant (M = .35, SD = .36), t(100) = 2.10, p = .038, d = .44; see Figure 2 in the

Appendix. However, evaluating the two advisors jointly did not reverse the preference for

22
ALGORITHM APPRECIATION

algorithmic advice; 75% of participants in the joint condition also preferred the algorithm (N =

39) over the other participant (N = 13).7

Discussion

Reliance on algorithmic advice appears robust to different presentations of advisors. The

results speak to the strength of algorithm appreciation. This durability might be impressive,

given that many decisions are affected by joint-versus-separate evaluation: willingness to pay for

consumer goods, willingness to pay for environmental issues, support for social issues, and voter

preferences (Hsee, 1996; 1998; Irwin, Slovic, Lichtenstein, & McClelland, 1993; Nowlis &

Simonson, 1997).

Thus far, our experiments have intentionally controlled for excessive certainty in one’s

own knowledge by providing advice from external advisors in both the human and algorithm

conditions. Doing so ensures that participants compare their own judgment with the advice in

both conditions. In Experiment 3, we examine whether such unwarranted confidence plays a

role in the use of algorithmic judgment.

Experiment 3: The Role of the Self

In the introduction, we alluded to the importance of distinguishing between the extent to

which individuals might disregard algorithmic advice from the extent to which they might

disregard advice in general. Addressing this question requires a comparison of how advice is

utilized both when it comes from human versus algorithmic sources, as well as when the human

judgment is produced by the participant him or herself. Such a design necessitates that we

abandon the WOA measure (because it does not measure advice to oneself), and instead rely on

7Not surprisingly, participants relied similarly on the advisor they chose, be it the algorithm (M = .52, SD = .37) or
person (M = .41, SD = .36), t(50) = .90, p = .371.

23
ALGORITHM APPRECIATION

each participant’s choice of human or algorithmic judgment when the human judgment either

comes from the self or another person.

We used the materials from Dietvorst et al.’s (2015) Experiment 3A, a study in which

people chose between their own estimate and an algorithm’s. The key comparison in the

Dietvorst et al. paper was between people who had no performance information regarding either

the human or the algorithm, versus people who saw the (imperfect) performance of the human,

the algorithm, or both. In our replication, we used the original control condition from Dietvorst

et al. (wherein people chose to use their own judgment versus an algorithm’s, with no

information about prior performance), and added a new condition in which participants chose

between another person’s answer and an algorithm’s. This method allowed us to manipulate the

source of human judgment: the participant’s own estimate versus another person’s.

Method

We aimed to collect a sample of 400 participants, following a power analysis based on

Dietvorst, et al. (2015). The final sample included 403 participants (177 women; 226 men; Mdn

age = 32). The more accurate participants’ answers, the greater their bonus payment (from $0.10

for an answer within 6 ranks of the truth, increasing in 15 cent increments for each closer rank, to

$1.00 for a correct answer). The experiment employed a 2-cell (self vs. other) between-subjects

design. Participants either chose between their own estimate (self condition) and an algorithm’s

estimate, or between another person’s estimate (other condition) and an algorithm’s (see Table

3). They made this choice prior to seeing the actual information that would inform the estimates

and prior to making any estimate of their own.

Table 3. Design of Experiment 3: Participants chose to base their bonus on a human or

algorithm estimate. The human was either the self or another participant.

24
ALGORITHM APPRECIATION

Self Condition Other Condition

Choose: Choose:
Algorithm’s Estimate Algorithm’s Estimate
vs. Own Estimate vs. Other Participant’s Estimate

Participants began the experiment by reading about the upcoming task: estimating the

rank of one U.S. state from 1 to 50 in terms of the number of airline passengers who departed

from the state in 2011. A rank of 1 indicated the most departing passengers, and a rank of 50

indicated the fewest departing passengers. Then, they read an overview of the information which

all judges would receive in order to make the estimate, including state-specific background

information such as number of major airports, median household income for 2008, etc. (see

Supplement).

Participants either read that in addition to background information about the state, they

could receive an estimate from an algorithm (self condition) or estimates from an algorithm or

another participant (other condition). All participants read a description of the algorithm, as

provided by Dietvorst et al (2015):

“…statistical model developed by experienced transportation analysts. …The

model does not have any additional information that you will not receive. This is

a sophisticated model, put together by thoughtful analysts.”

Prior to making their own estimate, participants chose how they wanted to determine

their bonus pay, which was based on accuracy. Specifically, participants in both conditions

chose whether they wanted their bonus to be determined by an estimate produced by a human

versus an algorithm. Because the conditions differed in whether the human was the participant

him or herself or another participant, this design allowed us to examine the influence of the self

25
ALGORITHM APPRECIATION

in adherence to algorithmic advice. Finally, participants reported their confidence in both human

and algorithmic estimates prior to making their own estimate or seeing what the advisor(s)

estimated.

Results

Overall, people preferred to base their bonus pay on algorithmic judgment rather than

human judgment. The majority of participants (88%) chose to determine their bonus pay based

on the algorithm’s estimate rather than another participant’s estimate, χ2 (1, N = 206) = 118.14,

p < .001, r = .76, consistent with results from Experiments 1A, 1B, 1C, 2, and 3. Similarly,

participants even chose the algorithm’s estimate over their own estimate (66%), χ2 (1, N = 197)

= 20.15, p < .001, r = .32. Importantly, the preference for algorithmic judgment was attenuated

by the introduction of the self; participants chose the algorithm less frequently when they could

choose their own estimate (rather than another participant’s), z = 6.62, p < .001; see Figure 2.

Other

Algorithm
Person

Self

0 20 40 60 80 100
Percent Choosing Algorithm

Figure 2. Percent of participants choosing to base their bonus on the estimate of the algorithm by
condition. More than 50% of participants chose the algorithm in both conditions, ps < .001.

26
ALGORITHM APPRECIATION

When we examined participants’ confidence in the advisors’ estimates, the pattern of

results is consistent with both our predictions and the overconfidence literature.8 Although

participants in both conditions were more confident in the accuracy of the algorithmic estimate

(M = 3.76, SD = .75) than human estimate (M = 2.69, SD = .83; t(402) = 21.28, p < .001),

participants were more confident in their own estimate (M = 2.86, SD = .91) than that of a fellow

participant (M = 2.53, SD = .71), t(369.93) = 4.03, p < .001, correcting for unequal variances.

Individuals’ confidence in the algorithmic advice remained consistent across conditions

(confidence in algorithmic estimate when contrasted against another participants’ judgment: M

= 3.80, SD = .72; confidence in algorithm when contrasted against participants’ own judgment:

M = 3.72, SD = .79; t(393.18) = 1.07, p = .286, correcting for unequal variances).

Discussion

Study 3 demonstrates a reduction in algorithm appreciation when individuals are forced

to choose between their own judgment and that of an algorithm. These results suggest that prior

findings of algorithm aversion may have been boosted by people’s excessive appreciation of

their own opinions rather than comparing two types of advisors. Our results are also consistent

with the robust literature on overconfidence which has repeatedly demonstrated that individuals

treat their judgment as superior to that of other people (Harvey, 1997). Although we also find

that individuals were more confident in their own judgment than that of another participant, they

appropriately judged the advice of a sophisticated algorithm as superior to both their own

judgment and that of another person.

Our results may appear to contradict the results from Dietvorst et al. (2015), where

participants’ preference for algorithmic judgment decreased when they saw the algorithm err.

8
Not pre-registered.

27
ALGORITHM APPRECIATION

However, a closer examination of the Dietvorst results reveals that prior to receiving

performance data, participants in those studies either preferred algorithmic judgment to human

judgment (as our participants do) or were indifferent between the two. Our current work focuses

on people’s perceptions of algorithmic advice prior to receiving any performance accuracy

feedback, and is thus consistent with the prior results.

The world outside of the laboratory is rife with situations in which we must decide

whether to rely on algorithmic judgment in the absence of performance data. Forecasts in

financial, geopolitical and environmental domains are often made years before accuracy data

become available. And when it comes to selecting employees or dating partners, our choices

rule out the possibility of observing the counterfactual of what would have happened had we

made another choice. The richness of decision-making scenarios in the real world makes it

doubly important to document use of advice across a variety of contexts.

Experiment 4: Decision-Maker Expertise

Our experiments thus far focused on how elements of the judgment context influence

how people respond to algorithmic versus human advice. Experiment 4 examined whether the

expertise of the decision-maker influences reactions to algorithmic versus human advice. We

recruited professionals whose work in the field of national security for the U.S. government

made them experts in geopolitical forecasting and compared their advice utilization with the

diverse online sample available through Amazon’s Mechanical Turk. We provided both samples

with the same forecasting tasks, and with identical advice. The advice came from forecasters in

a follow-up tournament to the Good Judgment Project (GJP). The GJP was a government funded

initiative aimed at improving forecasts of geopolitical events through statistical aggregation of

individual judgments (Mellers et al., 2014).

28
ALGORITHM APPRECIATION

Method

In selecting the sample size, we anticipated uncertainty in the number of experts who

might take the survey, and thus aimed to collect a sample size of 200 mTurk workers and 75

experts, with the goal of collecting 100 experts if possible. We stopped data collection from

experts when multiple days passed without new participants in order to ensure that all forecast

were made in a comparable time period. We recruited more mTurk participants than we pre-

registered because we over-estimated the number we would exclude based on repeat I.P.

addresses. The final sample included 301 mTurk participants (women = 154; men = 147; M age

= 39) and 70 U.S. national security professionals (women = 3; men = 67; M age = 46) for a total

of 371 participants. We recruited national security professionals with the help of a U.S.

government employee who worked in the field and shared the survey with email distribution lists

dedicated to the topic of national security. Most respondents were U.S. government employees

or contractors working for the government. We incentivized mTurk participants by entering

them into a raffle for $10. Participants who made more accurate forecasts received more entries

into the raffle. The national security experts were entered into a raffle for an iPad.

The experiment had a 2 (advisor: human vs. algorithm) X 2 (sample: lay vs. expert)

design. This experiment used the same main dependent variable, Weight on Advice (WOA), for

four tasks: a weight estimate, a business forecast, and two geopolitical forecasts. We included

additional measures to better understand the experience of the expert sample.

We chose the tasks to amplify the difference in expertise between samples. We expected

experts to feel greater expertise for the geopolitical forecasts than the lay sample. We expected

both samples to feel similarly low expertise for the weight estimate and business forecasts. The

weight estimate was identical to Experiment 1A. The forecasts were:

29
ALGORITHM APPRECIATION

 “What is the probability that Tesla Motors will deliver more than 80,000 battery-

powered electric vehicles (BEVs) to customers in the calendar year 2016?”

 “What is the probability that a North American country, the EU, or an EU

member state will impose sanctions on another country in response to a cyber

attack or cyber espionage before the end of 2016?”

 “What is the probability that the United Kingdom will invoke Article 50 of the

Lisbon Treaty before July 1, 2017?”

As in Experiment 1A, participants made two judgments. Prior to their second estimate, all

participants received human or algorithmic advice (see Table 1). For the forecasts, we provided

participants with information about the forecasting tournament:

The Good Judgment Open is a forecasting (prediction) tournament, hosted by

academic researchers, where thousands of people around the world compete to

make the most accurate forecasts (predictions) about global political events.

We provided high quality advice to all participants. The target in the photo weighed 164

pounds and the advice was 163. Tesla did not sell more than 80,000 battery-powered electric

vehicles (BEVs) in 2016 (it sold 76,230) and the advice was that the event was 20% likely to

occur. The cyber and Brexit events were more difficult events to predict. The cyber event

occurred just before the end of 2016, on December 29, 2016 and the advice was 12% likelihood

of occurrence. The Brexit event did occur on March 29, 2017 and the advice was 54%

likelihood of occurrence.

Following prior research (Mellers et al., 2014), we used Brier scores (Brier, 1950) to

assess forecasting accuracy. A Brier score is a measure of error that ranges from 0 to 2, with

higher numbers reflecting greater error. To calculate Brier scores, we squared the distance

30
ALGORITHM APPRECIATION

between the estimated probability and the actual result. For example, if a participant estimated a

.8 probability of an event occurring (and thus a .2 probability of it not occurring), and the event

occurred, we calculated: (.8 - 1)2 + (.2 - 0)2 = .08.

Participants also reported how frequently they made forecasts for their occupation and

their familiarity with the word algorithm:

 “For your job, how often do you make forecasts (predictions)?” on a scale from 1

= Barely ever to 7 = Multiple times a day.

 “How certain are you that you know what an algorithm is?” on a scale from 0 =

NA, I am certain that I do NOT know what it means; 1 = not at all certain to 7 =

extremely certain.

Results

The expert sample (national security experts) reported making forecasts for their jobs

more frequently (M = 3.73, SD = 2.30) than the lay sample (M = 2.29, SD = 2.02), t(95.30) =

4.84, p < .001, correcting for unequal variances. Participants overall said they were familiar with

algorithms (M = 5.04, SD = 1.72), but this did not significantly differ by expertise (lay: M = 4.96,

SD = 1.73, expert: M = 5.40, SD = 1.63), t(369) = -1.96, p = .051.

Consistent with our pre-registered analysis plan, we submitted WOA to a 2 (advisor:

algorithm vs. human) X 2 (expertise: expert vs. non-expert) repeated measures ANCOVA with

the four tasks as repeated measures and familiarity with algorithms as a covariate. This analysis

included those who answered all tasks (282 lay people and 61 experts). There is an effect of

familiarity, F(1, 338) = 8.86, p = .003, η² = 0.03, d = 0.29, such that participants who claimed

greater familiarity with algorithms took less advice, collapsing across advisors.

31
ALGORITHM APPRECIATION

Controlling for familiarity, we observed a main effect of advisor, F(1, 338) = 9.46, p =

.002, η² = 0.02, d = 0.29. As in earlier experiments, our participants placed more weight on

algorithmic than human advice. Furthermore, experienced judges (the national security experts)

took less advice than lay people, F (1, 338) = 32.39, p < .001, η² = 0.08, d = 0.60. Importantly,

we also observed a significant interaction between judge expertise and the source of advice:

whereas lay judges placed more weight on algorithmic than human advice, the experts heavily

discounted all advice sources, F (1, 338) = 5.05, p = .025, η² = 0.01, d = 0.23; see Figures 3 and

4.

0.7 Lay Sample

0.6
Weight on Advice (0-1)

0.5
0.4
0.3
0.2
0.1
0
Weight Tesla Cyber Brexit
Task

Human Algorithm

Figure 3. Weight on Advice (WOA) as a function of advisor for the lay sample.

32
ALGORITHM APPRECIATION

0.7 Expert (National Security) Sample

0.6
Weight on Advice (0-1)

0.5
0.4
0.3
0.2
0.1
0
Weight Tesla Cyber Brexit
Task

Human Algorithm

Figure 4. Weight on Advice (WOA) as a function of advisor for the expert sample.

Forecast Accuracy

The geopolitical and business forecasts were difficult events to predict. The Tesla event

did not occur, as Tesla sold merely 3,770 cars less than the number in the forecast question.

Thus, the advice, a low probability of the event occurring (20%), was very accurate (Brier score:

.08). The lower accuracy of the cyber forecast advice reflects its difficulty: the event occurred

only days before the deadline in the forecast question, and because the advice was a low

probability (12%) of the event occurring, the advice was not very accurate (Brier score: 1.55).

The advice was ambiguous for the Brexit forecast (54%) but because the event actually did

occur, the accuracy of the advice was fairly accurate (Brier: .42).

We submitted Brier scores to the same 2 X 2 repeated measures ANCOVA as WOA with

three forecasts as repeated measures and familiarity with algorithms as a covariate. There is an

effect of familiarity, F(1, 366) = 5.71, p = .017. Controlling for familiarity, there is no main

effect of advisor, F(1, 366) = .09, p = .764, but there is an effect of expertise, F (1, 366) = 9.53, p

33
ALGORITHM APPRECIATION

= .002, and an interaction, F (1, 366) = 4.16, p = .042; see Figure 5. Lay people and experts

achieved similar accuracy when they received advice from a person. Importantly, lay people

achieved greater accuracy than experts when they received advice from an algorithm because

experts heavily discounted algorithmic advice.

Lower Scores = Greater Accuracy

0.8
Brier Score (0 - 2)

0.4

0
Human Algorithm
Task

Lay Sample Expert Sample

Figure 5. Brier score as a function of advisor (another person / forecaster vs. algorithm) and
sample (lay vs. expert), collapsed across forecasts, Experiment 4. The lower the Brier Score, the
more accurate the forecast. A Brier Score of 0 means that the forecast was perfectly accurate.
Note: p < .001, in the algorithm conditions between samples. Interaction: F (1, 366) = 4.16, p =
.042.
Discussion

Unlike participants in our earlier experiments, the experts in Experiment 4 did not

recognize the value of algorithmic advice. Their adherence to their prior judgments and their

failure to utilize the information offered to them ultimately lowered their accuracy. Furthermore,

the experts weighed advice less than the 30% WOA observed in advice-taking research (see

Figure 4). These results are also consistent with prior research demonstrating that expert

attorneys are less likely than law students to give weight to advice in a verdict estimation task

(Jacobson, Dobbs-Marsh, Liberman & Minson, 2011). These results might help explain why

34
ALGORITHM APPRECIATION

pilots, doctors, and other experts are resistant to algorithmic advice (Meehl, 1954). Although

providing advice from algorithms may increase adherence to advice for non-experts, it seems

that algorithmic advice falls on deaf expert ears, with a cost to their accuracy.

General Discussion

Counter to the widespread conclusion that people distrust algorithms, our results suggest

that people readily rely on algorithmic advice. Our participants relied on advice from algorithms

over advice from other people whether they were predicting geopolitical events, the popularity of

songs, or romantic matches. They weighed algorithmic advice more heavily than human advice

and chose algorithmic judgment over human judgment when given the choice. They even

showed a willingness to choose algorithmic advice over their own judgment. Algorithm

appreciation proved robust to a variety of elicitation methods. It also held across a variety of

operationalizations and descriptions of algorithms (see Table 1). However, appreciation of

algorithms was not uniform. It was lower among less numerate participants and also among

experts, who were simply less open to taking any advice. The accuracy of their judgments

suffered as a result. It was also lower when algorithmic advice was pitted against the

participants’ own judgment. These results shed light on when people are most likely to improve

their accuracy by listening to algorithms and has implications for the use of algorithms by

decision makers within organizations.

Theoretical Implications

Our results suggest that one simple way to increase adherence to advice is to provide

advice from an algorithm. Many decisions are ripe for the use of algorithmic advising,

considering the low cost and widespread availability of algorithmic advice relative to expert

advice. For instance, a number of applications are now available to provide financial advice and

35
ALGORITHM APPRECIATION

investing guidance (Common Cents Lab, 2017). These sorts of “Robo-advisors” sometimes

begin serving those who find it too costly to access human advisors. However, given the

excellent performance record of algorithmic advice, there are many instances in which it can be

both cheaper and better.

Our results challenge the widespread assertion that people are averse to algorithms

(Bazerman, 1985; Dawes, 1979; Dawes, Faust, & Meehl, 1989; Kleinmuntz, 1990; Kleinmuntz

& Schkade, 1993; Meehl, 1954; Meehl, 1957). They suggest that the story of algorithm aversion

is not as straightforward as the literature might otherwise lead us to believe. Importantly, our

results are consistent with those of Dietvorst, et al. (2015) as well as Prahl and Van Swol (2017).

In those studies as well as in ours, participants were quite willing to rely on algorithmic advice

before seeing the algorithm err. Although it is important to understand how people react to

performance of human and algorithmic advisors, many consequential decisions are made without

the benefit of performance feedback. It is therefore useful to examine responses to algorithmic

advice prior to feedback, as we do in this paper.

Our experiments connect the research streams on perceptions of algorithmic judgment

and advice taking in quantitative judgments by employing the Weight on Advice (WOA)

measure. This continuous measure of advice utilization is more sensitive than the categorical

measure of choice utilized in much of the prior research on human responses to algorithmic

advice. In order to further compare our results with the literature on advice taking, we conducted

another study where participants received information about how the human and algorithmic

advice was produced (see Supplement for more details). Specifically, we told participants that

the human advice came from one person, randomly chosen from a pool of 314 people, whereas

the algorithmic advice was based on estimates from 314 people (see Table 1). Presenting

36
ALGORITHM APPRECIATION

participants with the number of judgments that produced the advice provided the opportunity to

measure how effectively people use algorithmic advice by comparing WOA to a normative

benchmark. Here, we also allowed the accuracy of human and algorithmic advice to vary, as it

often does in the real world.

A sizable literature shows that to maximize accuracy, people should simply average their

own guesses with that of another person, yielding a WOA of 50% (Galton, 1907; Soll & Larrick,

2009; Suroweicki, 2004). However, WOA is usually too low; that is, people discount advice

from others (Bonaccio & Dalal, 2006; Yaniv, 2004; Yaniv & Kleinberger, 2000). As in our prior

experiments, participants relied more on advice from the algorithm (MWOA = .34, SD = .34) than

another person (MWOA= .24, SD = .27), F(1, 669) = 17.68, p < .001, d = .33; see Figure 2 in the

Appendix.

Yet, we also replicate prior work, such that participants underweighted advice from the

person (M = .26, SD = .27) relative to how much they should weight advice from one other

person (WOA = .5). By contrast, participants who received advice from an algorithm that

averaged advice from 314 other people should have abandoned their own guess and weighted the

algorithm’s advice 100%. They failed to do so (M = .66, SD = .34), thereby underweighting

algorithmic advice to an even greater extent than they underweighted human advice, F(1, 669) =

275.08, p < .001, d = 1.30. Even though our participants weighed algorithmic advice more

heavily than advice from other people, they did not weigh it heavily enough. This result suggests

that although people display algorithm appreciation, there is still room for them to improve their

accuracy.

Practical Implications

37
ALGORITHM APPRECIATION

Understanding how people respond to algorithmic advice holds implications for any

decision maker or organization with the potential to learn from lessons produced by “big data.”

As organizations invest in the collection, analysis and exploitation of ever larger quantities of

data, they use algorithms to sift through the information and produce advice. Advances in

technology have improved the speed and efficiency of this process (Laney, 2012). Furthermore,

many organizations have begun to make their data publically available (for example, Google (as

used in Reips & Matzat, 2014) and Twitter (as used in Reips & Garaizar, 2011)). Algorithms

rely on data to hone the accuracy of their advice, so continued collection of data increases the

potential value of algorithmic advice in domains as diverse as movie recommendations and

medical diagnosis.

Future Research: Theory of Machine

As humans interact more frequently with programmed agents in their workplaces, homes,

and cars, we need to understand their theory of machine. Theory of machine, like theory of

mind, requires people to consider the internal processes of another agent. Philosophical work on

theory of mind considers how people infer other people’s intentions and beliefs (Dennett, 1987).

This paper tested people’s responses to algorithmic output, and more work on theory of machine

is needed to examine people’s lay perceptions of how algorithmic and human judgment differ in

their input, process, and output.

Social psychology has taught us much about how people think about others’ minds. For

instance, people differentiate others’ behavior based on whether they perceive the behavior as

intentional or unintentional (Malle & Knobe, 1997). People also use cues about experience for

mind perception (Gray, Gray, & Wegner, 2007). The fundamental attribution error ascribes

more credit to dispositional attributes (Lassiter, Geers, Munhall, Ploutz-Snyder, &

38
ALGORITHM APPRECIATION

Breitenbecher, 2002) and attitudes (Ajzen, Dalto, & Blyth, 1979; Jones & Harris, 1967), than the

situation warrants. Anthropomorphizing machines, like the self-driving car, influences our trust

in them (Waytz, Heafner, & Epley, 2014) and can backfire (Gray & Wegner, 2012). Rather than

testing how people impart human judgment on algorithms, this paper tests lay beliefs about how

algorithmic and human advice differ.

Often, algorithmic advice is produced by “black box” algorithms, where the user is not

privy to its inner workings. What matters most in these situations, and even when people do

have some information about the algorithm, is people’s lay theories about what kind of

information the algorithm uses as inputs, how the information is processed, and the value of the

output. Understanding how people expect algorithms and humans to differ might facilitate

examination of how people respond to algorithmic compared with human advisors.

Algorithmic Process. We find consistent evidence that participants who faced “black

box” algorithms in the experiments we present were willing to rely on that advice despite its

mysterious origins. We are comforted that the participant definitions of algorithms were similar

to our conceptualization. Nevertheless, it is worth asking how appreciation of algorithmic advice

is affected by transparency of the algorithm’s operations as well as people’s more sophisticated

understanding of them.

Given the excellent track record of so many algorithms, it might stand to reason that more

knowledge would increase algorithm appreciation. On the other hand, as algorithmic operations

become more complex, they also become inscrutable. Especially for the innumerate, learning

about linear models that minimize mean squared error without overfitting historical data might

raise more concerns and doubts than it assuages. It is easy to imagine that some information

could actually reduce users’ appreciation of the algorithm.

39
ALGORITHM APPRECIATION

Expertise Process. The experiments we present compare participants’ reliance on advice

from algorithms with advice from other people similar to them. That comparison is useful

because it allowed us to make the contest between algorithms and people a fair fight, and

allowed us to control for a number of potential differences between different sources of advice.

However, it is possible that people prefer advice from human experts than from others like them

(see Supplement), and that this preference is stronger than any appreciation of algorithms

(Önkal, Goodwin, Thomson, Gönül, & Pollock, 2009; Promberger & Baron, 2006). Future work

will have to investigate how people’s beliefs about others’ expertise affects reliance on their

advice. Any such research program will need to address the question of what exactly people

believe about expertise and how it improves the value of advice (a theory of expertise). The

operations of expert minds will remain an inscrutable “black box” to those who receive advice

from them. Thus, it may prove useful to compare a theory of expertise with a theory of machine.

Room for Increased Reliance on Algorithms. Our results provide a more nuanced

picture of individuals’ willingness to take algorithmic advice. On the one hand, we find that

participants adjusted their judgments more in light of algorithmic versus human input. On the

other hand, and consistent with prior work on advice taking, individuals adjusted their judgments

too little, and seemed largely insensitive to the quality of advice that was offered to them. While

an influential paper shows that people are more willing to use an imperfect algorithm if they are

able to adjust its output (Dietvorst, Simmons, & Massey, 2006), future work could also examine

how algorithmic advice can be presented in a way that maximizes its uptake prior to any

performance information.

Conclusion

40
ALGORITHM APPRECIATION

Technological advances make it possible for us to collect and utilize data like never

before. “Big data” is changing the way organizations function and communicate, both internally

and externally. Organizations use algorithms to hire and fire people (Ritchel, 2013), manage

employees’ priorities (Copeland & Hope, 2016), and help employees choose health care plans

(Silverman, 2015). Companies such as Stitch Fix provide clients with clothing recommendations

based on algorithms (Ahuja, 2015; Hu, 2014).

Information technologies may increase the prevalence of “big data” initiatives but how do

these initiatives and the algorithmic advice gleaned from them change how people see the world?

Organizations have an opportunity to learn from the ever-increasing amount of information they

can access. Yet, if they only focus on collecting and analyzing data and overlook how people

respond to the algorithmic advice produced by it, they will not fully capture that opportunity to

learn.

Without understanding how people incorporate information from algorithms into their

decisions, organizations run the risk of misusing the opportunities presented by technological

advances. On the one hand, our results suggest optimism about the potential for usage of

algorithmic advice. Indeed, we are deeply hopeful about the potential for humans to use

algorithmic advice to help them make wiser decisions. At the same time, our results raise

questions about the willing reliance on technological guidance. In an age when we routinely

allow our attention and our spending to be directed by technological systems, when we rarely

bother to understand those systems, let alone read the terms and conditions of our usage, we all

ought to be concerned about our vulnerability to manipulation. We hope our work stimulates

future research to explore the consequences of the human willingness to rely on algorithms.

Doing so can help decision makers increase their awareness of being potentially manipulated and

41
ALGORITHM APPRECIATION

also help them better extract the knowledge that is available in order to thrive in the world of

“big data.”

42
ALGORITHM APPRECIATION

References

Ahuja, S. (2015, May 26). What Stitch Fix figured out about mass customization. Harvard

Business Review. [Link]

Ajzen, I., Dalto, C. A., & Blyth, D. P. (1979). Consistency and bias in the attribution of attitudes.

Journal of Personality and Social Psychology, 37(10), 1871.

Bazerman, M. H. (1985). Norms of distributive justice in interest arbitration. Industrial and

Labor Relations Review, 38, 558-570. doi:10.2307/2523991

Bazerman, M. H., Loewenstein, G. F., & White, S. B. (1992). Reversals of preference in

allocation decisions: Judging an alternative versus choosing among alternatives.

Administrative Science Quarterly, 220-240.

Bazerman, M. H., Moore, D. A., Tenbrunsel, A. E., Wade-Benzoni, K. A., & Blount, S. (1999).

Explaining how preferences change across joint versus separate evaluation. Journal of

Economic Behavior and Organization, 39, 41–58.

Bonaccio, S., & Dalal, R. S. (2006). Advice taking and decision-making: An integrative

literature review, and implications for the organizational sciences. Organizational

Behavior and Human Decision Processes, 101(2), 127-151.

Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather

Review, 78, 1–3.

Common Cents Lab (2017, August 18). Retrieved from [Link]

[Link]/commoncents-lab/

Copeland, R., Hope, B. (2016, December 22) The World’s Largest Hedge Fund Is Building an

Algorithmic Model From its Employees’ Brains. The Wall Street Journal, Retrieved from

[Link]

43
ALGORITHM APPRECIATION

Dana, J., & Thomas, R. (2006). In defense of clinical judgment...and mechanical prediction.

Journal of Behavioral Decision Making, 19(5), 413–428.

Dawes, R. M. (1979). The robust beauty of improper linear models in decision making.

American psychologist, 34(7), 571. doi:10.1037/0003-066x.34.7.571

Dawes, R. M., & Corrigan, B. (1974). Linear models in decision making. Psychological

bulletin, 81(2), 95.

Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science,

243(4899), 1668-1674. doi:10.1126/science.2648573

Dennett, D. (1987). The Intentional Stance. Cambridge: MIT Press.

Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously

avoid algorithms after seeing them err. Journal of Experimental Psychology:

General, 144(1), 114.

Dietvorst, B. J., Simmons, J. P., & Massey, C. (2016). Overcoming algorithm aversion: People

will use imperfect algorithms if they can (even slightly) modify them. Management

Science.

Dijkstra, J. J. (1999). User agreement with incorrect expert system advice. Behaviour &

Information Technology, 18(6), 399-411. doi:10.1080/014492999118832

Dijkstra, J. J., Liebrand, W. B., & Timminga, E. (1998). Persuasiveness of expert systems.

Behaviour & Information Technology, 17(3), 155-163. doi:10.1080/014492998119526

Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (2002). The perceived utility of

human and automated aids in a visual detection task. Human Factors: The Journal of the

Human Factors and Ergonomics Society, 44(1), 79-94. doi:10.1518/0018720024494856

44
ALGORITHM APPRECIATION

Einhorn, H. J., & Hogarth, R. M. (1975). Unit weighting schemes for decision

making. Organizational behavior and human performance, 13(2), 171-192.

Galton, F. (1907). Vox populi. Nature, 75, 450–451.

Gino, F. (2008). Do we listen to advice just because we paid for it? The impact of advice cost on

its use. Organizational behavior and human decision processes, 107(2), 234-245.

Gino, F., & Moore, D. A. (2007). Effects of task difficulty on use of advice. Journal of

Behavioral Decision Making, 20(1), 21-35. doi:10.1002/bdm.539

Gray, H. M., Gray, K., & Wegner, D. M. (2007). Dimensions of mind

perception. Science, 315(5812), 619-619.

Gray, K., & Wegner, D. M. (2012). Feeling robots and human zombies: Mind perception and the

uncanny valley. Cognition, 125(1), 125-130.

Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus

mechanical prediction: a meta-analysis. Psychological Assessment, 12(1), 19.

Haak, T. (2017). Algorithm aversion (HR trends, 2017). Retrieved March 22, 2017, from

[Link]

Harrell, E. (2016, September). Managers shouldn’t fear algorithm-based decision making.

Harvard Business Review.

Harvey, N. (1997). Confidence in judgment. Trends in Cognitive Sciences, 1(2), 78–82.

Harvey, N., & Fischer, I. (1997). Taking advice: Accepting help, improving judgment, and

sharing responsibility. Organizational Behavior and Human Decision Processes, 70, 117-

133.

Hastie, R., & Dawes, R. M. (2001). Rational choice in an uncertain world: The psychology of

judgment and decision making. Thousand Oaks, Calif.: Sage.

45
ALGORITHM APPRECIATION

Hu, E. (2014, September 8). Try This On For Size: Personal Styling That Comes In The Mail

[Audio file]. National Public Radio: All Tech Considered. [Link]

Hsee, C. K. (1996). The evaluability hypothesis: An explanation for preference reversals

between joint and separate evaluations of alternatives. Organizational behavior and

human decision processes, 67(3).

Hsee, C. K., Loewenstein, G. F., Blount, S., & Bazerman, M. H. (1999). Preference reversals

between joint and separate evaluations of options: a review and theoretical analysis.

Psychological Bulletin, 125(5), 576.

Irwin, J. R., Slovic, P., Lichtenstein, S., & McClelland, G. H. (1993). Preference reversals and

the measurement of environmental values. Journal of Risk and Uncertainty, 6(1), 5-18

Jacobson, J., Dobbs‐Marsh, J., Liberman, V., & Minson, J. A. (2011). Predicting Civil Jury

Verdicts: How Attorneys Use (and Misuse) a Second Opinion. Journal of Empirical

Legal Studies, 8(s1), 99-119.

Jones, E. E.; Harris, V. A. (1967). "The attribution of attitudes. Journal of Experimental Social

Psychology, 3(1): 1–24. doi:10.1016/0022-1031(67)90034-0

Kahneman, D. (2013). Thinking, fast and slow (1st pbk. ed.). New York: Farrar, Straus and

Giroux.

Kleinmuntz, B. (1990). Why we still use our heads instead of formulas: toward an integrative

approach. Psychological Bulletin, 107(3), 296-310. doi:10.1037/0033-2909.107.3.296

Kleinmuntz, D. N., & Schkade, D. A. (1993). Information displays and decision processes.

Psychological Science, 4(4), 221-227. doi:10.1111/j.1467-9280.1993.tb00265.x

Laney, D. (2012). The Importance of' Big Data': A Definition. Gartner.

46
ALGORITHM APPRECIATION

Lassiter, G. D., Geers, A. L., Munhall, P. J., Ploutz-Snyder, R. J., & Breitenbecher, D. L. (2002).

Illusory causation: Why it occurs. Psychological Science, 13(4), 299-305.

Liberman, V., Minson, J. A., Bryan, C. J., & Ross, L. (2012). Naïve realism and capturing the

“wisdom of dyads”. Journal of Experimental Social Psychology, 48(2), 507-512.

Logg, J. M, Haran, U. & Moore, D. A., Is Overconfidence a Motivated Bias? Experimental

Evidence (2018). Harvard Business School Working Paper No. 18-099.

Malle, B. F., & Knobe, J. (1997). The folk concept of intentionality. Journal of Experimental

Social Psychology, 33(2), 101-121.

Mannes, A. E. (2009). Are we wise about the wisdom of crowds? The use of group judgments in

belief revision. Management Science, 55(8), 1267-1279.

Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of

the evidence. Minneapolis, MN: University of Minnesota Press.

Meehl, P. E. (1957). When shall we use our heads instead of the formula? Journal of Counseling

Psychology, 4(4), 268-273. doi:10.1037/h0047554

Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D.,

Atanasov, P., Swift, S. A., Murray, T., Stone, E., & Tetlock, P. E. (2014). Psychological

strategies for winning a geopolitical forecasting tournament. Psychological science,

25(5), 1106-1115. DOI: 10.1177/0956797614524255

Minson, J. A., Liberman, V., & Ross, L. (2011). Two to tango: Effects of collaboration and

disagreement on dyadic judgment. Personality and Social Psychology Bulletin, 37(10),

1325-1338.

Minson, J. A., & Mueller, J. S. (2012). The cost of collaboration: Why joint decision making

exacerbates rejection of outside information. Psychological Science, 23(3), 219-224.

47
ALGORITHM APPRECIATION

Moore, D. A., & Healy, P. J. (2008). The trouble with overconfidence. Psychological review,

115(2), 502.

Moore, D. A., & Klein, W. M. (2008). Use of absolute and comparative performance feedback in

absolute and comparative judgments and decisions. Organizational Behavior and Human

Decision Processes, 107(1), 60-74.

Moore, D. A., Tenney, E. R., & Haran, U. (2015). Overprecision in judgment. The Wiley

Blackwell handbook of judgment and decision making, 182-209.

Nowlis, S. M., & Simonson, I. (1997). Attribute-task compatibility as a determinant of consumer

preference reversals. Journal of marketing research, 205-218.

Önkal, D., Goodwin, P., Thomson, M., Gönül, S., & Pollock, A. (2009). The relative influence of

advice from human experts and statistical methods on forecast adjustments. Journal of

Behavioral Decision Making, 22(4), 390-409.

Petropoulos, F., Fildes, R., & Goodwin, P. (2016). Do “big losses” in judgmental adjustments to

statistical forecasts affect experts’ behaviour? European Journal of Operational

Research, 249(3), 842–852.

Prahl, A., & Van Swol, L. (2017). Understanding algorithm aversion: When is advice from

automation discounted?. Journal of Forecasting, 36(6), 691-702.

Pratt, M. G. (2009). From the editors: For the lack of a boilerplate: Tips on writing up (and

reviewing) qualitative research. Academy of Management Journal, 52(5), 856-862.

Promberger, M., & Baron, J. (2006). Do patients trust computers? Journal of Behavioral

Decision Making, 19, 455–468.

Reips, U. D., & Garaizar, P. (2011). Mining twitter: A source for psychological wisdom of the

crowds. Behavior research methods, 43(3), 635-642. doi:10.3758/s13428-011-0116-6

48
ALGORITHM APPRECIATION

Reips, U. D., & Matzat, U. (2014). Mining “Big Data” using big data services. International

Journal of Internet Science, 9(1), 1-8. Retrieved from [Link]

Ritchel, M. (2013, April 27) How big data is playing recruiter for specialized workers. New York

Times. Retrieved from: [Link]

Rogers, H. (1967). Theory of recursive functions and effective computability (Vol. 5). New York:

McGraw-Hill.

Saxe, R. (2007). Theory of mind. In The Oxford Handbook of Cognitive Neuroscience.

Schwartz, L. M., Woloshin, S., Black, W. C., & Welch, H. G. (1997). The role of numeracy in

understanding the benefit of screening mammography. Annals of Internal Medicine,

127(11), 966-972. doi:10.7326/0003-4819-127-11-199712010-0000.

Silverman. R. (2015, November 10). Picking a health plan? An Algorithm could help. New York

Times. Retrieved from: [Link]

Sinha, R. R., & Swearingen, K. (2001, June). Comparing Recommendations Made by Online

Systems and Friends. In DELOS workshop: Personalisation and recommender systems in

digital libraries, 106.

Sniezek, J. A., & Buckley, T. (1995). Cueing and cognitive conflict in judge-advisor decision

making. Organizational Behavior and Human Decision Processes, 62(2), 159-174.

doi:10.1006/obhd.1995.1040

Sniezek, J. A., Schrah, G. E., & Dalal, R. S. (2004). Improving judgement with prepaid expert

advice. Journal of Behavioral Decision Making, 17(3), 173-190.

Soll, J. B., & Larrick, R. P. (2009). Strategies for revising judgment: How (and how well) people

use others’ opinions. Journal of Experimental Psychology: Learning, Memory, and

Cognition, 35(3), 780-805. doi:10.1037/a0015145

49
ALGORITHM APPRECIATION

Soll, J. B., & Mannes, A. E. (2011). Judgmental aggregation strategies depend on whether the

self is involved. International Journal of Forecasting, 27(1), 81-102.

Sparrow, B., Liu, J., & Wegner, D. M. (2011). Google effects on memory: Cognitive

consequences of having information at our fingertips. Science, 333(6043), 776-778.

Surowiecki, J. (2004). The wisdom of crowds: Why the many are smarter than the few and how

collective wisdom shapes business, economies, societies, and nations. New York:

Doubleday.

Waytz, A., Heafner, J., & Epley, N. (2014). The mind in the machine: Anthropomorphism

increases trust in an autonomous vehicle. Journal of Experimental Social Psychology, 52,

113-117.

Wegner, D. M., & Ward, A. F. (2013). How Google is changing your brain. Scientific American,

309 (6), 58-61.

Woolley, K., & Risen, J. L. (2018). Closing your eyes to follow your heart: Avoiding

information to protect a strong intuitive preference. Journal of personality and social

psychology, 114(2), 230.

Yaniv, I. (2004). The benefit of additional opinions. Current directions in psychological

science, 13(2), 75-78.

Yaniv, I., & Kleinberger, E. (2000). Advice taking in decision making: Egocentric discounting

and reputation formation. Organizational behavior and human decision processes, 83(2),

260-281.

Yeomans, M., Shah, A., Mullainathan, S., & Kleinberg, J. (2018). Making sense of

recommendations. Unpublished manuscript. Retrieved from:

[Link]

50
ALGORITHM APPRECIATION

51
ALGORITHM APPRECIATION

Appendix

Figure 1. The photograph viewed by participants in Experiment 1A.

52
ALGORITHM APPRECIATION

Table 1. Participants forecasted the rank of 10 songs on the “Hot 100” in Experiment 1B.

Artist Advice for

Song Song Rank
"Love Lies" Khalid & Normani 56
"Never Be The Same" Camila Cabello 16
"I Fall Apart" Post Malone 25
"Most People Are Good" Luke Bryan 48
"Bad At Love" Halsey 30
YoungBoy Never
"Outside Today" Broke Again 37
"Broken Halos" Chris Stapleton 46
"Heaven" Kane Brown 35
"Perfect" Ed Sheeran 5
"Dura" Daddy Yankee 52

53
ALGORITHM APPRECIATION

0.5
Weight on Advice (0-1)

0.4

0.3

0.2

0.1
Weight (1A) Songs (1B) Attractiveness Researcher Weight (2) Age (General
(1C) Predictions Discussion)
(1D)

Advisor
Human Algorithm

Figure 2. Weight on Advice (WOA) as a function condition, Experiments 1A, 1B, 1C, the
“joint” conditions of Experiment 2, and the normative experiment in the general discussion. The
higher the WOA, the more participants revised their estimate toward the advice they received.
For Experiment 1D, means represent researcher’s predicted WOA for participants in Experiment
1C. The means from Error bars indicate standard errors.

54