Understanding accountability in algorithmic supply chains

Jennifer Cobbe Michael Veale Jatinder Singh

University of Cambridge University College London University of Cambridge
United Kingdom United Kingdom United Kingdom

ABSTRACT of them [11, 77, 104]. Writing in 1996, Nissenbaum argued that
Academic and policy proposals on algorithmic accountability often computer systems raise a particular form of this problem – they are
seek to understand algorithmic systems in their socio-technical usually produced by groups or organisations rather than individuals,
context, recognising that they are produced by ‘many hands’. In- and may include components developed by others [77]. Addressing
creasingly, however, algorithmic systems are also produced, de- the ‘many hands’ problem is increasingly a concern in the algo-
ployed, and used within a supply chain comprising multiple actors rithmic accountability literature (implicitly or explicitly), with pro-
tied together by flows of data between them. In such cases, it is the posals in recent years for accountability for algorithmic systems to
working together of an algorithmic supply chain of different actors operate at an organisational level [23, 39, 60, 62, 67, 75, 89, 98, 110].
who contribute to the production, deployment, use, and functional- Yet today’s computer systems—including AI technologies—are
ity that drives systems and produces particular outcomes. We argue increasingly modular, dependent on cloud-based technologies, and
that algorithmic accountability discussions must consider supply interconnected. The ‘agile turn’ of recent decades transformed soft-
chains and the difficult implications they raise for the governance ware development, distribution, and infrastructure, directly influ-
and accountability of algorithmic systems. In doing so, we explore encing how businesses are organised and computing resources are
algorithmic supply chains, locating them in their broader technical distributed [45]. Agile development means software (including ‘AI’
and political economic context and identifying some key features models) is now produced in short development cycles with continu-
that should be understood in future work on algorithmic gover- ous testing and iterative revision after deployment [45]. Computing
nance and accountability (particularly regarding general purpose resources are now generally modularised and distributed as a service,
AI services). To highlight ways forward and areas warranting atten- with a client-server model in which the server performs the compu-
tion, we further discuss some implications raised by supply chains: tation [45, 74]. The challenges of scaling services and increasingly
challenges for allocating accountability stemming from distributed portable client devices drove advances in data centres with flexible
responsibility for systems between actors, limited visibility due to resources and software becoming increasingly cloud-based [45, 74].
the accountability horizon, service models of use and liability, and Consequently, software development now often involves, to vari-
cross-border supply chains and regulatory arbitrage. ous degrees, integrating pre-built modular components provided
as services and controlled by others into a complete product: not
CCS CONCEPTS simply a system, but a system-of-systems [99].
As a result, digital technologies across society and the economy
• Social and professional topics → Computing / technology
are increasingly organised around data-driven supply chains in-
policy; Socio-technical systems.
volving several interconnected actors and their systems. In these
supply chains, data flows between actors, linking systems designed,
KEYWORDS
developed, owned, and controlled by different people and organisa-
Algorithmic accountability, supply chains, AI as a Service, general tions [79]: a sensor system (controlled by one actor) might connect
purpose AI, political economy, accountability horizon to an analytics system (controlled by another) which itself outputs
ACM Reference Format: into a decision-making system (controlled by a third). This is often
Jennifer Cobbe, Michael Veale, and Jatinder Singh. 2023. Understanding so even for seemingly simple applications; for example, a home
accountability in algorithmic supply chains. In 2023 ACM Conference on thermostat can be driven by data from a national weather service,
Fairness, Accountability, and Transparency (FAccT ’23), June 12–15, 2023, which is itself fed data from thermometers owned and operated
Chicago, IL, USA. ACM, New York, NY, USA, 12 pages. [Link]
by different actors. In such supply chains, the working together of
1145/3593013.3594073
services and systems controlled by different actors produces par-
ticular outcomes—hardware capabilities, software functionalities,
1 INTRODUCTION the workings of commercial and industrial processes, ‘AI’ decisions
The ‘many hands’ problem holds that accountability is difficult and outputs, and more. Supply chains are data-driven in that the
where many people have together contributed to activity or out- flow of data between actors links them together, allowing a system
come, as it may be impossible to allocate responsibility to any one controlled by one actor to interact with those controlled by others
and together produce some functionality [22, 99]. In the context
of AI and algorithmic systems, algorithmic supply chains are
This work is licensed under a Creative Commons Attribution-NonCommercial those where several actors contribute towards the production, de-
International 4.0 License. ployment, use, and functionality of AI technologies. In these supply
FAccT ’23, June 12–15, 2023, Chicago, IL, USA
chains, AI ‘as a service’ providers often play key roles [24].
© 2023 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0192-4/23/06.
[Link]

1186
FAccT ’23, June 12–15, 2023, Chicago, IL, USA Cobbe, Veale and Singh

By reconfiguring software production and distribution, the agile organisational proposals to improve accountability in these supply
turn also had significant political economic and other ramifica- chains, but instead hope to produce a shift in focus for algorithmic
tions [22, 45]. In bringing services together to produce functional- accountability as a field and indicate new research directions
ity through supply chains, developers now delegate control over
much of the underlying technologies to others, complicating the
governance of those technologies and the products they are part of.
It is no longer necessarily true that computer systems are produced
2 ACCOUNTABILITY IN ALGORITHMIC
by a group of developers or an organisation, or by a vendor simply SYSTEMS
integrating various standalone components into one product (itself Significant academic and policy work has sought various forms of
raising the ‘many hands’ problem [77]). Instead, they often now accountability for algorithmic systems [110]. Accountability is of-
involve a group of organisations arranged together in a data-driven ten seen either as a mechanism (particularly in Europe and non-US
supply chain, each retaining control over component systems they Anglophone countries) or a virtue (particularly in the US) [13]. As
provide as services to others. Moreover, certain key actors in sup- a mechanism, it is an institutional arrangement whereby an actor
ply chains—in particular, major cloud providers who often control provides accounts to a forum, who deliberates on those accounts
underlying technologies—provide many services to millions of cus- and may impose consequences [12]. A developer might provide
tomers, holding important positions across supply chains in many information to a regulator about their system, for example, with
sectors [22, 24]. The agile turn has thus reorganised many areas of the regulator then issuing a penalty or requiring design changes.
social and economic life – now reconstituted around data-driven Some algorithmic accountability literature explicitly views account-
supply chains with a few systemically important actors providing ability as a mechanism for holding actors to account for their sys-
the core infrastructure that underpins contemporary society. tems [1, 23, 61, 109]. By contrast, accountability as a virtue is a nor-
Algorithmic supply chains bring significant implications for gov- mative concept, a set of standards for evaluating behaviour—often
ernance and accountability frameworks and mechanisms relevant to tied to being transparent, responsible, and responsive—with ‘being
algorithmic systems. Allocating accountability across supply chain accountable’ seen as a positive quality of particular actors [13].
actors for producing, deploying, and using algorithmic systems is Some (predominantly technical) work has thus sought to improve
relevant to general academic, policy, and regulatory discussions the accountability of certain technologies by imbuing them with
around algorithmic accountability, and to more specific legislative such positive qualities. Yet applying accountability to algorithmic
efforts around regulation of AI. Here we argue that governance of systems in this way—rather than to the organisations responsible
and accountability for algorithmic systems as deployed and used for them—often equates accountability with technical functionality
in the real world must operate across the supply chains which will (for example, building ‘Accountable AI’) rather with human virtues
increasingly underpin, drive, and produce their outputs and effects. which are not reducible to technically tractable concepts [61].
Much of the policy and academic literature, however, is grounded We treat accountability as a mechanism, whereby actors are held
in an organisation-focused understanding of digital technologies. accountable for technologies they are responsible for. However,
Even recent work which seeks to address the ‘many hands’ prob- accountability for digital technologies is often challenging. The
lem through a relatively broad view of accounting for algorithmic ‘many hands’ problem—that often no one person is responsible for
systems is typically focused on making specific stages of system outcomes which multiple people helped produce—has long been
lifecycle more transparent [7, 29, 39, 75, 85] or framed around the recognised: computer systems are rarely produced by an individ-
perspective of a single organisation [23, 89, 109, 110]. This atten- ual who can be held accountable, but by teams and organisations
tion now paid to organisations’ accountability for their algorithmic with many people contributing [77]. Moreover, modular software
systems was long overdue, but the focus on organisational account- development—where software developed by one organisation uses
ability has largely obscured the dynamics of algorithmic supply a library developed by another, for example—further complicates
chains. We therefore still lack ways to conceive of these chains, to things [77]. Much software is too complex, relying on too many
bring them within legal, regulatory, and governance mechanisms, components, for any one person to account for all of its workings.
and to appropriately distribute responsibility and accountability. In the context of algorithmic accountability, specifically, a key
This paper contributes to understanding these challenges. First conceptual shift has been in understanding these systems not as
(§2), we discuss recent trends in algorithmic accountability and iden- ‘algorithms’ but as algorithmic systems: “intricate, dynamic arrange-
tify limitations regarding supply chains. Next (§3), we describe and ments of people and code” [96]. This recognises that ‘algorithms’ are
contextualise AI services and algorithmic supply chains and identify produced and work within human contexts and in practice cannot
key features of how they are structured and operate. Then (§4) we be understood separately from them. Simultaneously, explanations
discuss important implications of these features for accountability: of (ML) model workings are increasingly recognised as insufficient
the distributed nature of responsibility in supply chains (§4.1); the to account for algorithmic systems [23, 34, 109]. Much research
limited understanding individual actors may have of the broader has therefore gradually moved away from seeking transparency
chain due to the ‘accountability horizon’ (§4.2); and providers’ ef- or explanations of models (though this remains an important area
forts to structure supply chains to maximise control and commercial of work) to understanding algorithmic systems more broadly as
advantage while minimising legal risk and accountability (§4.3). socio-technical phenomena. Much of this reflects—implicitly or
In all, we argue, algorithmic accountability work must urgently explicitly—an understanding that algorithmic systems are often the
address the technological, legal, and political economic dynamics result of ‘many hands’: produced by and deployed and used within
of algorithmic supply chains. We do not offer concrete technical or teams and organisations. To account for an algorithmic system,

1187
Understanding accountability in algorithmic supply chains FAccT ’23, June 12–15, 2023, Chicago, IL, USA

one needs to account for the collective efforts of the organisational 3 AI SERVICES AND SUPPLY CHAINS
processes involved in producing, deploying, and using it. Significant barriers to entry limit the number of organisations that
The term ‘algorithmic system’ is now widely used in algorith- can produce bespoke state-of-the-art AI technologies in-house,
mic accountability, with academic and policy literature commonly either for their own use or to bring to market [24]. Developing,
suggesting ways to improve accountability for their organisational maintaining, and renewing advanced AI technologies typically re-
aspects. Some proposals seek lower-level mechanisms to document quires large and relevant quantities of data, potentially from multi-
the choices and decisions made by people in developing, deploying, ple sources and labelled or moderated, relating to many use-cases,
or using a system, such as for datasheets [39] or data cards [86] contexts, and subjects. Cutting-edge model development requires
to describe datasets, or model cards [29, 75] and factsheets [7] to scarce expertise in model training, testing and deployment, all with
describe model specification and capabilities. Such proposals of- significant storage, compute, and networking needs.
ten recognise accountability as a positive quality (i.e. a virtue) and Companies with these capabilities now offer commercial ac-
seek improved transparency of algorithmic production and deploy- cess to cloud-based AI technologies ‘as a service’ (AIaaS) [24, 64].
ment processes. Higher-level proposals seek to integrate lower-level Major companies including Amazon [3], Microsoft [72], Google
mechanisms and provide ways of understanding holistically the (Alphabet) [42], and IBM [50] offer networked access to various
process of producing, deploying, and using algorithmic systems, state-of-the-art AI capabilities, including both model-building ser-
such as for auditability [110], reviewability [23], contestability [56], vices and pre-built (and ‘general purpose’) models in areas such
traceability [62], and others [109]. These have mainly reflected as language, speech, vision, and analytics (see [64]), or generative
accountability as a mechanism, and sought ways to support institu- models for producing text, images, audio, or video. Some companies
tional mechanisms and accountability relationships between actors offer specific services to customers, such as facial recognition [20],
and forums. Though coming from different perspectives, these var- hiring [48, 87], or medical diagnostics [51, 65]. And some operate as
ious lower- and higher-level mechanisms all essentially recognise platforms for all the above, looking to connect developers, clients
that algorithmic accountability—either as a virtue or a mechanism— and infrastructure providers, among others, in a multi-sided market
must reflect the ‘many hands’ nature of AI technologies. – Amazon and Microsoft, for example, offer access to models from
More recently, a ‘second wave’ of algorithmic accountability other providers alongside their own [73, 100], whereas other plat-
research has sought to address more structural concerns around the forms are primarily an intermediary (such as HuggingFace [37]).
development, deployment, and effects of algorithmic systems [81]. AI services can be integrated into apps and Web services, analytics
This work moves from creating better methods to scrutinise sys- systems, business and industrial processes, workflows, and with IoT
tems in situ to considering whether such systems should be built at devices with real-world physical effects (collectively: ‘applications’).
all, how, why, and who gets to govern them. This echoes longer- Low marginal cost and effort means this will likely become the
standing critical work in fields such as surveillance studies, which primary way that organisations integrate AI capabilities [24].
has considered the structural impacts of technologies of sorting AI services take various forms [64]. Here we focus on services
and profiling on societies, and in which arguments exist against offering access to pre-built ‘general purpose’ models and to cus-
using these technologies altogether [38, 47, 66]. While literature tomised models tailored using tools offered by providers. In these,
in these fields considers issues such as the cumulative effects of providers take major roles in the technology’s production and dis-
systems on individuals and communities [38], they typically con- tribution, developing and hosting systems on their (owned or man-
sider systems themselves through an organisation-centric lens – aged) infrastructure. Services are typically accessed through ap-
equating particular functionality (credit scoring, criminal profil- plication programming interfaces (‘APIs’) controlled by providers,
ing, airport screening, targeting advertising), with either the actor which allow the underlying system’s capabilities to be integrated
authorising the action (bank, police department, interior ministry, into applications by customers (Fig. 1). This client-server model thus
online platform), or a particular technology provider or contractor. allows providers’ algorithmic systems to run on their infrastructure,
Yet following the agile turn, an organisation-centric view is a less under their control, even while they are deployed by customers in
meaningful frame for analysis. Consequential algorithmic systems applications across many contexts and use-cases. There are typ-
are commonly produced, deployed, used, and have effects through ically few (if any) checks on customers’ identities or intentions,
and within supply chains. It is therefore no longer the case that soft- services use standard-form contracts (at least for smaller clients),
ware is generally developed by particular teams or organisations and customers are billed on the API calls made [24].
(who may have integrated components developed by others into An application’s supply chain may involve several AI and non-AI
their finished product). Instead, as we argue, functionality results services, potentially from multiple providers. Indeed, an AI service
from the working together of multiple actors across various stages of may be only one part of a broader, more complex chain for a given
production, deployment, and use of AI technologies (connected by application, which may integrate multiple AI and other services. Ac-
data flows across organisational, legal, technical, visibility bound- tors in these chains are broadly ‘upstream’ or ‘downstream’ from the
aries). This does not mean that a particular single organisation will perspective of others – though this distinction can blur where actors
never be appropriate to hold to account, but that identifying the take multiple positions in a chain. AI services themselves have sup-
actors and processes that led to the functionality of any particular ply chains, such as for dataset production activities like data gath-
algorithmic system becomes significantly less straightforward. ering and labelling [69] (see §4.3.2), typically involving data from
We next (§3) explore key features of algorithmic supply chains, customers’ application deployments [24] and from providers’ own
followed by their challenges and implications for the mechanism user-facing platforms and services. Customers therefore appear in
of algorithmic accountability (§4).

1188
FAccT ’23, June 12–15, 2023, Chicago, IL, USA Cobbe, Veale and Singh

third party third party customer provider third party

1) data
2) data
3) data (provider API input)
4) data (provider API output)
5) data

Figure 1: Sequence diagram of a simplified data-driven supply chain with an AI service. The customer sends input data to
provider’s API, who performs some computation, before returning the results to the customer. Some of the broader supply-chain
is illustrated, where the customer has previously received data from other third parties, and later, sends some data to another.
supply chains for deployment and production of some AI services, chain. Studying an actor and their systems in isolation from supply
while end-users of customers’ applications and providers’ services chain contexts is akin to studying an algorithmic model in isolation
are themselves drawn into production. Firms using AI services may from its broader organisational context (the limitations of which are
themselves provide services to their own customers [74], such as increasingly recognised (see §2)). The dynamics of interdependence
proctoring software sold to universities repackaging Amazon’s fa- in algorithmic supply chains—how they are structured, the relative
cial recognition service [4], or copywriting software repackaging importance of actors, and how problems spread—are therefore key
OpenAI’s text generation model (GPT-3 [103]). considerations for algorithmic accountability, as we now explore.
We identify several key features of algorithmic supply chains:
3.1.1 Supply chain interdependencies are structured by technologi-
• production, deployment, and use are split between several cal, legal, and political economic factors. Interdependence between
interdependent actors; actors gives algorithmic supply chains their structure and func-
• supply chain actors and data flows perpetually change; tionality. Certain actors—typically (AI and non-AI) cloud service
• major providers’ operations are increasingly integrated providers—have leveraged AI technologies they own, production
across markets and between production and distribution; processes they control, and cheaply-accessed networking technolo-
and gies to pursue particular interdependencies with others and strate-
• supply chains are increasingly consolidating around system- gically position themselves in markets and supply chains of many
ically important providers. kinds. Technologies afford certain capabilities to those who use or
We draw out their implications for accountability in §4. control them [30, 41, 78]. They can therefore also afford the ability
to do things that fulfil the needs of others. Because, as we discuss in
3.1 Supply chains split production, deployment, §3.1, people doing things for each other produces interdependence
between them [35], different technologies can afford different kinds
and use between interdependent actors
of interdependencies. Networking and data processing technolo-
In algorithmic supply chains, different aspects of production, de- gies, for example, allow the stages of production and deployment of
ployment, and use of AI technologies are split between multiple AI technologies to be distributed geographically. They can therefore
actors tied together by data flows. As a result, the activities of be done by different people, each of whom does something for the
the various actors in supply chains each depend on the actions others, producing interdependence between them.
of by others. This may involve various interacting AI and non-AI However, technologies and their affordances cannot determine
technologies, such as cloud services, servers, data centres, data interdependencies or the structure of supply chains. Affordances
sources, and content delivery networks, controlled by different ac- are not objective properties of technologies, but depend on context
tors. The working together of the various actors who control these and perspective [30, 32, 41, 78, 108]. How providers can strategically
technologies—each doing something that enables, supports, or facil- position themselves is thus shaped both by their technologies’ af-
itates the actions of others—produces a particular outcome (see Fig. fordances and by social, legal, and political economic factors which
2). Each actor in a supply chain may not be aware of the others, nor also influence how actors relate to each other, what they do for each
have consciously decided to work together towards that outcome – other, and the interdependencies that arise. Accordingly, to posi-
indeed, they may have limited understanding of actors even one or tion themselves in supply chains and markets, providers have also
two steps removed (see §4.2). However, each depends on something leveraged political economic factors such as economies of scale and
done by others, and their role in a supply chain is contingent on favourable legal frameworks such as intellectual property, interme-
the activities of actors both up– and downstream of them. diary liability, and data protection [24, 25, 27]. Political economic
Actors in algorithmic supply chains are thus interdependent, each and legal factors—not just technological—are thus important in
doing something to fulfil the needs of others (such as processing a producing and structuring algorithmic supply chains.
particular data input and returning an output, or providing infras-
tructure to support application deployment). The interdependence 3.1.2 Some actors are core players in supply chains. Supply chain
of the various actors responsible for developing, deploying, and actors are generally not equal in their interdependence with each
operating algorithmic systems in supply chains means they are other, and some may do things that others particularly depend
not individual, independent actors as such. Instead, these actors, on. Their services, for example, may be relied upon by multiple
their relations, and their role in the workings and effects of AI others, as is often the provider’s aim in offering general purpose
technologies can only be understood in the context of that supply services. Providers may depend on each customer only a little,

1189
Understanding accountability in algorithmic supply chains FAccT ’23, June 12–15, 2023, Chicago, IL, USA

A B

Figure 2: A representative AI supply chain. The application developer (blue) initiates a series of data flows by sending input data
to an AI service provider (grey). One AI service provider (red) appears at multiple key points in the supply chain – providing
infrastructure (A) for an AI service offered by (grey); providing an AI service (B) to another cloud service provider (orange); and
providing technical infrastructure (C) for application deployment.

while customers may depend on the provider for business-critical undocumented set of actors and interdependencies found in many
application functionality. Supply chain interdependencies are thus chains. Statistical guarantees may not hold when systems are com-
often asymmetric, with certain actors—typically including at least posed together, and it is not straightforward to evaluate a whole
those responsible for production of AI technologies—performing system when each individual component may have been evaluated
core functions for others, while others are more peripheral. Various under different threat models (or other critiera) [58, 64]. Unless
contextual signs might indicate that an actor is core in a particular identified by the provider, actors ‘downstream’ from them may be
chain. For example, they may be the application developer who calls unaware of a problem until they notice some unexpected behaviour.
the supply chain into existence. They may perform some function Even then, because they have delegated key aspects of production
(such as providing an AI service) which is crucial to application (and possibly deployment) of the technologies their application
functionality. They may provide a key step between actors (such relies on to other actors (such as AI service providers), customers
as offering access to another provider’s technology through their may struggle to understand where in a supply chain the problem
API) upon which subsequent steps of the chain depend. Or they has arisen, why, and what they can do to mitigate it.
may appear at multiple different points in the chain providing
cloud-based technical architecture on which functions performed
by other actors rely. Some actors may also be more interchangeable 3.2 Supply chains are transient and dynamic
and replaceable than others – the barrier to entry for applying a
with unstable interdependencies
specific off-the-shelf API is typically substantially lower than to
generate the underlying technology to begin with. Agile development combined with services-based distribution mod-
These asymmetries of interdependence produce asymmetries els has produced algorithmic supply chains which operate as dy-
in power [35]: where one actor depends more on things done by namic processes of data flow between a changing number and ar-
another than that other depends on them, the balance of power rangement of actors. Just as critical engagement with algorithmic
between them favours the second actor [35]. An application de- systems must recognise that they change over time [96], so do
veloper, for example, who uses a major provider’s AI service, will algorithmic supply chains. Indeed, because data flow ties actors
depend more on that provider than the provider—who has many together, a chain may differ each time it is instantiated. At various
customers—will depend on that one developer. Power balances in points there may be multiple possible directions for data to flow
algorithmic supply chains thus arise relationally yet asymmetrically between actors depending on the outcomes of analyses performed
and change over time as the relations and interdependencies be- on it. A face detected in a video stream using one service, for exam-
tween actors evolve [35]. These power balances are not determined ple, might trigger a flow to a separate facial recognition service to
by actors’ relations to the technologies involved, but through their identify the person (with its own supply chain and associated data
relation to and interdependence with each other. As we note (§3.1.1), flows) and back again. This might trigger a flow to a third system
this is subject to many potential influences, of which factors like to record and alert of the presence of a particular individual. Supply
control of production processes and APIs are just some. But, by chains can thus be dynamically instantiated, and their structure
leveraging their technologies alongside legal, social, and political may vary depending on the input data and the outputs of com-
economic factors, providers can hold significantly asymmetrical ponent systems. A supply chain’s structure—the actors involved,
positions as core actors in many supply chains, with power balances what they do for each other, the interdependencies and power bal-
between them and others heavily in their favour. ances between them—may therefore only be fully apparent once
the functionality or outcome has been been produced.
3.1.3 Interdependence helps problems propagate. Supply chain in- However, technical, legal, or economic relations between actors
terdependencies mean problems with one actor’s technologies can do often persist across multiple instances of a particular supply
propagate through other actors’ systems. Where an AI service is chain. An application developed to use a particular provider’s ser-
biased in some way (such as facial recognition performing poorly vice will typically use that service repeatedly, even if the path of
on particular demographics [15]), that bias will be inherited by data flow between actors differs between instances. As such, while
applications relying on that service [64]. Such a cascade’s effects supply chains may change overall, bilateral relations between par-
may be complex and unpredictable given the dynamic and largely ticular actors may remain relatively consistent. However, the nature

1190
FAccT ’23, June 12–15, 2023, Chicago, IL, USA Cobbe, Veale and Singh

of their relationship—the services provided and used, for example— resources needed to improve models, while offsetting some research
may still change over time. An application developer may introduce and development costs by bringing it into a process paid for by
new features, for instance, which use additional services offered customers [24]. They can thus lower the net cost of developing
by the particular provider. They may deprecate features such that more accurate and more generalisable systems [24].
particular services are no longer needed. They might employ ad- However, vertical integration has limits. AI providers might not
ditional support services for rapid growth in application resource operate in-house data cleaning and labelling processes, for instance
requirements (for example, where an application ‘goes viral’). The (key parts of training, testing, and updating models). The business
provider may change their terms of service (altering the legal rela- benefits to providers of bringing these processes ‘in house’ are
tionship between them) or withdraw particular services (resulting potentially outweighed by the commercial advantages of extending
in changes in the developer’s application). These are just some of supply chains across borders to exploit differences in laws (§4.3.2).
the ways that relationships between actors may change. Aspects of AI production are often instead outsourced to low-paid
and insecure workers in the Global South [46, 84] (through data
3.3 Some actors are integrated across markets cleaning and labelling services offered by companies like Sama
and between production and distribution AI [95], or through Mechanical Turk [2]). Moreover, some major
providers now offer access through their services to generative
Some providers of AI and other cloud services commonly found in
(foundation) models produced and controlled by others, marking a
algorithmic supply chains have reached high levels of integration;
shift towards less integration in some emerging product sectors.
both horizontally (across markets and sectors), and vertically (across
production and distribution processes). This has implications for
3.3.3 Providers all the way down. Though some prominent AI
their positioning and role in algorithmic supply chains.
providers are both horizontally and vertically integrated, most are
3.3.1 Horizontal integration. Horizontally integrated companies not. Instead, they tend to specialise in a few closely-related services,
operate across markets and sectors. The most prominent cloud such as algorithmic recruitment, processing legal documents, cer-
providers (Amazon, Microsoft, Google, Alibaba, IBM) offer various tain medical processes [64], and even ‘algorithmic governance’ and
services across many related and adjacent markets and may appear ‘ethical AI’ (see [33]), without operating across traditional cloud
repeatedly in a supply chain. Some such services are AI-related; service markets. These specialist providers typically ‘rent’ techno-
others are infrastructure for applications (storage, database, content logical infrastructure from a larger provider rather than operating
delivery, credential management, and so on); still others are user- their own (OpenAI, for example, exclusively uses Microsoft’s Azure
facing, from business and consumer web-based services (such as cloud services [73]). This reflects the fact that developing advanced
maps or photo backup) to software packages for customers and their AI technologies and operating them at scale will in many cases
users (such as Microsoft 365). This allows a single provider to re- require technical resources beyond the means of all but the biggest
purpose their AI and other technologies across a range of services, providers. As a result, whether through their own AI services or
both infrastructural and user-facing. It is common for providers to through those of others who depend on their cloud infrastructure,
purchase potential competitors and new market entrants, either to major providers like AWS, Microsoft Azure, and Google Cloud will
obtain intellectual property, to expand their services across markets, be crucial players in future AI development and distribution.
or to stifle emerging competition in existing markets. Providers can Some AI-specific providers’ services can be accessed only
also simplify how existing customers bring AI services within their through a larger provider’s interface and brought by customers
applications by providing tools to facilitate integrating them with into applications through that specific provider’s cloud, rather than
their other services. Providers may financially incentivise customers through a competitor (OpenAI’s commercial services can be ac-
to use several of their services instead of those of competitors. cessed only through Azure [73]). The larger provider’s cloud of-
fering thus operates as a platform through which they facilitate
3.3.2 Vertical integration. Vertically integrated companies control and can gatekeep market access to the smaller provider’s service.
multiple stages of production and distribution. Several major AI In some cases, one cloud provider’s interface may be used to ac-
providers—primarily Amazon, Microsoft, and Google—own key cess a specialist AI provider’s model [100], where that specialist
infrastructure for producing their systems and distributing them as provider itself uses a different cloud provider for their supporting
services across markets: data centres and servers; content delivery infrastructure for development [6]. That is to say, several larger
networks; APIs and customer-facing interfaces; and network infras- cloud providers may be involved at different stages of production
tructure. Vertical integration offers bespoke technical infrastructure and deployment of specialist AI providers’ services (and indeed,
specific to these providers’ needs which they can use for many ser- those of others).
vices across markets to exploit economies of scale. High resolution
media (requiring significant resources), for example, thus encour-
ages vertical integration, as does state-of-the-art AI production
3.4 Supply chains are increasingly consolidating
(requiring more data, bigger and more complex models, intensive around systemically important providers
compute, and sophisticated training and testing processes). Ver- The dynamics of interdependence and integration mean that algo-
tically integrated providers can link deployment of systems by rithmic supply chains are increasingly consolidating around (pri-
customers to their production processes, testing and further refin- marily) Amazon, Microsoft, and Google [22, 24, 93]. Several fac-
ing those AI technologies using customers’ input data, applications, tors tend towards consolidation, including competitive advantages
and real-world use cases [24]. This allows providers to reduce the offered by integration. These companies span markets, offering

1191
Understanding accountability in algorithmic supply chains FAccT ’23, June 12–15, 2023, Chicago, IL, USA

developers ‘all-in-one’ packages with easy access to state-of-the- ways of making the technology more transparent or understand-
art technologies, which readily scale and enable ‘global’ reach. In able (though this can help understand specific points in particular
AI production, they leverage bespoke and advanced computing systems’ lifecycle). Instead, algorithmic accountability work must
resources and expertise, significant quantities of data represent- consider broader factors: how providers leverage technology and
ing real-world deployments and use-cases, and economies of scale law to structure interdependencies, integrate their operations (§3.3),
across AI and non-AI customer bases. They can therefore offer consolidate their position (§3.4), increase their control and power
services at lower cost, broader scale, greater technical sophistica- while minimising legal accountability (§4.3.1), and extend their sup-
tion, and with potentially easier access than many competitors. ply chains across borders to minimise cost and legal risk and max-
Moreover, their substantial financial resources help consolidate imise commercial benefit (§4.3.2). The dynamics of supply chains,
their position through purchases of and investments in potential the legal and political economic factors influencing their structure,
competitors (such as Google’s purchase of DeepMind [40], or Mi- and the relations and interdependencies between actors that result
crosoft’s investment in OpenAI [73]). are all significant considerations from a view of accountability as
As a result, major providers are systemically important for the po- a mechanism—one, in particular, for investigating, understanding,
litical economy, governance, and accountability of AI. Even where assessing, challenging, and contesting power. They are also impor-
an application does not use a major provider’s AI services (using the tant in considering who should be accountable, to whom, for what,
developer’s own AI technology, for example, or obtaining it from and through which mechanisms and institutional arrangements
a smaller provider), major providers’ non-AI services may form
significant parts of supply chains for either that application or the 4.1 Responsibility for algorithmic systems is
AI service it uses (or both). These providers are therefore core actors distributed between several actors
in many supply chains, strategically positioning themselves across
Governance and accountability mechanisms around algorithmic
markets in a process of enclosure of AI-technological infrastructure
systems should address the distributed responsibility in algorithmic
and, by extension, of businesses, institutions, organisations, and
supply chains. Different actors control aspects of commissioning,
sectors relying on supply chains involving their services. They are
designing, developing, deploying, using, or monitoring a particular
thus positioned in commercially beneficial interdependencies both
AI technology. Responsibility for the workings and outcomes of
with other actors in particular supply chains, but also more broadly
supply chains is thus distributed among several actors who may not
– a few dominant providers underpin important social and eco-
be straightforward to identify nor consistent across instances. Even
nomic processes while themselves depending to various degrees on
when some actors are influential, there is therefore typically no one
many actors in social, legal, technological, and political economic
actor in overall control of a supply chain. Existing accountability
processes which help produce and maintain their position.
literature, however, typically assumes that (while models or input
data might change) the actors and components remain relatively
4 (IMPLICATIONS FOR) ACCOUNTABILITY IN stable. Yet directing governance and accountability mechanisms at,
ALGORITHMIC SUPPLY CHAINS or allocating accountability to, the wrong actors in supply chains
risks undermining the stated goals of these mechanisms. Account-
Algorithmic supply chains bring difficult implications for gover-
ability involves a relationship where an actor provides accounts
nance and accountability. Much algorithmic accountability research
of their activities to a forum, who imposes consequences to cor-
often reflects an organisation-focused understanding of accountabil-
rect the actor if needed [12]. For accountability mechanisms to
ity (§2). Yet the production, deployment, and use of AI technologies
succeed, it is therefore crucial that the right actors are assigned to
in supply chains is split between multiple actors who together
the appropriate relationships. In this context, those who are fac-
produce its workings and effects and whose part in producing func-
tually responsible for various aspects of production, distribution,
tionality cannot be understood separately from the chain (§3.1).
and use of algorithmic systems must be identified correctly so that
Organisation-focused framings cannot properly capture this distri-
accountability can be allocated accordingly.
bution of responsibilities between actors across the stages of the
AI lifecycle, which also challenges assignments of accountability in 4.1.1 Legal accountability and distributed responsibility. Some ju-
relevant legal frameworks (§4.1). Moreover, problems with systems risdictions have sought to address distributed responsibility in data-
can propagate widely downstream through supply chains (§3.1.3), driven supply chains more generally. The Court of Justice of the
yet particular actors are often unaware of the broader chain, and the European Union (CJEU) has attempted to contend with this in data
limits of visibility across supply chains make interventions like risk protection law, for example. A key question in data protection law
assessments difficult (§4.2). It is therefore crucial for governance is who is a data controller – factually in control of, and therefore
and accountability mechanisms to understand the actors in supply primarily responsible in law for, personal data processing [36]. The
chains, what they do for each other, which of them take core roles, CJEU has repeatedly held that multiple parties can be controllers for
and the interconnections and interdependencies between them. At some or all aspects of a chain of processing [16–18, 24, 68]. Where
the same time, however, the dynamic, transient nature of supply several actors have common interests in the processing, they may
chains (§3.2)—which can potentially be instantiated each time and be joint controllers [36]; where their interests in the processing di-
unfold differently as data is processed—is also challenging. verge, they may be separate controllers. In doing so, the Court made
Moreover, algorithmic supply chains are structured through in- several observations: actors can be controllers if they have influence
teractions between technological, legal, social, and political eco- despite not having actual access to the personal data [16–18], con-
nomic factors (§3.1.1). It is therefore not enough to attend only to trollers are not typically responsible for parts of the chain before or

1192
FAccT ’23, June 12–15, 2023, Chicago, IL, USA Cobbe, Veale and Singh

after those they actually influence [18]; and using another actor’s 4.1.2 Allocating accountability. Governance and accountability
platform does not exempt a controller from their obligations [16]. mechanisms should therefore be grounded more clearly in and
Recognising the plurality of actors in chains of processing is wel- emphasise an understanding of the distribution of responsibility in
come, but even data protection law’s more nuanced assignment of algorithmic supply chains. Not every actor in a supply chain will be
roles and responsibilities may not readily map to algorithmic supply responsible for the outcome of the algorithmic system – some will
chains [24, 45, 68]. Under current understandings, AI service cus- provide only supporting services which do not meaningfully affect
tomers are likely data controllers (the dominant party, ultimately outcomes. Neither will actors who are in some way responsible
responsible for compliance and accountability), while providers be equally responsible – some play a bigger role than others in
may be data processors (the subordinate party, acting only under determining outcomes. Nor will they be responsible for the whole
the instruction of a controller, with limited obligations) [24, 68, 74]. supply chain – different actors control different aspects of it. Ac-
Yet this assignment of legal roles and responsibilities does not de- countability should thus be allocated to actors across supply chains
scribe the real interdependencies and power relations between AI based on a proper understanding of their technological and polit-
service providers (who are in control of their technologies, often ical economic dynamics. This requires processes and criteria for
core actors in supply chains, potentially systemically important identifying the distribution of responsibility across supply chains
more generally, typically presenting customers with ‘take-it-or- and allocating accountability to those actors, for which activities,
leave-it’ contracts, and to a large extent determining AI-driven accounting to whom, and with what possible consequences.
functionality in customers’ applications through their production It is therefore important to understand the distribution of re-
processes) and their customers (potentially small companies with- sponsibility in algorithmic supply chains in terms of who is doing
out AI expertise, typically with no access to the provider’s systems, what for whom, who is performing what key functions for oth-
control over them, or knowledge of how they work) [24]. Even ers, who is core to certain supply chains, and who is systemically
where providers are likely controllers for aspects of the service— important. Particular attention is due to systemically important
such as where they use customer data for service improvement— actors – primarily Amazon, Microsoft, Google, and perhaps a few
they typically attempt to minimise responsibility by claiming in others. Though technological and political economic dynamics tend
their service agreements to be processors [24]. Yet the CJEU has towards consolidation around these companies, and though non-AI
consistently held that the factual situation outweighs contractual services often provide supporting infrastructure, it is still impor-
or other arrangements, and regulators have contradicted claimed tant to have ways to determine which aspects of supply chains
assignments of legal roles in other kinds of data-driven supply are key to their outcomes and effects, as opposed to those which
chains [106]. Challenging providers’ claims, however, would in- could be interchanged without affecting those things. The latter,
volve litigation or regulatory investigation. Moreover, given the while potentially significant, are perhaps less of an urgent subject
need for joint controllers to agree the division of controllers’ duties of governance and accountability mechanisms than the former.
and responsibilities between themselves [36], it is not clear how
joint controllership can work where actors don’t necessarily know 4.2 The accountability horizon limits visibility
of each other or have any direct relationship. across supply chains
The EU’s proposed AI Act suffers from related tensions. It recog-
A significant challenge for governance and accountability mecha-
nises that the ‘user’ of an AI system (in this context, generally
nisms in algorithmic supply chains is the accountability horizon
the customer of an AI service) may differ from its ‘provider’, and
– the point beyond which an actor cannot ‘see’, which depends
envisages circumstances where a user of an AI service does so for
on the actor and the chain. Supply chain actors will generally be
a purpose not intended by the provider, and thus in law becomes
able to know whom they interact with directly (a first ‘step’ in the
responsible for the underlying system [28]. However, this does
chain), and perhaps whom those first step actors interact with in
not reflect supply chain interdependencies and dynamics, where
turn (a second ‘step’), but may not be able to know about the data
production, deployment, and use are distributed between actors.
flows and interconnections beyond [22, 99]. Moreover, distributed
Instead, in this circumstance, the Act would potentially make actors
responsibility between actors (§4.1) means each has incomplete
several steps downstream from production responsible for ensur-
information even if they do know who is up- and downstream
ing that the AI technology complies with production-related legal
of them. AI service providers, in control of production, may lack
requirements around training and testing, accountability, and risk
knowledge of downstream contexts and use-cases of application
management. While the user would inevitably be unable to com-
deployments [64]. Those responsible for deployment and use typ-
ply (due to the actual distribution of practical responsibilities in
ically lack access to models and often to information about their
algorithmic supply chains), the actor who developed and controls
specification, training, testing, validation, and so on (and thus may
that technology and is thus factually responsible for production
have limited understanding of their capabilities and limitations).
would face no obligations. This may incentivise actors who can
The accountability horizon is thus a problem for producers of
never provide assurance of compliance to pretend they can – eas-
algorithmic systems in the earliest stages of developing their tech-
ily done due to the Act’s self-regulatory framework and limited
nologies (problem framing, §4.2.1) and for legal and other gover-
planned regulatory capacity [107]. Regulatory systems which hold
nance frameworks based around risk management (§4.2.2).
supply chain actors downstream of production to account for design
and development may do little to regulate those who are factually 4.2.1 The accountability horizon makes problem framing difficult.
responsible for production and who benefit financially from poten- Many algorithmic issues stem from choices around problem defi-
tially unlawful API queries, effectively shielding them from liability. nition and framing that inform system design. Complex concepts

1193
Understanding accountability in algorithmic supply chains FAccT ’23, June 12–15, 2023, Chicago, IL, USA

may be formalised poorly, tasks may be incompletely captured, risks that might arise [24, 53]. Similarly, without knowledge of or
and different contexts may be insufficiently considered [97]. The influence over production, customers cannot reliably assess how
‘many hands’ problem made critical questions of identifying who systems are developed, nor ensure that systems are appropriate to
framed the problem and when it was framed difficult to answer the risks arising in their context [64]. Even where they have some
[82]. Supply chain dynamics giving rise to the accountability hori- knowledge, models are regularly updated, and customers may lack
zon complicate this further. Those responsible for production have visibility or capacity to reassess. In many cases, therefore, no actor
limited capacity to understand the contexts of deployment and use will have sufficient knowledge of or control over both production
by others, while the actors closest to the problem—those deploying and deployment to be able to reliably assess or mitigate the impacts
or using the system—are generally unable to influence its design. and risks. Risk management approaches to governance and account-
Moreover, due to the split between production and deployment, ability of AI technologies are therefore arguably not appropriate in
application developers necessarily engage in their own problem this context (despite their importance in emerging laws applying
framing – determining whether they need an AI service to address to algorithmic systems, such as the EU’s AI Act [28]).
a particular problem and, if so, which is most suitable. Yet they may
4.2.3 Expanding the accountability horizon. The accountability
lack capacity to determine which service (if any) is most appropri-
horizon thus poses major problems for accountability. Interven-
ate to their needs (particularly if organisations swap organisational
tions are needed to expand the horizon and better place actors
and IT know-how for license managers [10]). This is further com-
to know more about their own supply chains, and support others
plicated by the fact that not all services are fungible, or adaptable
in knowing more about theirs. Yet organisation-focused tools to
to a range of different framings. Services may only accept certain
provide information on points in the AI lifecycle (such as [39, 75])
kinds of input data, produce certain kinds of output data, or be
are of limited help where information about interconnections be-
amenable to certain kinds of alteration and customisation. They
tween actors is needed. Tracking data flow between actors could
may be developed with particular underlying assumptions which
help understand interconnections beyond the first few steps [99],
can (or should) preclude their deployment or use in other con-
as could legal and institutional mechanisms requiring information
texts [64]. Supply chain integration may further reduce flexibility
about arrangements. ‘Know your customer’ requirements around
in problem framing, as technical hurdles to limit interoperability
on-boarding for AI services (common in financial services) could
and cost implications make components less readily swappable
help providers understand customers’ purposes and intentions [24],
(particularly where services are strategically bundled by providers).
as can technical measures for monitoring how their services are
More cynically, actors may encourage problem framing which
(mis)used [53, 54] (though these only give some visibility over one
increases demand for their own products and services. For example,
or two steps in the chain). Recent CJEU data protection jurispru-
organisations selling technologies for input data, such as cameras
dence on transparency rights confirms that data subjects have the
and other environmental sensors, may also sell workplace moni-
right to know the identities of recipients of their personal data [19],
toring tools which take advantage of the data produced by these
which may help understand data flows. However, where the data
sensors. Application developers with low problem framing capacity
controller does not know those identities—perhaps likely due to the
might adopt these tools without properly identifying whether they
accountability horizon—data subjects can instead be told about the
need workplace monitoring at all. The dependency of application
categories of recipients [19] (significantly less useful information).
developers on supply chains may therefore risk the autonomy of
A particular difficulty, however, is that accountability is contex-
those organisations [10]. Indeed, using AI services leaves healthcare,
tual [5, 12, 23, 80, 109, 110]. The information needed to account
education and other established sectors vulnerable to unbundling
for an algorithmic system depends on the actors responsible for
and rebundling of their fundamental operations, leaving each stage
its development, deployment, and use, on the forum owed the ac-
amenable to value extraction through servitisation [10].
count, and on the broader context [23]. As such, the mechanisms
needed to record, process, and provide information about algorith-
4.2.2 The accountability horizon makes risk management difficult.
mic systems—such as [7, 21, 29, 39, 62, 75]—are also contextual. Yet
Many academic, policy, and legislative initiatives propose impact as-
the accountability horizon makes understanding context difficult
sessments, risk assessments, and risk management mechanisms to
for those who account for their part in supply chains. They may
mitigate harms of AI technologies (for example, [1, 8, 23, 28, 31, 34,
not know whom they need to account to, so may not know what
39, 43, 49, 52, 53, 56, 57, 59, 60, 63, 70, 75, 88, 89, 92, 94, 110]). Data
information to retain, about what aspects of their processes, and in
controllers’ management of risks to data subjects’ fundamental
what form. They may also not know which actors upstream from
rights is also core to the EU’s data protection regime [25, 36]. Typi-
them they can obtain accounts from. In general, the difficulties
cally, certain actors—who may differ between legal frameworks—
raised by the accountability horizon are not easily overcome.
have some obligation to identify and mitigate risks to individuals
or their rights arising from technologies they develop or control.
However, the accountability horizon makes effective risk man-
4.3 Providers structure supply chains to
agement difficult if not impossible. These mechanisms require minimise accountability
knowledge of both the AI technology’s specification and devel- Supply chain dynamics allow providers to maximise commercial
opment (i.e. production) and the purpose and context of its applica- benefit while minimising legal accountability by (i) extending con-
tion (deployment) [24, 64]. Yet without advance knowledge of their trol over downstream deployment of AI technologies (§4.3.1) and
customers’ many, varied, and changing application contexts and (ii) extending their own supply chains across borders to engage
uses, providers cannot properly account for the range of potential in regulatory arbitrage (§4.3.2). Providers thus use a techno-legal

1194
FAccT ’23, June 12–15, 2023, Chicago, IL, USA Cobbe, Veale and Singh

strategy to position themselves advantageously in markets and cross-border nature of supply chains and difficulties of enforcement
shape supply chains to maximise revenue and reduce risk [26, 27]. remains a significant accountability challenge.
4.3.1 Servitised distribution models give providers control beyond
deployment. Nissenbaum observed that, in the mid-1990s, software 5 CONCLUSIONS AND FURTHER RESEARCH
vendors often demanded property protection for their products The ‘many hands’ problem has motivated efforts to provide infor-
while denying, as far as possible, accountability for them [77]. Soft- mation about the production, deployment, and use of algorithmic
ware licensing agreements precluded ownership by users and em- systems by teams and organisations (§2). The emergence of AI ‘as a
phasised the producer’s rights, while disclaiming their legal ac- service’ (or ‘general purpose AI’) and developments associated with
countability for the software or anything it might do – even where cloud computing and the services model of software distribution
harms resulted directly from defects in it [77]. Developers thus (§3) challenge organisation-focused understandings of algorithmic
attempted to maintain control over their software to the extent accountability (§4) in ways that have not been widely addressed.
possible given the distribution model at the time (typically physical AI technologies now often involve algorithmic supply chains,
media), while generating artificial scarcity for an information prod- with their production, deployment, and use split between multi-
uct to maximise revenue with minimal risk and responsibility. This ple actors who together produce the technology’s outcomes and
produced, as Nissenbaum puts it, a ‘vacuum’ of accountability [77]. functionality (§3.1). Major providers—now highly integrated both
Software’s distribution has moved away from (licensed) physical horizontally and vertically (§3.3)—are systemically important play-
media to the service-based, API-centric models described [45]. Com- ers (§3.1.2), and supply chains are increasingly consolidating around
bined with asymmetrical interdependence in algorithmic supply them (§3.4). Issues with particular systems can propagate through
chains (§3.1.2), service models offer providers new ways to extend supply chains (§3.1.3), while they often change between instances,
control past the point of deployment. Because providers depend less making it difficult to understand how they operate or who is in-
on individual customers than customers depend on them, providers volved (§3.2). Together, these dynamics of interdependence, perpet-
can impose contractual service agreements and use APIs as tools ual change, integration, and consolidation produce supply chains
to advantageously structure their relations with customers and in which responsibility for algorithmic systems is distributed be-
others. Where vendors once sought expansive intellectual property tween interdependent actors (§4.1) and visibility across the actors
protections, today providers seek to use service agreements to max- involved is low (§4.2). This challenges existing legal accountability
imise control over deployment of their technologies by reserving frameworks while limiting the effectiveness of mechanisms like risk
rights to dictate terms of use and change, withdraw, or cancel prod- assessments. Moreover, splitting production and deployment makes
ucts and services at will. Providers disclaim legal accountability it difficult to appropriately develop or choose AI services (§4.2.1). At
for things that happen through use of their services [55, 71, 105], the same time, the services distribution model allows providers to
and attempt to position themselves as data processors (§4.1) even use terms of service and APIs to minimise legal accountability and
when using customer data for their own purposes and thus are maximise control over technologies beyond deployment (§4.3.1),
likely the controller for that processing [24]. Providers can also while simultaneously extending their own production processes
use APIs as ‘projections’ [14] of the asymmetric balances of power across borders to exploit differences in regulatory regimes (§4.3.2).
with customers, to destabilise attempts to hold providers to account: In all, the characteristics of algorithmic supply chains we have
using APIs as tools to shutter businesses, undermine research, and identified and the implications they raise challenge existing ap-
evade scrutiny [9], while contractually giving themselves those proaches to algorithmic accountability. Future algorithmic account-
rights and using changes in information policy to control [91]. ability research must therefore contend with supply chain dynam-
ics: how they are structured, how they develop over time, how
4.3.2 Cross-border supply chains permit regulatory arbitrage. As
AI’s functionality and effects are produced through them, and—
we describe, data processing and networking technologies afford a
importantly—how distributed responsibility challenges governance
geographical distribution of AI production and deployment (§3.1.1).
mechanisms and the accountability horizon limits visibility. This
The same technologies allow various production-related activities
requires a broad view of supply chains, seeking to understand who
to themselves be distributed geographically, incentivised by juris-
is involved, what they are doing, and how to allocate accountability
dictional differences in cost and regulation. This allows regulatory
between them. Importantly, supply chains are structured by legal
arbitrage, where companies in one jurisdiction exploit legal and
and political economic factors, which must be properly understood,
political economic conditions in other jurisdictions to maximise
as well as technological ones. If governance and accountability
commercial benefit while minimising legal accountability. This
mechanisms are to hold those responsible for developing, deploy-
often involves contracting third-parties (such as Sama AI [95] or
ing, and using AI technologies to account for their workings and
Supahands [102]) to undertake some aspects of production. For
effects, the dynamics of supply chains must be urgently addressed.
example, differences in privacy and data protection laws and labour
protections can lower the legal risk of dataset production activities
like data cleaning, and labelling [44, 69, 83]. Environmental factors ACKNOWLEDGMENTS
like cheap water and energy and lax planning and waste laws can JC and JS are members of the Compliant & Accountable Systems
influence the location of compute and storage [76]. Group, which acknowledges the financial support of UK Research &
While some laws—such as the EU’s data protection law [36] and Innovation (EP/P024394/1, EP/R033501/1, ES/T006315/1), The Alan
AI Act [28] and California’s Consumer Privacy Act [101]—have Turing Institute, and Microsoft (via the Microsoft Cloud Computing
sought extra-territorial effect to address regulatory arbitrage, the Research Centre). MV is supported by the Fondation Botnar.

1195
Understanding accountability in algorithmic supply chains FAccT ’23, June 12–15, 2023, Chicago, IL, USA

REFERENCES [32] Laurence Diver. 2018. Law as a User: Design, Affordance, and the Technological
[1] Ada Lovelace Institute. 2020. Examining the Black Box: Tools for Assessing Mediation of Norms. SCRIPTed: A Journal of Law, Technology & Society (2018).
Algorithmic Systems. [Link] Issue 1.
the-black-box-tools-for-assessing-algorithmic-systems/ [33] EAIDB: The Ethical AI Database . 2023. Market Map.
[2] Amazon. [n. d.]. Mechanical Turk. [Link] [Link]
[3] Amazon. 2021. Artificial Intelligence Services. [Link] [34] Lilian Edwards and Michael Veale. 2017. Slave to the algorithm: Why a right to
machine-learning/ai-services. (Accessed on 04/11/2021). an explanation is probably not the remedy you are looking for. Duke L. & Tech.
[4] Amazon. 2021. Case Study: WeShine. [Link] Rev. 16 (2017), 18.
[5] Mike Ananny and Kate Crawford. 2018. Seeing without knowing: Limitations of [35] Norbert Elias. 1984. What Is Sociology? (revised ed.). Columbia University Press.
the transparency ideal and its application to algorithmic accountability. New Me- [36] European Union. 2016. Regulation (EU) 2016/679 of the European Parliament
dia and Society 20, 3 (2018), 973–989. [Link] and of the Council of 27 April 2016 on the protection of natural persons with
[6] Anthropic. 2023. Anthropic Partners with Google Cloud. regard to the processing of personal data and on the free movement of such
[Link] data, and repealing Directive 95/46/EC (General Data Protection Regulation).
[7] Matthew Arnold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep OJEU L119 (4 May 2016), 1–88.
Mehta, Aleksandra Mojsilović, Ravi Nair, K Natesan Ramamurthy, Alexandra [37] Hugging Face. 2022. The AI community building the future. [Link]
Olteanu, David Piorkowski, et al. 2019. FactSheets: Increasing trust in AI co.
services through supplier’s declarations of conformity. IBM Journal of Research [38] Oscar H. Gandy. 2010. Engaging Rational Discrimination: Exploring Reasons
and Development 63, 4/5 (2019), 6–1. for Placing Regulatory Constraints on Decision Support Systems. Ethics and
[8] Alejandro Barredo Arrieta et al. 2020. Explainable Artificial Intelligence (XAI): Information Technology 12, 1 (March 2010), 29–42. [Link]
Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI. [39] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman
Information Fusion 58 (2020). Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018. Datasheets
[9] Jef Ausloos and Michael Veale. 2020. Researching with Data Rights. Technology for datasets. arXiv preprint arXiv:1803.09010 (2018).
and Regulation 2020, 1 (2020), 136–157. [40] Samuel Gibbs. 2014. Google buys UK artificial intelligence startup Deepmind for
[10] Agathe Balayn and Seda Gürses. 2021. Beyond Debiasing: Regulating AI and Its £400m. [Link]
Inequalities. [Link] uk-artificial-intelligence-startup-deepmind.
[11] Mark Bovens. 1998. The quest for responsibility. Accountability and citizenship in [41] James J Gibson. 2014. The Ecological Approach to Visual Perception (Classic ed.).
complex organisations. Cambridge University Press. Taylor Francis.
[12] Mark Bovens. 2006. Analysing and Assessing Public Accountability. A Concep- [42] Google. 2021. Cloud AI Building Blocks.
tual Framework. EUROGOV, European Governance Papers No. C-06-01 (2006). [Link] (Accessed on
[13] Mark Bovens. 2010. Two Concepts of Accountability: Accountability as a Virtue 04/11/2021).
and as a Mechanism. West European Politics 33, 5 (2010), 946–967. https: [43] Government of Canada. 2020. Algorithmic Impact Assessment.
//[Link]/10.1080/01402382.2010.486119 [44] Mary L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley
[14] Tania Bucher. 2013. Objects of Intense Feeling: The Case of the Twitter API. from Building a New Global Underclass. Houghton Mifflin Harcourt, Boston.
Computational Culture: A Journal of Software Studies 3 (2013). [45] Seda Gürses and Joris van Hoboken. 2017. Privacy After the Agile Turn. In
[15] Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accu- Cambridge Handbook of Consumer Privacy, Jules Polonetsky, Omer Tene, and
racy disparities in commercial gender classification. In Conference on Fairness, Evan Selinger (Eds.).
Accountability, and Transparency. PMLR, 77–91. [46] Kori Hale. 2019. Google & Microsoft Banking On Africa’s AI Labeling Work-
[16] CJEU. 2018. Case C-210/16, Unabhängiges Landeszentrum für Daten- force. [Link]
schutz Schleswig-Holstein v Wirtschaftsakademie Schleswig-Holstein GmbH. banking-on-africas-ai-labeling-workforce.
ECLI:EU:C:2018:388. [47] Bernard E Harcourt. 2007. Against Prediction. University of Chicago Press,
[17] CJEU. 2018. Case C-25/17, Jehovan todistajat. ECLI:EU:C:2018:551. Chicago.
[18] CJEU. 2019. Case C-49/17, Fashion ID GmbH & [Link] v Verbraucherzentrale [48] HireVue. 2022. End-to-End Hiring Experience Platform: Video Interviewing,
NRW eV. ECLI:EU:C:2019:629. Conversational AI & More | HireVue. [Link]
[19] CJEU. 2023. Case C-154/21, Österreichische Post. ECLI:EU:C:2023:3. [49] Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer,
[20] Clearview AI. [n. d.]. [Link] [Link] Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021. Towards ac-
[21] Richard Cloete, Chris Norval, and Jatinder Singh. 2022. Auditable Aug- countability for machine learning datasets: Practices from software engineering
mented/Mixed/Virtual Reality: The Practicalities of Mobile System Transparency. and infrastructure. In Proceedings of the 2021 ACM Conference on Fairness, Ac-
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 5, 4, Article 149 (dec 2022), countability, and Transparency. 560–575.
24 pages. [Link] [50] IBM. 2021. IBM Watson products and solutions. [Link]
[22] Jennifer Cobbe et al. 2019. What lies beneath: Transparency in online service en/watson/products-services.
supply chains. Journal of Cyber Policy 5, 1 (2019). [51] Infermedica. 2022. Call Center Triage. [Link]
[23] Jennifer Cobbe, Michelle Seng Ah Lee, and Jatinder Singh. 2021. Reviewable center-triage.
Automated Decision-Making: A Framework for Accountable Algorithmic Sys- [52] Information Commissioner’s Office and The Alan Turing Institute. 2020. Ex-
tems. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and plaining decisions made with AI.
Transparency. 598–609. [53] Seyyed Ahmad Javadi, Richard Cloete, Jennifer Cobbe, Michelle Seng Ah Lee, and
[24] Jennifer Cobbe and Jatinder Singh. 2021. Artificial Intelligence as a Service: Legal Jatinder Singh. 2020. Monitoring Misuse for Accountable ’Artificial Intelligence as
Responsibilities, Liabilities, and Policy Challenges. Forthcoming in Computer a Service’. Association for Computing Machinery, New York, NY, USA, 300–306.
Law & Security Review (2021). [Link]
[25] Jennifer Cobbe and Jatinder Singh. 2023. Data Protection Doesn’t Work. (2023). [54] Seyyed Ahmad Javadi, Chris Norval, Richard Cloete, and Jatinder Singh. 2021.
[Link] Monitoring AI Services for Misuse. In Proceedings of the 2021 AAAI/ACM Con-
[26] Julie E Cohen. 2012. “Piracy,” “Security,” and Architectures of Control. In ference on AI, Ethics, and Society. 597–607.
Configuring the Networked Self: Law, Code, and the Play of Everyday Practice. [55] Dimitra Kamarinou, Christopher Millard, and Kuan Hon, W. 2015. Privacy in
Yale University Press, New Haven, CT. the Clouds: An Empirical Study of the Terms of Service and Privacy Policies of
[27] Julie E Cohen. 2019. Between Truth and Power: The Legal Constructions of 20 Cloud Service Providers. Queen Mary School of Law Legal Studies Research
Informational Capitalism. Oxford University Press. Paper No. 209/2015 (2015). [Link]
[28] EU Commission et al. 2021. Proposal for a regulation of the European Parliament id=2646447
and of the Council laying down harmonised rules on artificial intelligence [56] Margot Kaminski and Jennifer M Urban. 2021. The Right to Contest AI. Columbia
(Artificial Intelligence Act) and amending certain Union legislative acts. COM Law Review 121, 7 (2021).
(2021) 206 (2021). [57] Ansgar Koene et al. 2019. A governance framework for algorithmic account-
[29] Anamaria Crisan, Margaret Drouhard, Jesse Vig, and Nazneen Rajani. 2022. In- ability and transparency. European Parliamentary Research Service, Panel for the
teractive Model Cards: A Human-Centered Approach to Model Documentation. Future of Science and Technology PE 624.262 (April 2019).
In 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, [58] Blagovesta Kostova, Seda Gürses, and Carmela Troncoso. 2020. Privacy Engi-
Republic of Korea) (FAccT ’22). Association for Computing Machinery, New neering Meets Software Engineering. On the Challenges of Engineering Privacy
York, NY, USA, 427–439. [Link] ByDesign. [Link] arXiv:2007.08613 [cs]
[30] Jenny Davis. 2020. How Artifacts Afford: The Power and Politics of Everyday [59] PM Krafft, Meg Young, Michael Katell, Jennifer E Lee, Shankar Narayan, Micah
Things. MIT Press. Epstein, Dharma Dailey, Bernease Herman, Aaron Tam, Vivian Guetler, et al.
[31] Demos et al. 2020. Algorithm Inspection and Regulatory Access. 2021. An Action-Oriented AI Policy Toolkit for Technology Audits by Com-
munity Advocates and Activists. In Proceedings of the 2021 ACM Conference on
Fairness, Accountability, and Transparency. 772–781.

1196
FAccT ’23, June 12–15, 2023, Chicago, IL, USA Cobbe, Veale and Singh

[60] Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Rei- [83] Billy Perrigo. [n. d.]. OpenAI Used Kenyan Workers on Less Than $2 Per Hour
denberg, David G. Robinson, and Harlan Yu. 2017. Accountable algorithms. to Make ChatGPT Less Toxic. TIME ([n. d.]). [Link]
University of Pennsylvania Law Review 165, 3 (Feb. 2017), 633–705. chatgpt-kenya-workers/
[61] Joshua A. Kroll. 2020. Accountability in Computer Systems. In The Oxford [84] Billy Perrigo. 2023. Exclusive: OpenAI Used Kenyan Workers on Less Than
Handbook of Ethics of AI, Markus D. Dubber, Frank Pasquale, and Sunit Das (Eds.). $2 Per Hour to Make ChatGPT Less Toxic. [Link]
Oxford University Press, 0. [Link] chatgpt-kenya-workers.
013.10 [85] Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022. Data
[62] Joshua A. Kroll. 2021. Outlining Traceability: A Principle for Operationalizing Cards: Purposeful and Transparent Dataset Documentation for Responsible AI.
Accountability in Computing Systems. In Proceedings of the 2021 ACM Conference In Conference on Fairness, Accountability, and Transparency.
on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). [86] Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022. Data Cards:
Association for Computing Machinery, New York, NY, USA, 758–771. https: Purposeful and Transparent Dataset Documentation for Responsible AI (FAccT
//[Link]/10.1145/3442188.3445937 ’22). Association for Computing Machinery, New York, NY, USA, 1776–1826.
[63] Michelle Seng Ah Lee and Jatinder Singh. 2021. Risk identification questionnaire [Link]
for unintended bias in machine learning development lifecycle. Available at [87] Pymetrics. 2021. Talent Matching Platform. [Link]
SSRN (2021). [88] Emilee Rader, Kelley Cotter, and Janghee Cho. 2018. Explanations as mecha-
[64] Kornel Lewicki, Michelle Seng Ah Lee, Jennifer Cobbe, and Jatinder Singh. 2023. nisms for supporting algorithmic transparency. In Proceedings of the 2018 CHI
Out of Context: Algorithmic Fairness in "Artificial Intelligence as a Service". conference on human factors in computing systems. 1–13.
arXiv:2302.01448 (2023). [Link] [89] Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell,
[65] Lunit. 2022. AI will be the new standard of care. By Lunit. [Link] Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker
io/en. Barnes. 2020. Closing the AI accountability gap: defining an end-to-end frame-
[66] David Lyon (Ed.). 2003. Surveillance as Social Sorting: Privacy, Risk, and Digital work for internal algorithmic auditing. In Proceedings of the 2020 Conference on
Discrimination. Routledge, London ; New York. Fairness, Accountability, and Transparency. 33–44.
[67] Michael A Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach. [90] Chris Reed and Laura Edgar. 2021. Consumer Protection in the Cloud. In Cloud
2020. Co-designing checklists to understand organizational challenges and Computing Law (second ed.), Christopher Millard (Ed.). Oxford University Press,
opportunities around fairness in ai. In Proceedings of the 2020 CHI Conference on Oxford, 218–254. [Link]
Human Factors in Computing Systems. 1–14. [91] Joel R. Reidenberg. 1998. Lex Informatica: The Formulation of Information
[68] René Mahieu, Joris van Hoboken, and Hadi Asghari. 2019. Responsibility for Policy Rules through Technology. Texas Law Review 76, 3 (1998), 553–594.
Data Protection in a Networked World: On the Queston of the Controller, [92] Dillon Reisman et al. 2018. Algorithmic Impact Assessments: A practical frame-
“Effective and Complete Protection” and its Application to Data Access Rights in work for public agency accountability (AI Now).
Europe. Journal of Intellectual Property, Information Technology and E-Commerce [93] Felix Richter. 2022. Amazon, Microsoft & Google Dominate Cloud Mar-
Law 10 (2019). Issue 1. ket. [Link]
[69] Kira J. M. Matus and Michael Veale. 2022. Certification Systems for Machine cloud-infrastructure-service-providers.
Learning: Lessons from Sustainability. Regulation & Governance 16, 1 (2022), [94] Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens,
177–196. [Link] Ari Anisfeld, Kit T Rodolfa, and Rayid Ghani. 2018. Aequitas: A bias and fairness
[70] Danaë Metaxa, Joon Sung Park, Ronald E Robertson, Karrie Karahalios, Christo audit toolkit. arXiv preprint arXiv:1811.05577 (2018).
Wilson, Jeff Hancock, Christian Sandvig, et al. 2021. Auditing algorithms: [95] Sama AI. [n. d.]. [Link] [Link]
Understanding algorithmic systems from the outside in. Foundations and Trends® [96] Nick Seaver. 2013. Knowing Algorithms. paper presented at Media in Transition
in Human–Computer Interaction 14, 4 (2021), 272–344. 8 (2013).
[71] Johan David Michels, Christopher Millard, and Felicity Turton. 2021. Standard [97] Andrew D. Selbst, Danah Boyd, Sorelle A. Friedler, Suresh Venkatasubramanian,
Contracts for Cloud Services. In Cloud Computing Law (second ed.), Christopher and Janet Vertesi. 2019. Fairness and Abstraction in Sociotechnical Systems.
Millard (Ed.). Oxford University Press, Oxford, 49–99. [Link] In Proceedings of the Conference on Fairness, Accountability, and Transparency
oso/9780198716662.003.0003 (FAT* ’19). ACM, New York, NY, USA, 59–68. [Link]
[72] Microsoft. 2021. Cognitive Services. [Link] [98] Jatinder Singh et al. 2016. Responsibility & Machine Learning: Part of a Process.
cognitive-services. (Accessed on 04/11/2021). [99] Jatinder Singh, Jennifer Cobbe, and Chris Norval. 2019. Decision Provenance:
[73] Microsoft. 2023. Microsoft and OpenAI extend partnership. [Link] Harnessing Data Flow for Accountable Systems. IEEE Access 7 (2019), 6562–
[Link]/blog/2023/01/23/microsoftandopenaiextendpartnership. 6574.
[74] Christopher Millard. 2021. Cloud Computing Law (2 ed.). Oxford University [100] Swami Sivasubramanian. 2023. Announcing New Tools for Building with Gener-
Press. ative AI on AWS. [Link]
[75] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasser- new-tools-for-building-with-generative-ai-on-aws/.
man, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. [101] State of Califorina. 2018. The California Consumer Privacy Act (CCPA).
2019. Model cards for model reporting. In Conference on Fairness, Accountability, [102] Supahands. [n. d.]. [Link] [Link]
and Transparency. 220–229. [103] The Economist. 2022. Artificial Intelligence Is Permeating Business at
[76] Vincent Mosco. 2014. Dark Clouds. In To the Cloud: Big Data in a Turbulent Last. [Link]
World. Paradigm, Boulder, CO. is-permeating-business-at-last
[77] Helen Nissenbaum. 1996. Accountability in a Computerized Society. Science [104] Dennis F. Thompson. 1980. Moral Responsibility of Public Officials: The Problem
and Engineering Ethics (1996). of Many Hands. American Political Science Review 74, 4 (1980), 905–916. https:
[78] Donald Norman. 1988. The Design of Everyday Things. Basic Books. //[Link]/10.2307/1954312
[79] Chris Norval, Jennifer Cobbe, and Jatinder Singh. 2020. Towards an accountable [105] Felicity Turton, Dimitra Kamarinou, Johan David Michels, and Christopher
Internet of Things: A call for ‘reviewability’. In Privacy by Design for the Internet Millard. 2021. Privacy in the Clouds, Revisited: An Analysis of the Privacy
of Things: Building Accountability and Security. The Institution of Engineering Policies of 40 Cloud Computing Services. Queen Mary Law Research Paper No.
and Technology. 354/2021 (2021). [Link]
[80] Chris Norval, Kristin Cornelius, Jennifer Cobbe, and Jatinder Singh. 2022. Dis- [106] Michael Veale, Midas Nouwens, and Cristiana Santos. 2022. Impossible Asks: Can
closure by Design: Designing Information Disclosures to Support Meaning- the Transparency and Consent Framework Ever Authorise Real-Time Bidding
ful Transparency and Accountability. In 2022 ACM Conference on Fairness, After the Belgian DPA Decision? Technology and Regulation 2022 (2022), 12–22.
Accountability, and Transparency (Seoul, Republic of Korea) (FAccT ’22). As- [Link]
sociation for Computing Machinery, New York, NY, USA, 679–690. https: [107] Michael Veale and Frederik Zuiderveen Borgesius. 2021. Demystifying the Draft
//[Link]/10.1145/3531146.3533133 EU Artificial Intelligence Act. Computer Law Review International 22, 4 (2021),
[81] Frank Pasquale. 2019. The Second Wave of Algorithmic Accountability. Law 97–112. [Link]
and Political Economy Project (2019). [Link] [108] Peter-Paul Verbeek. 2005. What Things Do: Philosophical Reflections on Technol-
wave-of-algorithmic-accountability/ ogy, Agency, and Design. Penn State Press.
[82] Samir Passi and Solon Barocas. 2019. Problem Formulation and Fairness. In [109] Maranke Wieringa. 2020. What to account for when accounting for algorithms:
Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* a systematic literature review on algorithmic accountability. In Proceedings of
’19). ACM, New York, NY, USA, 39–48. [Link] the 2020 conference on fairness, accountability, and transparency. 1–18.
[110] Rebecca Williams et al. 2022. From transparency to accountability of intelligent
systems: Moving beyond aspirations. Data & Policy 4 (2022).

1197