1

Introduction to Data Privacy

1.1 Introduction
Organizations dealing with banking, insurance, retail, healthcare, and
manufacturing across the globe collect large amounts of data about their
customers. This is a valuable asset to the organizations as these data can
be mined to extract a lot of insights about their customers. For example,
mining these data can throw light on customers’ spending/buying, credit
card usage, and investment patterns and health issues, to name a few. This
information is used by companies to provide value-added services to their
customers, which in turn results in higher revenue and profit. But these data
might contain customers’ personal identification information, and when in
the hands of a data snooper, they can be exploited.
Large companies across the globe outsource their IT and business pro-
cess work to service providers in countries like India, China, Brazil, etc. The
outsourced work may involve application maintenance and development,
testing, data mining/analysis, statistical analysis, etc. Business applications
contain sensitive information, such as personal or financial and health-
related data. Sharing such data can potentially violate individual privacy and
lead to financial loss to the company. Serious concerns have been expressed
by general public about exposing person-specific information. The issue of
data leakage, either intentional or accidental exposure of sensitive informa-
tion, is becoming a major security issue.
An IDC survey [19] claims that data leakage is the number one threat,
ranked higher than viruses, Trojan horses, and worms. To address the privacy
of an individual’s data, governments across the globe have mandated regula-
tions that companies have to adhere to: HIPAA (Health Insurance Portability
and Accountability Act) in the United States, FIPPA (Freedom of Information
and Protection of Privacy Act) in Canada, Sarbanes–Oxley Act, Video Privacy
Protection, U.S. Declaration of Human Rights, and the EU’s Data Protection
Directive are just a few examples. Companies need to look into methods and
tools to anonymize sensitive data. Data anonymization techniques have been
the subject of intense investigation in recent years for many kinds of struc-
tured data, including tabular, transactional data, and graph data.

1
2 Data Privacy: Principles and Practice

In Chapter 2, Static Data Anonymization, we discuss relational data, also

known as multidimensional data, which are the most widely found data
structure in enterprises currently. This chapter focuses on privacy pres-
ervation methods for multidimensional data. Multidimensional data are
simple in structure, and a rich set of data protection algorithms, such as
randomization, generalization, k-anonymization, l-diversity, and t-closeness,
is described.
Anonymization techniques for multidimensional data are simple in
structure and very commonly found across enterprises. Apart from mul-
tidimensional data, other types of data structures, such as graph, longi-
tudinal data, sparse high-dimensional transaction data, time series data,
spatiotemporal data, semistructured XML data, and big data, are also pres-
ent across enterprises. These data structures are complex, contain sensitive
customer information, and should therefore be protected. There are unique
challenges in designing anonymization techniques for these complex data
structures, though. Anonymization techniques used for the protection of
multidimensional data are not directly applicable to these complex data
structures. Chapter 3 discusses some of the anonymization techniques for
complex data structures.
Any anonymization design is a function of many inputs. One of the impor-
tant inputs is, “who are we protecting these data from?” The answer builds a
profile of adversaries who are expected to attack the data. Chapter 4 explains
various profiles of adversaries, their techniques, and what safeguards can be
implemented against such threats.
Data mining is the first application of data privacy that we discuss. We
explore two areas of mining: association rule mining and clustering. Each of
these areas works with the goal of knowledge discovery. However, ensuring
privacy is important for users/customers/patients to willingly share their
data for analysis. In Chapter 5, we explore a few prominent privacy preserva-
tion algorithms and conclude with a discussion on their impact on the utility
of data.
The second application of privacy, test data, is increasingly becoming an
area for privacy preservation. High-quality testing requires high-quality test
data. Outsourcing of testing has brought data privacy concerns to the fore.
Hence, in Chapter 6, we discuss the need for privacy and current trends and
list appropriate algorithms for each class of data. Measuring the utility of
test data and also ascertaining the overall quality of test data are important
to understand if a balance between privacy and utility has been achieved.
Finally, the chapter also highlights some problems with the current anony-
mization options available.
In data mining and testing, there are times when a need arises to create
external supplementary data. Classifiers require training data, which are
not available initially. While testing too, special processes like error data
handling, performance benchmarking, etc., require data not available in
the original source data. Synthetic data generation solves both these issues.
Introduction to Data Privacy 3

In Chapter 7, we visit each class of personally identifiable information (PII)

and explore the techniques available to generate synthetic data. The safety
aspects of synthetic data are also covered in this chapter.
Run-time preservation of privacy is needed in cases where changes to the
data itself cannot be made. Another challenge is that different roles of users
require different levels of protection. In such cases, tokenization is a good
solution. Tokens preserve the formats of sensitive data, which make them
look just like the original data. Chapter 8 covers use cases, implementation
and examples.
Chapter 9, the final chapter, explores the compliance side of data privacy.
Most privacy implementations are direct results of compliance mandates at
the regional or organizational level. In this chapter, we explain the rules and
definitions in some of the relevant privacy regulations. Most of this chap-
ter is dedicated to HIPAA, which is the definitive privacy law in the United
States for healthcare data.
Appendix A lists the principles of anonymization that are referred
throughout this book. These principles are applicable across domains, thus
providing a concrete guideline for data privacy implementations.
Appendix B (PPTDM Manifesto) summarizes the best practices to be
applied while ­preserving privacy in a test data setting.

1.2 What Is Data Privacy and Why Is It Important?

Thousands of ambulance service staff and housing benefits claimants have
had their personal information accidently leaked in the latest UK data breach
blunder (January 4, 2014; news in [Link]/news/
thousands-of-personal/-details.)
Natural healthcare chain Community Health Systems (CHS) says that
about 4.5 million pieces of “non-medical patient identification data related
to our physician practice” have been stolen …. August 18, 2014/News
[Link]/news/45-million-records-stolen-from/
NASDAQ-listed outsourcing firm EXL Service has lost a key client due to
the breach of confidential client data by some of its employees.

• Economic Times (India) on November 6, 2013

There are numerous such incidents where customers’ confidential personal

information has been attacked by or lost to a data snooper. When such
untoward incidents occur, organizations face legal suits, financial loss, loss
of image, and, importantly, the loss of their customers.
There are many stakeholders of data privacy in an organization; these are
shown in Figure 1.1. Let us define these stakeholders.
4 Data Privacy: Principles and Practice

Government—Compliance
Customer/Record owner and regulations

Adversary/Data snooper

Company—Banks, insurance,
healthcare, or retail
Data analysts

Data anonymizer
Data Data
Pro Data Data s es
du
cti Data Data at aba
on dd
dat
aba m ize Testers
ses o ny
An

Business operations
employee

FIGURE 1.1
Data privacy—stakeholders in the organization.

Company: Any organization like a bank, an insurance company, or an

e-commerce, retail, healthcare, or social networking company that holds
large amounts of customer-specific data. They are the custodians of cus-
tomer data, which are considered very sensitive, and have the responsibility
of protecting the data at all costs. Any loss of these sensitive data will result
in the company facing legal suits, financial penalties, and loss of reputation.
Customer/record owner: An organization’s customer could be an individual or
another organization who share their data with the company. For example,
an individual shares his personal information, also known as PII, such as
his name, address, gender, date of birth, phone numbers, e-mail address,
and income with a bank. PII is considered sensitive as any disclosure or loss
could lead to undesired identification of the customer or record owner. It has
been shown that gender, age and zip code are sufficient to identify a large
population of people in the United States.
Government: Government defines what data protection regulations that
the company should comply with. Examples of such regulations are the
HIPPA Act, the EU Data Protection Act, and the Swiss Data Protection Act.
It is mandatory for companies to follow government regulations on data
protection.
Introduction to Data Privacy 5

Data anonymizer: A person who anonymizes and provides data for analysis
or as test data.
Data analyst: This person uses the anonymized data to carry out data m­ ining
activities like prediction, knowledge discovery, and so on. Following govern-
ment regulations, such as the Data Moratorium Act, only anonymized data
can be used for data mining. Therefore, it is important that the provisioned
data support data mining functionalities.
Tester: Outsourcing of software testing is common among many companies.
High-quality testing requires high-quality test data, which is present in pro-
duction systems and contains customer-sensitive information. In order to test
the software system, the tester needs data to be extracted from production
systems, anonymized, and provisioned for testing. Since test data contain
customer-sensitive data, it is mandatory to adhere to regulatory compliance
in that region/country.
Business operations employee: Data analysts and software testers use anony-
mized data that are at rest or static, whereas business operations employees
access production data because they need to support customer’s business
requirements. Business operations are generally outsourced to BPO (busi-
ness process outsourcing) companies. In this case too, there is a requirement
to protect customer-sensitive data but as this operation is carried out during
run-time, a different set of data protection techniques are required to protect
data from business operations employees.
Adversary/data snooper: Data are precious and their theft is very common.
An adversary can be internal or external to the organization. The anonymiza-
tion design should be such that it can thwart an adversary’s effort to identify
a record owner in the database.
Companies spend millions of dollars to protect the privacy of customer
data. Why is it so important? What constitutes personal information? Personal
information consists of name, identifiers like social security number, geo-
graphic and demographic information, and general sensitive information, for
example, financial status, health issues, shopping patterns, and location data.
Loss of this information means loss of privacy—one’s right to freedom from
intrusion by others. As we will see, protecting one’s privacy is nontrivial.

1.2.1 Protecting Sensitive Data

“I know where you were yesterday!” Google knows your location when
you use Google Maps. Google maps can track you wherever you go
when you use it on a smart phone. Mobile companies know your exact loca-
tion when you use a mobile phone. You have no place to hide. You have lost
your privacy. This is the flip side of using devices like smart phones, Global
positioning systems (GPS), and radio frequency identification (RFID). Why
should others know where you were yesterday? Similarly, why should
6 Data Privacy: Principles and Practice

others know your health issues or financial status? All these are sensitive
data and should be well protected as they could fall into the wrong hands
and be exploited. Let us look at a sample bank customer and an account
table. The customer table taken as such has nothing confidential as most of
the information contained in it is also available in the public voters data-
base and on social networking sites like Facebook. Sensitiveness comes in
when the ­customer table is combined with an accounts table. A logical rep-
resentation of Tables 1.1 and 1.2 is shown in Table 1.3.
Data D in the tables contains four disjointed data sets:

1. Explicit identifiers (EI): Attributes that identify a customer (also called

record owner) directly. These include attributes like social security
number (SSN), insurance ID, and name.
2. Quasi-identifiers (QI): Attributes that include geographic and demo-
graphic information, phone numbers, and e-mail IDs. Quasi-
identifiers are also defined as those attributes that are publicly
available, for example, a voters database.
3. Sensitive data (SD): Attributes that contain confidential information
about the record owner, such as health issues, financial status, and
salary, which cannot be compromised at any cost.
4. Nonsensitive data (NSD): Data that are not sensitive for the given context.

TABLE 1.1
Customer Table
Explicit Identifiers Quasi-Identifiers
ID First Name DOB Gender Address Zip Code Phone
1 Ravi 1970 Male Fourth Street 66001 92345-67567
2 Hari 1975 Male Queen Street 66011 98769-66610
3 John 1978 Male Penn Street 66003 97867-00055
4 Amy 1980 Female Ben Street 66066 98123-98765

TABLE 1.2
Account Table
Sensitive Data
Account Account Account Credit Nonsensitive
ID Number Type Balance Limit Data
1 12345 Savings 10,000 20,000
2 23456 Checking 5,000 15,000
3 45678 Savings 15,000 30,000
4 76543 Savings 17,000 25,000
Introduction to Data Privacy 7

TABLE 1.3
Logical Representation of Customer and Account Tables
Explicit
Identifiers Quasi-Identifiers Sensitive Data
Zip Account Account Account Credit
ID Name DOB Gender Address Code Number Type Balance Limit
1 Ravi 1970 Male Fourth 66001 12345 Savings 10,000 20,000
Street
2 Hari 1975 Male Queen 66011 23456 Checking 5,000 15,000
Street
3 John 1978 Male Penn Street 66003 45678 Savings 15,000 30,000
4 Amy 1980 Female Ben Street 66066 76543 Savings 17,000 25,000

The first two data sets, the EI and QI, uniquely identify a record owner and
when combined with sensitive data become sensitive or confidential. The
data set D is considered as a matrix of m rows and n columns. Matrix D is a
vector space where each row and column is a vector

D = [DEI] [DQI] [DSD] (1.1)

Each of the data sets, EI, QI, and SD, are matrices with m rows and i, j, and
k columns, respectively. We need to keep an eye on the index j (represent-
ing QI), which plays a major role in keeping the data confidential.
Apart from assuring their customers’ privacy, organizations also have
to comply with various regulations in that region/country, as mentioned
earlier. Most countries have strong privacy laws to protect citizens’ per-
sonal data. Organizations that fail to protect the privacy of their customers
or do not comply with the regulations face stiff financial penalties, loss of
reputation, loss of customers, and legal issues. This is the primary reason
organizations pay so much attention to data privacy. They find themselves
in a Catch-22 as they have huge amounts of customer data, and there is a
compelling need to share these data with specialized data analysis com-
panies. Most often, data protection techniques, such as cryptography and
anonymization, are used prior to sharing data. In this book, we focus only
on anonymization.
Anonymization is a process of logically separating the identifying
information (PII) from sensitive data. Referring to Table 1.3, the anony-
mization approach ensures that EI and QI are logically separated from
SD. As a result, an adversary will not be able to easily identify the record
owner from his sensitive data. This is easier said than done. How to
effectively anonymize the data? This is the question we explore through-
out this book.
8 Data Privacy: Principles and Practice

1.2.2 Privacy and Anonymity: Two Sides of the Same Coin

This brings up the interesting definition of privacy and anonymity. According
to Skopek [1], under the condition of privacy, we have knowledge of a person’s
identity, but not of an associated personal fact, whereas under the condition
of anonymity, we have knowledge of a personal fact, but not of the associated
person’s identity. In this sense, privacy and anonymity are flip sides of the
same coin. Tables 1.4 and 1.5 illustrate the fundamental d ­ ifferences between
privacy and anonymity.
There is a subtle difference between privacy and anonymity. The word pri-
vacy is also used in a generic way to mean anonymity, and there are specific
use cases for both of them. Table 1.4 illustrates an anonymized table where
PII is protected and sensitive data are left in their original form. Sensitive
data should be in original form so that the data can be used to mine useful
knowledge.
Anonymization is a two-step process: data masking and de-identification.
Data masking is a technique applied to systematically substitute, suppress,
or scramble data that call out an individual, such as names, IDs, account
numbers, SSNs, etc. Masking techniques are simple techniques that perturb
original data. De-identification is applied on QI fields. QI fields such as date

TABLE 1.4
Example of Anonymity
Personal Identity Sensitive Data
Zip Account Account Account Credit
SSN Name DOB Gender Address Code Number Type Balance Limit
X X X X X X
X X X X X X
X X X X X X
X X X X X X
Note: X, identity is protected.

TABLE 1.5
Example of Privacy
Personal Identity Sensitive Data
Zip Account Account Account Credit
SSN Name DOB Gender Address Code Number Type Balance Limit
X X X X
X X X X
X X X X
X X X X
Note: X, sensitive data are protected.
Introduction to Data Privacy 9

of birth, gender, and zip code have the capacity to uniquely identify indi-
viduals. Combine that with SD, such as income, and a Warren Buffet or Bill
Gates is easily identified in the data set. By de-identifying, the values of QI
are modified carefully so that the relationship is till maintained by identities
cannot be inferred.
In Equation 1.1, the original data set is D which is anonymized, resulting
in data set D′ = T(D) or T([DEI][DQI][DSD]), where T is the transformation func-
tion. As a first step in the anonymization process, EI is completely masked
and no longer relevant in D′. As mentioned earlier, no transformation is
applied on SD and it is left in its original form. This results in D′ = T([DQI]),
which means that transformation is applied only on QI as EI is masked
and not considered as part of D′ and SD is left in its original form. D′ can
be shared as QI is transformed and SD is in its original form but it is very
­difficult to identify the record owner. Coming up with the transformation
function is key to the success of anonymization design and this is nontrivial.
We spend a lot of time on anonymization design, which is generally applied
on static data or data at rest.
The other scenario is protecting SD, as shown in Table 1.5, which is applied
on data in motion. The implementation of this is also very challenging.
It is dichotomous as organizations take utmost care in protecting the
privacy of their customers’ data, but the same customers provide a whole
lot of personal information when they register on social network sites like
Facebook (of course, many of the fields are not mandatory but most peo-
ple do provide sufficient personal information), including address, phone
numbers, date of birth (DOB), details of education and qualification, work
experience, etc. Sweeney [2] reports that zip code, DOB, and gender are
­sufficient to uniquely identify 83% of population in the United States. With
the amount of PII available on social networking sites, a data snooper with
some background knowledge could use the publicly available information
to re-identify customers in corporate databases.
In the era of social networks, de-identification becomes highly challenging.

1.3 Use Cases: Need for Sharing Data

Organizations tend to share customer data as there is much insight to be
gained from customer-sensitive data. For example, a healthcare provider’s
database could contain how patients have reacted to a particular drug or
treatment. This information would be useful to a pharmaceutical company.
However, these sensitive data cannot be shared or released due to legal, finan-
cial, compliance, and moral issues. But for the benefit of the organization and
the customer, there is a need to share these data responsibly, which means
the data are shared without revealing the PII of the customer. Figure 1.2 sets
14 Data Privacy: Principles and Practice

But access to a customer’s trade account during these processes would expose
a lot of sensitive information and this is not acceptable to the customer,
and the regulation in the country will not permit such access. Therefore, these
data have to be protected. But the question is how and what data are to be
protected? A technique known as tokenization is used here wherein the sen-
sitive data that should not be seen by the BPO employee are replaced with
a token. This token has no relationship with the original data, and outside
the context of the application the token has no meaning at all. All these are
executed during run-time. Like privacy preserving data mining and test data
management, protection of sensitive data is a subject in itself and is covered
in depth in this book.

1.4 Methods of Protecting Data

One of the most daunting tasks in information security is protecting sensitive
data in enterprise applications, which are often complex and distributed. What
methods are available to protect sensitive data? Some of the methods avail-
able are cryptography, anonymization, and tokenization, which are briefly dis-
cussed in this section, and a detailed coverage is provided in the other chapters
in the book. Of course, there are other one-way functions like hashing.
Cryptographic techniques are probably one of the oldest known tech-
niques for data protection. When done right, they are probably one of the
safest techniques to protect data in motion and at rest. Encrypted data have
high protection, but are not readable, so how can we use such data? Another
issue associated with cryptography is key management. Any compromise
of key means complete loss of privacy. For the use cases discussed in this
book, cryptographic techniques are not used widely. Of course, there are
techniques like secure multiparty computation (MPC) and zero-knowledge
proof (ZKP), which are discussed in detail in Duan and Canny [3].
Anonymization is a set of techniques used to modify the original data in
such a manner that it does not resemble the original value but maintains the
semantics and syntax. Regulatory compliance and ethical issues drive the
need for anonymization. The intent is that anonymized data can be shared
freely with other parties, who can perform their own analysis on the data.
Anonymization is an optimization problem, in that when the original data are
modified they lose some of its utility. But modification of the data is required
to protect it. An anonymization design is a balancing act between data pri-
vacy and utility. Privacy goals are set by the data owners, and utility goals are
set by data users. Now, is it really possible to optimally achieve this balance
between privacy and utility? We will explore this throughout this book.
Tokenization is a data protection technique that has been extensively used in
the credit card industry but is currently being adopted in other domains as well.
Introduction to Data Privacy 15

Tokenization is a technique that replaces the original sensitive data with non-
sensitive placeholders referred to as tokens. The fundamental difference
between tokenization and the other techniques is that in tokenization, the origi-
nal data are completely replaced by a surrogate that has no connection to the
original data. Tokens have the same format as the original data. As tokens are
not derived from the original data, they exhibit very powerful data protection
features. Another interesting point of tokens is, although the token is usable
within its native application environment, it is completely useless elsewhere.
Therefore, tokenization is ideal to protect sensitive identifying information.

1.5 Importance of Balancing Data Privacy and Utility

In the introductory section, we looked at how an enterprise provisions data
for purposes like application testing, knowledge discovery, and data analy-
sis. We also emphasized the importance of privacy preservation of customer
data before publishing them. Privacy preservation should also ensure util-
ity of data. In other words, the provisioned data should protect the indi-
vidual’s privacy and at the same time ensure that the anonymized data are
useful for knowledge discovery. By anonymizing the data, EI are completely
masked out, QI is de-identified by applying a transformation function, and
SD is left in its original form. There is a strong correlation between QI and SD
fields. So, as part of privacy preservation, this correlation between QI fields
and SD fields should not be lost. If the correlation is lost, then the resulting
data set is not useful for any purpose.
As a transformation function is applied on QI, it is obvious that the cor-
relation between QI fields and SD fields is affected or weakened, and this
indicates how useful the transformed data are for the given purpose. Let
us take an example from the healthcare domain to illustrate this important
relationship between privacy and utility. HIPAA states that if any of the data
elements are associated with health information, it makes that information
personally identifiable. HIPAA defines 18 attributes as PII that include name,
SSN, geographic information, demographic information telephone number,
admission date, etc. [4]. Therefore, in any privacy preserving data analysis of
health data, it should be ensured that any of these 18 attributes, if present,
should be completely anonymized. If so much information is stripped off,
then how can the remaining data be useful for the analysis? Let us take an
example of a patient getting admitted to a hospital. According to the HIPAA
privacy rules, the admission date is part of the patient’s PII and therefore
should be anonymized. The healthcare provider can share the patient’s
medical data to external partners for the analysis, but it will be impossible to
analyze the efficacy of the treatment as the date of admission is anonymized
as per HIPAA privacy laws. HIPAA’s intention is to protect patient privacy,
16 Data Privacy: Principles and Practice

but it impacts medical research in the process. Therefore, it is extremely

important to ensure the utility of the data while preserving privacy. In other
words, there needs to be a balance between privacy and utility of anony-
mized data. Figure 1.5 provides a map of privacy versus utility.
In the previous section, we looked at different mechanisms to protect data.
Cryptographic mechanism provides low utility (0) and high privacy (1) when
data are encrypted and it provides high utility (1) and low privacy (0) when
data are decrypted. The privacy or utility in a cryptographic mechanism is
either black (0) or white (1), whereas in anonymization methods, it is “shades
of gray,” meaning that is possible to control the levels of privacy or utility.
Anonymization can be viewed as constrained ­optimization—produce a data set
with smallest distortion that also satisfies the given set of privacy requirements.
But how do you balance the two contrasting features—privacy and utility?
Anonymized data are utilized in many areas of an organization like data min-
ing, analysis, or creating test data. An important point to remember here is each
type of requirement or analysis warrants a different anonymization design.
This means that there is no single privacy versus utility measure. To understand
privacy versus utility trade-off, let us take the original data given in Table 1.6.

1. Original data table with no privacy but high utility

2. High correlation between QI and SD (attributes fields)

Op
t
1 Op imum
tim
um priva
uti c y,
lity
Privacy

0
0 Utility 1

FIGURE 1.5
Privacy versus utility map.

TABLE 1.6
Original Table with Strong Correlation between QI and SD
Name Zip Code Gender Income
Chen 56001 Male 25K
Jenny 56015 Female 8K
Alice 56001 Female 30K
Ram 56011 Male 5K
Introduction to Data Privacy 17

Table 1.6 shows four individuals. Although many rows have not been shown
here, let us assume that the ZIP CODE and INCOME are correlated, in that
the ZIP CODE 56001 primarily consists of high-income individuals. Table 1.7
is a modified version of Table 1.6. Let us not worry about the techniques used
to anonymize data, but focus just on the results. We can see that the names
have been changed, the original ZIP CODES have been replaced with differ-
ent values and INCOME values are unchanged.
Let us assess gains and losses for this anonymization design.
Privacy gain: Names are substituted (hence protected), financial standing is
not attributed to another zip code, and geographical location is anonymized.
Utility loss: Gender information is preserved, names are substituted while pre-
serving demographic clues, correlation is preserved but the zip code is different.
Another design can have just “XXXX” for all names, 56001 for all zip codes,
and “Male” for all gender values. We can agree that this anonymization
design scores well in terms of privacy, but utility is pathetic. Privacy gain:
Names are completely suppressed, financial standing cannot be inferred,
and geographical location is not compromised. Utility loss: Presence of
females in the population, meaningless names lose demographic clues, flat
value of zip code annuls the correlation.
This shows that anonymization design drives the extent of privacy and
utility, which are always opposed to each other. The two designs also show
that privacy or utility need not be 0 and 1 as in encryption; rather, both are
shades of gray as stated earlier. A good design can achieve a balance between
them and achieve both goals to a reasonable extent.
One way to quantify privacy is on the basis of how much information an
adversary can obtain about the SD of an individual from different dimen-
sions in the data set [5–8]. These references state that SD fields can be iden-
tified (or estimated/deduced) using QI fields. This is a very simple way
to quantify privacy. In fact, this model does not capture many important
dimensions, such as background knowledge of the adversary, adversary’s
knowledge of some of the sensitive data, the complexity of the data structure,
etc. We discuss this in sufficient detail in Chapter 4.
The utility loss of a particular anonymization technique is measured
against the utility provided by the original data set. A measure of utility

TABLE 1.7
Anonymized Table with Generalized Values—Correlation
between QI and SD Is Broken
Name Zip Code Gender Income
Yang 56000 Male 25K
Emma 56010 Female 8K
Olivia 56000 Female 30K
Krishna 56010 Male 5K
18 Data Privacy: Principles and Practice

is also the correlation between QI and SD preserved in the anonymized

data. There are many anonymization techniques in use today, which can be
broadly classified into perturbative and nonperturbative techniques. Each
of these techniques provides its own privacy versus utility model. The core
goals of these anonymization techniques are (1) to prevent an adversary
from identifying SD fields and (2) to ensure minimal utility loss in the ano-
nymized data set by ensuring high correlation between the QI and SD fields.
This is easier said than done. These are extremely difficult goals to meet.
To address this complex set of problem patterns, we have defined a rich set
of anonymization design principles in Appendix A.

1.5.1 Measuring Privacy of Anonymized Data

Given a data set D, a data anonymizer can create different anonymized
data sets D1′, D2′,…, Dn′ based on different anonymization algorithm com-
binations for each attribute. Each of these anonymized data sets will have
different privacy versus utility trade-offs. Privacy is a relative measure.
This means that the privacy of D1′ is measured against another anony-
mized data set D2′. There are multiple ways to measure the difference in
privacy. These approaches are broadly classified into statistical and proba-
bilistic methods. Some statistical approaches measure privacy in terms of
the difference or variation in perturbed variables. The larger the variance,
the better the privacy of the perturbed data. This technique is generally
used for statistical databases.
Probabilistic methods measure privacy loss when an adversary has knowl-
edge of the distribution of the data in the original data set and background
information about some tuples in the data set. For example, consider the
simple example in Table 1.8.
Bob is the adversary and has some background information about Alice as
she is his neighbor. Bob knows that Alice smokes heavily but does not really
know what disease she is suffering from. However, he has knowledge about
the distribution of the sensitive fields in a table containing medical records
of a hospital that he has noticed Alice visiting. Bob then uses the knowledge
of the distribution of SD fields and background information about Alice to
identify her illness, which is cancer.

TABLE 1.8
Background Knowledge of the Adversary
about the Distribution of SD Fields
Name Zip Code Gender Disease
John Smith 46001 Male Hypertension
Tom Henry 46005 Male Gastritis
Alice Williams 46001 Female Cancer
Little Wood 46011 Male Asthma
Introduction to Data Privacy 19

1.5.2 Measuring Utility of Anonymized Data

Assume that in the original data D, QI, and SD are highly correlated.
An example could be the correlation between demographic and geographic
information, such as year of birth, country of birth, locality code, and income
[10]. Data set D contains the truth about the relationship between demo-
graphic and geographic information and income. While anonymizing D, the
truth should be preserved for the data to be useful. When D is anonymized
to D′ using a transformation function T, D′ = T(D), the QI fields are distorted
to some extent in D′. Now, how true is D′? Does the correlation between
QI and SD fields in D′ still exist? Each anonymization function will provide
­different levels of distortion. If Q is the distribution of QI fields in D and Q′
is the distribution of QI fields in D′, then the statistical distance measure of Q
and Q′ provides an indication of the utility of D′ [11]. This reference provides
a number of approaches to measure utility.
In Chapter 6, we show that the quality of anonymized test data sets is
one of the drivers for test coverage. The higher the quality of test data, the
higher will be the test coverage. High-quality test data are present in produc-
tion systems and contain PII. Privacy preservation results in poor test data
­quality or utility. Reduced utility reflects lower test coverage. We examine
different anonymization approaches and resulting utility.

1.6 Introduction to Anonymization Design Principles

Anonymization design is not straightforward. As we saw in Section 1.5,
achieving a balance between privacy and utility has many dependencies. So,
what are the drivers for anonymization design? Factors that drive anony-
mization design for a given requirement are illustrated in Figure 1.6.
When there is a need for data privacy, organizations generally use either
a commercial or a home-grown product for anonymizing data. It is critical
to ensure that an organization’s data anonymization program is not limited
by the features of the product. Many organizations fail to maintain a bal-
ance between privacy and utility. It is generally difficult to determine how
much anonymization is required, which results in either loss of informa-
tion or the anonymized data set becoming unusable. Even with adoption
of the best of breed data anonymization products, an organization’s anony-
mization program may not be successful. In addition to this, the pressures
of regulatory compliance force many organizations to be very defensive
and adopt very high privacy standards that will render the data unusable
for any research. Take, for example, HIPAA or Swiss Data Protection Act,
which are highly restrictive with an intention to protect the privacy of an
individual. If enough care is not taken, then the anonymized data could
20 Data Privacy: Principles and Practice

Business domain

Environment Data semantics

Data type and Anonymization

Regulations
structure design

Utility
Application
requirement

Privacy
requirement

FIGURE 1.6
Drivers for anonymization design.

have very little utility. In this context, irrespective of which tool an organi-
zation uses, there is a need for a mechanism to monitor privacy versus util-
ity for various privacy requirements. Unfortunately, quantifying privacy
and utility is nontrivial. Therefore, it is critical to provide assurance of high
quality of data anonymization during the initial phase of the anonymiza-
tion life cycle. To support this, we felt it is necessary to define a set of design
principles. These principles will provide the required guidelines for the
data anonymizer to adopt the correct design for a given anonymization
requirement.
As software architects, we start the architecting process by following a
set of architecture principles that will guide us to come up with the correct
design for the system. We base our work here on a similar approach. In [12],
the authors classify principles into two broad types—scientific and norma-
tive. Scientific principles are laws of nature and form the fundamental truths
that one can build upon. Normative principles act as a guide and need to be
enforced. Similarly, a data anonymizer needs guidance, and the anonymiza-
tion design principles should be enforced to ensure proper anonymization
design. These principles are fundamental in nature and are applicable to all
aspects of anonymization. They connect the high-level privacy and utility
requirements to low-level implementation.
Introduction to Data Privacy 21

In this book, all principles are explained in the following form:

• Principle Name
• Rationale
• Implications

These anonymization principles can be found in Appendix A.

1.7 Nature of Data in the Enterprise

1.7.1 Multidimensional Data
Multidimensional data also referred to as relational data are the most com-
mon format of data available today in many enterprises. In a relational
table, each row is a vector that represents an entity. The columns repre-
sent the attributes of the entity. As relational data are the most common
data format, a lot of attention has been paid to privacy preservation of
relational data [2,13,14]. As described earlier, a row of data in a relational
table is classified into explicit identifiers, quasi-identifiers, sensitive data,
and nonsensitive data. Both perturbative and nonperturbative techniques
could be used to protect the data. As a rule, EI are completely masked out,
QI are anonymized, and SD are left in their original form. (These terms
are explained in detail in Chapter 2.) Depending on the sensitivity of data,
appropriate data protection techniques can be applied. The fundamental
differences between anonymizing multidimensional data and other data
structures are as follows:

• In a multidimensional data table, each record or row is independent

of others; therefore, anonymizing a few of the records will not affect
other records.
• Anonymizing a tuple in a record will not affect other tuples in the
record.

Other complex data structures, such as graph, longitudinal, or time series

data, cannot be viewed in this way. Privacy preservation for multidimen-
sional data can be classified into (1) random perturbation methods and
(2) group anonymization techniques, such as k-anonymity or l-diversity.
These techniques are used to prevent identity disclosure and attribute
disclosure.
22 Data Privacy: Principles and Practice

[Link] Challenges in Privacy Preservation of Multidimensional Data

The challenges in this kind of data preservation are as follows:

1. Difficulty in identifying the boundary between QI and SD in the

presence of background knowledge of the adversary
2. High dimensionality of data poses a big challenge to privacy
preservation
3. Clusters in sensitive data set
4. Difficulty in achieving realistic balance between privacy and utility

1.7.2 Transaction Data

Transaction data are a classic example of sparse high-dimensional data.
A transaction database holds transactions of a customer at a supermarket or
it can be used to hold the diagnosis codes of a patient in a hospital. Privacy
of transaction data is very critical as an adversary who has access to this
database can obtain the shopping preferences of customers and exploit that
information. But the problem with transaction database is that it is of very
high dimensionality and sparsely filled. A supermarket will have thou-
sands of products contributing to the high dimensionality of the transaction
database. Moreover, the transactional data contained in the database are
binary—either 0 or 1. An event of a transaction is represented by 1; other-
wise, it would be a 0 (Table 1.9).
In this table, P1–Pn represents the products in the supermarket. The cus-
tomer Hari has made a transaction on P3 and P6, which means his shopping
cart contains product P3 and P6, say bread and cheese. There is nothing sen-
sitive about a customer buying bread and cheese. But if the product hap-
pens to be a blood glucose or blood pressure monitor, then that transaction
is sensitive from the customer’s perspective as he would not want others to
know that he is diabetic. It is the sensitivity of the transaction that needs to be
protected. Privacy preservation techniques used in the case of relational data
table will not be applicable here.

TABLE 1.9
Sample Sparse High-Dimensional Transaction Database
in a Supermarket
Name P1 P2 P3 P4 P5 P6 Pn
Hari 1 1
Nancy 1 1
Jim 1 1
Introduction to Data Privacy 23

[Link] Challenges in Privacy Preservation of Transaction Data

Some of the challenges in privacy preservation of transaction data are
as follows:

1. High dimensionality.
2. Sparsity.
3. Conventional privacy preservation techniques used for relational
tables that have fixed schema are not applicable on transaction data.

1.7.3 Longitudinal Data

Longitudinal studies are carried out extensively in the healthcare domain.
An example would be the study of the effects of a treatment or medicine
on an individual over a period of time. The measurement of the effects is
repeatedly taken over that period of time on the same individual. The goal
of longitudinal study is to characterize the response of the individual to the
treatment. Longitudinal studies also help in understanding the factors that
influence the changes in response. Consider the following table that illus-
trates the effect of treatment for hypertension in a patient (Table 1.10).
The table contains a longitudinal set D, which has three disjoint sets of data—
EI, QI, and SD. A few important characteristics of the data set D that must be
considered while designing an anonymization approach are as follows:

• Data are clustered—composed of repeated measurements obtained

from a single individual at different points in time.
• The data within the cluster are correlated.
• The data within the cluster have a temporal order, which means the
first measurement will be followed by the second and so on [15].

TABLE 1.10
Sample Longitudinal Data Set in the Healthcare Domain
Systolic Diastolic
ID Name DOB ZIP Service Date Diseases (mmHg) (mmHg)
1 Bob 1976 56711 30/05/2012 Hypertension 180 95
2 Bob 1976 56711 31/05/2012 Hypertension 160 90
3 Bob 1976 56711 01/06/2012 Hypertension 140 85
4 Bob 1976 56711 02/06/2012 Hypertension 130 90
5 Bob 1976 56711 03/06/2012 Hypertension 125 85
6 Bob 1976 56711 04/06/2012 Hypertension 120 80
7 Alice 1969 56812 31/03/2012 Hypertension 160 90
24 Data Privacy: Principles and Practice

These are to be noted because the anonymization design should ensure

that these characteristics of D are preserved in the anonymized data set D′,
otherwise, the truth in the data will be lost.

[Link] Challenges in Anonymizing Longitudinal Data

Anonymization design for longitudinal data should consider two aspects:

1. The characteristics of longitudinal data in the anonymized data set

D′ should be maintained.
2. Anonymization designs aim to prevent identity and attribute
disclosure.

Consider the longitudinal data set D, which has three disjoint sets of data
(EI, QI, and SD). EI are completely masked to prevent identification. QI
are ­anonymized using generalization and suppression to prevent iden-
tity disclosure. In the case of longitudinal data, anonymizing identity attri-
butes alone is not sufficient to prevent an adversary from re-identifying the
patient. An adversary can still link some of the sensitive attributes to the
publicly available data, that is, medical records. Now comes the need to pre-
vent attributes disclosure. For longitudinal data, an anonymization design
that prevents identity as well as attributes disclosure is required [16]. There
are a number of techniques to prevent identity disclosure, such as perturba-
tive and nonperturbative techniques. Effective anonymization techniques
are required to prevent attributes disclosure, but these techniques should
also ensure that they preserve the characteristics of longitudinal data.

1.7.4 Graph Data

Graph data are interesting and found in many domains like social networks,
Electronics, Transportation, Software, and Telecom. A graph G = (V,E)
consists of a set of vertices together with a set of vertex pairs or edges.
Graphs are interesting as they model almost any relationship. This is
especially relevant in modeling networks like financial networks and also
social networks like Facebook, LinkedIn, and Twitter. It is in these types
of applications, we see the need for privacy preservation of graph data.
Social networks have many users and contain a lot of personal informa-
tion, such as network of friends and personal preferences. Social network
data analytics is a rich source of information for many companies that
want to understand how their products are received by the customers. For
example, a bank would like to get feedback from their customers about the
various financial products and services they offer. The bank can have its
own page on, say, Facebook where its customers provide their views and
feedback. Publishing these data for mining and analysis will compromise
Introduction to Data Privacy 25

the privacy of the customers. Therefore, it is required to anonymize the

data before provisioning it for analytics. However, graph data are complex
in nature. The more complex the data structure, the easier it is to identify
entities [17].
Due to the complexity of graph data, the anonymization design applied
is different compared to that applied to relational data tables. In the case
of relational data tables, each row is treated as an entity and its QI are ano-
nymized to prevent identity disclosure. The attack model in relational data
is also straightforward. An adversary will use an external data source to
identify an individual via the QI. Graph data are more complex, and because
of their complexity they provide more avenues for re-identification. Consider
the network shown in Figures 1.7 and 1.8.
Figure 1.7 depicts a network with original data of users. The same has
been anonymized in Figure 1.8. Will this anonymization be enough to
thwart an adversary’s attempt to re-identify the users? The simple answer
is no. The many challenges in anonymizing graph data are discussed next.

Hari Ram

Jane Jack

Bob Alice

FIGURE 1.7
Graph network with original data.

A B

F C

E D

FIGURE 1.8
Modified graph network.
26 Data Privacy: Principles and Practice

[Link] Challenges in Anonymizing Graph Data

Privacy of graph data can be classified into three categories [18]:

1. Identity disclosure
2. Link disclosure
3. Content/attribute disclosure

Apart from these, graph metrics such as betweenness, closeness, reachability,

path length, and centrality can also be used to breach the privacy of t­arget
individuals in a graph network.
Identity disclosure: Identity disclosure occurs when it is possible to identify
the users in the network.
Content disclosure: Just as in relational table, sensitive content is associated
with each node (entity). This sensitive content is classified into explicit iden-
tifiers like name, SSN, and QI, such as demographics, gender, date of birth,
and other sensitive data such as preferences and relationships.
Link disclosure: Links between users are highly sensitive and can be used to
identify relationships between users.
It is very challenging to anonymize graph data as it is difficult to devise
an anonymizing design. For the graph to maintain privacy and utility after
anonymization, the techniques need to alter the graph, but just enough to
prevent identity, content, and link disclosure. At the same time, changing
the nodes or labels could affect other nodes in the network and could poten-
tially alter the structure of the network itself. Meanwhile, there are multiple
avenues to identify a user in a graph network since it is difficult to model
the adversary’s background knowledge. How to provision graph data in a
privacy preserving way for mining and analysis? What anonymization tech-
niques are available for graph data?

1.7.5 Time Series Data

Time series data result from taking measurements at regular intervals of
time from a process. An example of this could be temperature measurement
from a sensor or daily values of a stock, the net asset value of a fund taken on
a daily basis, or blood pressure measurements of a patient taken on a weekly
basis. We looked at longitudinal data where we considered the response of
a patient to blood pressure medication. The measurements have a temporal
order. So, what is the difference between longitudinal data and time series
data? Longitudinal data are extensively used in the healthcare domain, espe-
cially in clinical trials. Longitudinal data represent repeated measurements
taken on a person. These measurements are responses to a treatment or drug,
and there is a strong correlation among these measurements. A univariate
time series is a set of long measurements of a single variable taken at regular
Introduction to Data Privacy 27

TABLE 1.11
Sample Time Series Data Table Showing Weekly Sales of Companies
Company
ID Name Address Week 1 Week 2 Week 3 Week 4 Week 5
1 ABC Park Street, 56001 10,000 12,000 17,000 8,000 11,000
2 ACME Kings Street, 56003 15,000 17,000 18,000 20,000 21,000
3 XYZ Main Street, 56022 20,000 23,000 25,000 26,000 30,000
4 PQR Queen Street, 56021 14,000 18,000 19,000 19,500 21,000

intervals, say, blood pressure measurements of a patient taken over a period

of time. These measurements need not necessarily be a response to a drug
or treatment. Longitudinal data have very small dimensions compared to
time series data that have high dimensional and keep growing. Sample time
series data showing weekly sales of their products are given in Table 1.11.
It can be observed from the table that each record has three disjoint sets
of data—EI, QI, and SD. This is very similar to the structure of multidi-
mensional data. But that is where the similarity ends. In multidimensional
data, each record is independent of the others and can be anonymized with-
out affecting other records. The tuples in each record can be anonymized
­without affecting other tuples in the record. But this approach cannot be
used with time series data because of its large size, high dimensionality, and
pattern. This makes privacy preservation rather challenging.

[Link] Challenges in Privacy Preservation of Time Series Data

Some of the challenges in privacy preservation of the time series data are as
follows:

• High dimensionality
• Retaining the statistical properties of the original time series data
like mean, variance, and so on
• Supporting various types of queries like range query or pattern
matching query
• Preventing identity disclosure and linkage attacks

References
1. J.M. Skopek, Anonymity: The production of goods and institutional design,
Fordham Law Review, 82(4), 1751–1809, 2014, [Link]
vol82/iss4/4/.
2. L. Sweeney, k-Anonymity: A model for protecting privacy, International Journal
of Uncertainty, Fuzziness and Knowledge Based Systems, 10 (5), 557–570, 2002.
 CHAPTER 1
Data Governance and
Simple Privacy Approaches

Data privacy is a large and long-lived field. I want you to picture it like an old road,
packed with interesting side streets and diversions but hard to navigate if you don’t
know the way. This chapter is your initial orientation to this road. In this chapter and
throughout this book, I’ll help you map important parts of the privacy landscape, and
you’ll find areas where you want to learn more and deviate from the original path.
Applying this map within your organization means uncovering who is doing
what, what their responsibilities are, and what data privacy needs exist in your
organization.1
You might have heard the phrase data governance only once or hundreds of times,
but it is often left unexplained or open for interpretation. In this chapter, you’ll learn
where data governance overlaps with data privacy for practical data science purposes
and learn simpler approaches for solving privacy problems with data, such as pseudo‐
nymization. You’ll also learn how governance techniques like documentation and
lineage tracking can help identify privacy problems or ways to implement privacy
techniques at the appropriate step.

1 Throughout this book, I’ll use the term organization as a word to describe your workplace. If you are at a
small agile data science consultancy, a massive corporation, or a midsize nonprofit, you will have a vastly dif‐
ferent experience. This book should be useful for all groups—take the advice and learnings and use your own
knowledge of your work to fit them to your size and culture.

1
 If you already know or work in data governance, I recommend
skimming or skipping this chapter. If governance and data man‐
agement are new to you, this chapter will show you the founda‐
tions needed to apply the advanced techniques you’ll learn in later
chapters.

This chapter will help give you tools and systems to identify, track, and manage sensi‐
tive data. Without this foundation, it will be difficult to assess privacy risk and miti‐
gate those concerns. Starting with governance makes sense, because privacy fits well
into the governance frameworks and paradigms, and these areas of work support one
another in data systems.

Data Governance: What Is It?

Data governance is often used as an “all-encompassing” way to think about our data
decisions, like whether to opt in to allowing a service to contact you or determining
who has access rights to a given database. But what does the phrase really refer to,
and how can you make it actionable?
Data governance is literally governing data. One way to govern happens via a transfer
of rights people individually and communally possess. Those rights are passed onto
elected officials who manage tasks and responsibilities for individuals who have no
time, expertise, or interest. In data governance, individuals transfer rights when data
is given to an organization. When you use a website, service, or application, you agree
to whatever privacy policy, terms, and conditions or contract is presented by those
data processors or collectors at that time. This is similar to living in a particular state
and implicitly agreeing to follow the laws of that land.
Data governance helps manage whose data you collect, how you collect and enhance
it, and what you do with it after collection. Figure 1-1 illustrates how privacy and
security relate to data governance, via an imaginary island where users and their data
are properly protected by both privacy and security initiatives. In this diagram, you
can see the sensitive data inside a tower. Security initiatives are supported by Privacy
by Design.2 Regulations and compliance provide a moat that keeps sensitive data sep‐
arate. Privacy technologies you will learn in this book are bridges for users and data
stakeholders, allowing them to gather insights and make decisions with sensitive data
without violating individual privacy.

2 Privacy by Design is a set of principles developed by Ann Cavoukian outlining measures technologists can use
to ensure systems are architected and software is designed with privacy in mind from the beginning. You will
hear it used quite often in conversations with experienced governance experts. I recommend taking time to
read and explore these principles and determine how they fit your data work. These principles are included in
Chapter 11.

2 | Chapter 1: Data Governance and Simple Privacy Approaches

Figure 1-1. Mapping data governance

Data governance can be described as a mixture of people, process, and technology.

Regardless of your organization size, there is always some amount of data governance
work to be done. If you are at a large organization, there is probably a large team or
committee creating standards, which turn into policies and procedures; those then
need to be implemented in the organization’s technology. If your organization is
small, this might be the job of your technical or legal leader. Let’s zoom into the tech‐
nology section, as this is likely where you’ll be asked to help take these policies and
procedures and ensure they are actually part of regular data processing.
What elements of taking governance standards and policies and implementing them
in technology are important for data scientists? Table 1-1 outlines significant areas
and related questions within data governance that you will grapple with as a data
scientist.

Data Governance: What Is It? | 3

Table 1-1. Data governance in data science
Data lineage/Origin Policies and controls
Where did the data come from? What laws or internal policies apply to this data?
Whose data is it? Is it possible to contact them? Where, when, and how was it collected?
Was this data acquired from someone else, and if so, did What privacy or security concerns do you need to address when
they document how it was processed and who it belongs using this data?
to?
How did the processing change the data? What was the privacy policy and terms at collection time?
Is the metadata for lineage information easily accessible Did the data come from a third party? If so, what are the
and queryable? restrictions and obligations, contractual or otherwise, for this
data?

Data reliability/Knowledge Data privacy and security

What are the concerns around understanding the data and How is access to sensitive data managed and monitored?
systems (i.e., including collection, transformation, and
downstream systems)?
Does the data have an understandable documentation trail from Does the organization know if and when data has been
the moment it was collected? breached? How?
When there are data quality problems, do you know how to Who is responsible for managing privacy controls? Security
diagnose and resolve them? controls?
Are there data storage infrastructure or old data stores that are When someone invokes their data rights (i.e., GDPR), is
undocumented or even unknown? there a system that is well documented and understood to
apply these rights?
Is the data well documented and understood? (Know your data.) What data loss prevention technologies and privacy
technologies do you use and how?

You are likely already focused on many of these questions since data is a major part
of your job. You might have personally suffered from a lack of data documentation,
incomplete understanding of how a certain database came to be, and issues with data
labeling and quality. Now you have a new word to use to describe these qualities:
governance!
Working on the governance side of data administration or management is really
about focusing on how to collect and update information about the data throughout
its lifecycle. The regulatory, privacy, and security concerns shape that information
and ensure governance decisions and frameworks expedite measures like individual
data rights and appropriate usage of data. If your data does not come from individu‐
als, there may be other concerns with regard to proprietary data or related security
issues that guide governance initiatives.
When you think about governing data in a concrete way, you begin to look at tasks
such as documenting the ever-changing data flows at your organization. It seems
obvious and easy, but on closer look it is anything but.

4 | Chapter 1: Data Governance and Simple Privacy Approaches

Let’s say you have a huge data lake that gets fed from 10 different sources, some
external, some internal. How can you actually begin to govern that data? What would
a scalable and easy-to-use solution look like? What happens when those data flows
change? It may be enough just to document the code or the workflows that are
actively running and in use and to leave the rest for future work. But what do you do
with data from partners or other external data collection systems? You’ll need to
coordinate this documentation so the legal, privacy, and risk departments can use it
for auditing and assessment. This process should not be solved with piecemeal and
temporary solutions but instead addressed as holistically as possible.
To begin, let’s identify which data is the most important to protect for the purpose of
practical data privacy. How can you identify sensitive data? What exactly is sensitive
data?

Identifying Sensitive Data

In the context of privacy, sensitive data is normally defined as person-related data or
even just personally identifiable data.3 It includes your full name, your email address,
your gender, your mailing address, your IP address, your social media profile, your
phone number, your Social Security number or other national identification number,
your credit card number, your birthday, your health records or biometric data (i.e.,
fingerprints, iris scan, or even your gait!).
All of these fall under the category of personally identifiable data, or what many refer
to as personally identifiable information (PII). This data is specific to you; it can be
used alone or in combination with other sources of information about you to directly
identify, indirectly identify, or re-identify you. This is the most sensitive and fre‐
quently regulated data because it provides a nearly or completely unique identifier.
As defined in this book, sensitive data includes:
PII
Data that is unique or close to unique for a particular individual. This is usually
defined in policy and regulations and can include things that you might not
expect, such as your IP address, date of birth, and workplace.
Person-related data
Data that relates to a person but doesn’t fall under PII. This could be anything
related to their personhood, including interests, beliefs, locations, and online and
offline behaviors and activities.

3 Your organization might have their own definition of sensitive data that varies from the one used in this
book. Ensure you use appropriate terms internally when bringing up these topics.

Identifying Sensitive Data | 5

Proprietary and confidential data
Data deemed sensitive for contractual or business-related purposes. Its release
would endanger a business or other legal relationship or agreement.
One thing I hope this book will do is expand your definition of sensitive data to
include a broader range. For example, do you think your phone location is sensitive?
Or does it depend on where you are? If you are sitting at home, your phone location
also reveals your mailing address or residence, which again is personally identifiable.
What about if you are at work? Or in a movie theater? Or at a close friend’s house?
What about your political affiliation and areas of interest? Voting history? Religious
beliefs and practices? Your friendships, partnerships, and who you connect with?
What about your daily routines, the news you consume, your music and entertain‐
ment choices, the devices you own?
These questions begin to reveal the range of privacy preferences. Some individuals
might be comfortable sharing their location at work or might even be required to do
so. Others might see this as an invasion of privacy. Whereas one person might be
quite open about their personal relationships, political choices, and religion, another
person might see these topics as deeply personal and sensitive. This echoes the notion
of contextual privacy and social privacy as discussed in the Preface. Here, regulations
step in, giving individuals more choice regarding their privacy preferences and the
ability to communicate that sensitivity to data collectors via changes in privacy poli‐
cies and consent choices.
It also means, however, that when you work with data, you recognize the range of
what is considered sensitive. You should be aware of the additional privacy risk cre‐
ated when person-related data is combined in a new way that inadvertently exposes
the individuals. For example, if I track your location throughout the day, I would
likely learn where you work, where you eat, your daily personal activities, and where
you live. Even if I were to collect data just while you were moving (i.e., driving data),
I would be able to identify some of those attributes. Even if I collected data only when
you were in the presence of others, I could likely still infer things about you, such as if
you travel with family or a friend or if you like to shop at a particular store or com‐
mute along a particular route and at what times.

6 | Chapter 1: Data Governance and Simple Privacy Approaches

Similarly, it’s been shown by researchers that a series of Facebook likes can be used to
infer things such as gender, sexual orientation, and political beliefs—even the marital
status of your parents.4 These are inferences, not necessarily fact—but it is clear that
online behavior and social network behavior create unique breadcrumbs and reveal
patterns that expose personal and private traits of an individual. This proves any
person-generated data is potentially identifiable.
The power of inference combined with vast quantities of information can identify
individuals even when that is not the intention. In targeted advertising, sensitive
attributes are often inferred and combined without consent, leading to recommenda‐
tions that might leak sensitive information such as sexual orientation or political
views. When an advertiser is choosing targeted groups, the more factors they specify,
the more easily they could erroneously target a particular individual or very small tar‐
get group. If an advertiser is acting maliciously, they can use the information they
have to figure out how to target that exact person or get fairly close.
For these reasons, the term sensitive data could mean any person-related data,
regardless of if it is directly personally identifiable or not. Person-related data, partic‐
ularly in large quantities or in aggregate, is identifiable. In this book, when I say sen‐
sitive data, I am referring not only to PII but also to a broader range of person-related
data, which could be used in combination with other information to identify a person
or small group of people.
A final category of sensitive data is data that is proprietary or confidential for non-
person-related reasons. This can be trade secrets, proprietary information about the
business, or a particular product or information that falls under confidentiality clau‐
ses. This could be data shared between parent companies and their subsidiaries,
which must be kept secret due to internal policies or confidentiality agreements. Or it
could be sensitive internal data that, if leaked, would give competitors an edge or
compromise the company in another manner. This type of sensitive information also
benefits from approaches and technologies you’ll learn in this book.

I am a supporter of whistleblowing. If you have data you believe

should be publicly known but is considered sensitive, think about
the techniques you will learn in this book as ways to release that
data publicly or to the appropriate authorities in a responsible and
thoughtful way.

4 This work was published by some of the researchers who later worked on Cambridge Analytica. See: Kosinki
et al., “Private traits and attributes are predictable from digital records of human behavior”, 2013.

Identifying Sensitive Data | 7

The first step toward protecting sensitive data is reliably identifying it. Once data
is identified and documented as sensitive, you can then figure out how best to
protect it.

Identifying PII
PII falls under a particular legal category in most data protection regulations, which
requires close attention when data governance is implemented at an organization. If
your organization collects any personal data—even for employees—then this data
often has special governance requirements. Frequently, there is a lack of documenta‐
tion or categorization of PII because it often shows up in text files, log files, or other
unstructured data, which are all notoriously poorly documented.
There are several tools built explicitly for PII discovery in unstructured data using a
variety of methods. I’ve also seen teams successfully build their own tools and sys‐
tems for PII discovery. Many tools use fairly brittle methods like regular expressions
(which are strings used to match patterns) or string entropy (for finding things like
application programming interface [API] keys, cryptographic keys, or passwords). I
have also successfully built deep learning models to identify PII in message text. Your
results with these approaches will vary and should be evaluated depending on your
use cases.

PII discovery is never perfect and never will be. It’s important to
talk with your risk teams (privacy, legal, security) about this fact,
whether you purchase a PII discovery toolkit or build your own.
It’s safest to treat human-input data as extremely sensitive (e.g., as
PII), regardless of the “cleaning processes.” If you would like to use
human-input data without extra protections associated with PII,
you need to properly identify risks and ensure they are addressed
and appropriately tested.

If you are working with a backlog of undocumented data and you fear there is a lot of
PII contained therein, take a look at an easy-to-use open source tool. After you see
how far that will take you, determine whether you need to invest in a more advanced
or expensive approach. I can recommend Microsoft’s Presidio, which also includes
some basic pseudonymization techniques covered later in this chapter.
The best approach to manage and track PII is to actually track the data as it comes in
and to label and manage this data as it traverses the system so you don’t need to
aggressively search or discover it later. One of the ways you can start the habit and
culture of detecting PII early and often is to build a culture of documentation around
data collection and data use. Table 1-1 gives you a good start. To develop a compre‐
hensive approach, you might involve numerous parts of your organization, including

8 | Chapter 1: Data Governance and Simple Privacy Approaches

 Chapter 1
Security and Cryptography Concepts
Learning Objectives
After studying this chapter, you should be able to:
■ Describe the five main security objectives
■ Explain the main uses of cryptography
■ Describe four types of cryptographic algorithms
■ Understand the concept of public-key infrastructure

Essential to understanding the concept of information privacy is an understanding of information

security. As described in Chapter 2, “Information Privacy Concepts,” there are areas of overlap as
well as areas of nonoverlap between these two fields. This book provides numerous examples of
privacy management, design, and technology concepts that require the use of the corresponding
security concepts. This chapter provides a brief overview of information security concepts in
Sections 1.1 through 1.4. Then, Sections 1.5 through 1.10 survey important aspects and implications
of cryptography, which is the fundamental technology underlying much of information security. Then,
Section 1.11 introduces the important topic of public-key infrastructure. Finally, Section 1.12 provides
an introduction to network security.

1.1 Cybersecurity, Information Security, and

Network Security
It is useful to start this chapter with a definition of the terms cybersecurity, information security, and
network security. A reasonably comprehensive definition of cybersecurity based on one in NISTIR
(Small Business Information Security: The Fundamentals, 2016) follows:

2
 1.1 Cybersecurity, Information Security, and Network Security 3

Cybersecurity is the prevention of damage to, unauthorized use of, exploitation of, and—if
needed—the restoration of electronic information and communications systems, electronic
communications services, wire communication, and electronic communication, including
information contained therein, to ensure its availability, integrity, authentication, confiden-
tiality, and nonrepudiation.

As subsets of cybersecurity, we can define the following:

■ Information security: Preservation of confidentiality, integrity, and availability of information.

In addition, other properties, such as authenticity, accountability, nonrepudiation, and reliability,
can also be involved.
■ Network security: Protection of networks and their services from unauthorized modification,
destruction, or disclosure and provision of assurance that the networks perform their critical
functions correctly, without harmful side effects.

Cybersecurity encompasses information security, with respect to electronic information, and network
security. Information security also is concerned with physical (e.g., paper-based) information. However,
in practice, the terms cybersecurity and information security are often used interchangeably.

Security Objectives
The cybersecurity definition just given introduces three key objectives that are at the heart of infor-
mation and network security:

■ Confidentiality: Also known as data confidentiality, the property that information is not made
available or disclosed to unauthorized individuals, entities, or processes. A loss of confidential-
ity is the unauthorized disclosure of information.
■ Integrity: This term covers two related concepts:
■ Data integrity: Ensures that data (both stored and in transmitted packets) and programs
are changed only in a specified and authorized manner. A loss of data integrity is the
unauthorized modification or destruction of information.
■ System integrity: Ensures that a system performs its intended function in an unimpaired
manner, free from deliberate or inadvertent unauthorized manipulation of the system.
■ Availability: Ensures that systems work promptly and that service is not denied to authorized
users. A loss of availability is the disruption of access to or use of information or an information
system.
4 CHAPTER 1 Security and Cryptography Concepts

Note
We can define information as communication or representation of knowledge such as facts, data,
or opinions in any medium or form, including textual, numerical, graphic, cartographic, narrative,
or audiovisual; and data as information with a specific representation that can be produced,
processed, or stored by a computer. Security literature typically does not make much of a
distinction between the two, nor does this book.

These three concepts form what is often referred to as the CIA triad. These three concepts embody
the fundamental security objectives for both data and information and computing services.
For example, the NIST (National Institute of Standards and Technology) standard FIPS 199
(Standards for Security Categorization of Federal Information and Information Systems) lists
confidentiality, integrity, and availability as the three security objectives for information and for
information systems.
Although the use of the CIA triad to define security objectives is well established, many in the
security field feel that additional concepts are needed to present a complete picture, as illustrated
in Figure 1.1. The list that follows describes two of the most commonly mentioned additional
security concepts:

ity
tial
en In
te
nfid gr
Co ity
Acc

Information
oun

and Network
ity

Security
tabi

ntic
lity

he
Aut

Availability

FIGURE 1.1 Essential Information and Network Security Objectives

■ Authenticity: The property of being genuine and being able to be verified and trusted; confi-
dence in the validity of a transmission, a message, or a message originator. This means verify-
ing that users are who they say they are and that each input arriving at the system came from a
trusted source.
■ Accountability: The security goal that generates the requirement for actions of an entity to
be traced uniquely to that entity. This supports nonrepudiation, deterrence, fault isolation,
 1.1 Cybersecurity, Information Security, and Network Security 5

intrusion detection and prevention, and after-action recovery and legal action. Because truly
secure systems are not yet an achievable goal, it must be possible to trace a security breach to a
responsible party. Systems must keep records of their activities to permit later forensic analysis
to trace security breaches or to aid in transaction disputes.

The Challenges of Information Security

Information and network security are both fascinating and complex. Some of the reasons follow:

1. Security is not as simple as it might first appear to the novice. The requirements seem to be
straightforward; indeed, most of the major requirements for security services can be given self-
explanatory, one-word labels: confidentiality, authentication, nonrepudiation, integrity. But the
mechanisms used to meet those requirements can be quite complex, and understanding them
may involve rather subtle reasoning.
2. In developing a particular security mechanism or algorithm, designers must always consider
potential attacks on those security features. In many cases, successful attacks are designed by
looking at the problem in a completely different way and exploiting an unexpected weakness in
the mechanism.
3. Because of point 2, the procedures used to provide particular services are often counterintuitive.
Typically, a security mechanism is complex, and it is not obvious from the statement of a par-
ticular requirement that such elaborate measures are needed. It is only when the various aspects
of the threat are considered that elaborate security mechanisms make sense.
4. Having designed various security mechanisms, it is necessary to decide where to use them. This
is true both in terms of physical placement (e.g., at what points in a network are certain security
mechanisms needed?) and in a logical sense (e.g., should containers or virtual machines be used
to isolate personal information, or should an integrated access control mechanism be used to
protect all types of data, with access determined by user roles and privileges?).
5. Security mechanisms typically involve more than a particular algorithm or protocol.
They also require that participants be in possession of some secret information (e.g., an
encryption key), which raises questions about the creation, distribution, and protection of
that secret information. There also may be a reliance on communications protocols, whose
behavior may complicate the task of developing the security mechanism. For example, if
the proper functioning of a security mechanism requires setting time limits on the transit
time of a message from sender to receiver, then any protocol or network that introduces
variable, unpredictable delays might render such time limits meaningless. It is worth
mentioning here that security mechanisms are generally not secret, and indeed they may
be open source, but secrecy is provided by maintaining the security of encryption keys, as
discussed subsequently.
6 CHAPTER 1 Security and Cryptography Concepts

6. Information and network security are essentially a battle of wits between a perpetrator who tries
to find holes and a designer or an administrator who tries to close them. The great advantage
that the attacker has is that he or she need find only a single weakness, while the designer must
find and eliminate all weaknesses to achieve perfect security.
7. There is a natural tendency on the part of users and system managers to perceive little benefit
from security investment until a security failure occurs.
8. Security requires regular, even constant, monitoring, and this is difficult in today’s short-term,
overloaded environment.
9. Security is still too often an afterthought to be incorporated into a system after the design is
complete rather than being an integral part of the design process.
10. Many users and even security administrators view strong security as an impediment to efficient
and user-friendly operation of an information system or use of information.

These difficulties apply equally as well to information privacy. The difficulties just enumerated will be
encountered in numerous ways as we examine the various privacy threats and mechanisms throughout
this book.

1.2 Security Attacks

ITU-T Recommendation X.800 (Security Architecture for Open Systems Interconnection [OSI]) defines
a general security architecture that is useful to managers as a way of organizing the task of providing
security. The OSI security architecture focuses on security attacks, mechanisms, and services. These
can be defined briefly as follows:

■ Security attack: Any action that compromises the security of information owned by an
organization.
■ Security mechanism: A process (or a device incorporating such a process) that is designed to
detect, prevent, or recover from a security attack.
■ Security service: A processing or communication service that enhances the security of the
data processing systems and the information transfers of an organization. Security services are
intended to counter security attacks, and they make use of one or more security mechanisms to
provide the services.

In the literature, the terms threat and attack are commonly used, with the following meanings:

■ Threat: Any circumstance or event that has the potential to adversely impact organiza-
tional operations (including mission, functions, image, or reputation), organizational assets,
 1.2 Security Attacks 7

individuals, other organizations, or the nation through an information system via unauthorized
access, destruction, disclosure, modification of information, and/or denial of service.
■ Attack: Any kind of malicious activity that attempts to collect, disrupt, deny, degrade, or
destroy information system resources or the information itself.

Sections 1.2 through 1.4 provide an overview of the concepts of attacks, services, and mechanisms.
The key concepts that are covered in these sections are summarized in Figure 1.2.

Passive Attacks Active Attacks

Release of Data
message Replay
contents modification

Traffic Denial of
Masquerade
analysis service

(a) Attacks

Access Data
Authentication control confidentiality

Data Availability
Nonrepudiation
integrity service

(b) Services

Cryptographic Data Digital Authentication

algorithms integrity signature exchange

Routing Access
Traffic padding Notarization
control control

(c) Mechanisms

FIGURE 1.2 Key Concepts in Security

X.800 classifies security attacks as either passive attacks or active attacks. A passive attack attempts to
learn or make use of information from the system but does not affect system resources. An active attack
attempts to alter system resources or affect their operation.
8 CHAPTER 1 Security and Cryptography Concepts

Passive Attacks
Passive attacks are in the nature of eavesdropping on, or monitoring of, transmissions. The goal of the
attacker is to obtain information that is being transmitted. Two types of passive attacks are the release
of message contents and traffic analysis:

■ Release of message contents: The act of an adversary in successfully eavesdropping on a com-

munication, such as a telephone conversation, an electronic mail message, or a transferred file.
■ Traffic analysis: A form of attack in which the contents of transmitted data blocks are not
examined. Assume that there was a way of masking the contents of messages or other informa-
tion traffic so that adversaries, even if they captured the message, could not extract the infor-
mation from the message. The common technique for masking contents is encryption. With
encryption protection in place, an adversary might still be able to observe the pattern of these
messages. The adversary could determine the location and identity of communicating hosts and
could observe the frequency and length of messages being exchanged. This information might
be useful in guessing the nature of the communication that was taking place.

Passive attacks are very difficult to detect because they do not involve any alteration of the data.
Typically, the message traffic is sent and received in an apparently normal fashion, and neither the
sender nor the receiver is aware that a third party has read the messages or observed the traffic pattern.
However, it is possible to prevent the success of these attacks, usually by means of encryption. Thus,
the emphasis in dealing with passive attacks is on prevention rather than detection.

Active Attacks
Active attacks involve some modification of stored or transmitted data or the creation of false data.
There are four categories of active attacks: replay, masquerade, modification of messages, and denial
of service.

■ A masquerade takes place when one entity pretends to be a different entity. A masquerade
attack usually includes one of the other forms of active attack. For example, authentication
sequences can be captured and replayed after a valid authentication sequence has taken place,
thus enabling an authorized entity with few privileges to obtain extra privileges by impersonat-
ing an entity that has those privileges.
■ Replay involves the passive capture of a data unit and its subsequent retransmission to produce
an unauthorized effect.
■ Data modification simply means that some portion of a legitimate message is altered or that
messages are delayed or reordered to produce an unauthorized effect. For example, a message
stating “Allow John Smith to read confidential file Accounts” might be modified to say “Allow
Fred Brown to read confidential file Accounts.”
 1.2 Security Attacks 9

■ A denial-of-service attack prevents or inhibits the normal use or management of communica-

tion facilities. Such an attack may have a specific target; for example, an entity may suppress all
messages directed to a particular destination (e.g., the security audit service). Another form of
service denial is the disruption of an entire network, either by disabling the network or by over-
loading it with messages so as to degrade performance.

Active attacks present the opposite characteristics of passive attacks. Whereas passive attacks are
difficult to detect, measures are available to prevent their success. On the other hand, it is quite
difficult to prevent active attacks absolutely because to do so would require physical protection of all
IT systems, communication facilities, and paths at all times. Instead, the goal is to detect them and to
recover from any disruption or delays caused by them. Because detection has a deterrent effect, it may
also contribute to prevention.
Figure 1.3 illustrates the types of attacks in the context of a client/server interaction. A passive attack
(part b of Figure 1.3) does not disturb the information flow between the client and server but is able to
observe that flow.

Client Server

(a) Normal flow

Client Server Client Server

Attacker Attacker

(b) Passive attack (eavesdrop, traffic analysis) (c) Active attack (masquerade,
data modification)

Client Server Client Server

Attacker Attacker

(d) Active attack (masquerade, data (e) Active attack (replay)

modification, denial of service)

FIGURE 1.3 Security Attacks

10 CHAPTER 1 Security and Cryptography Concepts

A masquerade can take the form of a man-in-the-middle attack (see part c of Figure 1.3). In this type
of attack, the attacker intercepts masquerades as the client to the server and as the server to the client.
Another form of masquerade is illustrated in part d of Figure 1.3. Here, an attacker is able to access
server resources by masquerading as an authorized user.
Data modification may involve a man-in-the middle attack, in which the attacker selectively modifies
communicated data between a client and server (see part c of Figure 1.3). Another form of data modi-
fication attack is the modification of data residing on a server or another system after an attacker gains
unauthorized access (see part d of Figure 1.3).
Part e of Figure 1.3 illustrates the replay attack. As in a passive attack, the attacker does not disturb
the information flow between client and server but does capture client messages. The attacker can then
subsequently replay any client message to the server.
Part d of Figure 1.3 also illustrates denial of service in the context of a client/server environment. The
denial of service can take two forms: (1) flooding the server with an overwhelming amount of data and
(2) triggering some action on the server that consumes substantial computing resources.

1.3 Security Services

A security service is a capability that supports one or more of the security requirements (confidenti-
ality, integrity, availability, authenticity, accountability). Security services implement security policies
and are implemented by security mechanisms.
The most important security services are shown in part b of Figure 1.2 and summarized in Table 1.1.

TABLE 1.1 Security Services

Service Description
Authentication A person’s identity is determined before access is granted.
Access Control Persons are allowed or denied access to resources for specific purposes.
Data Confidentiality Information is only available to persons intended to use or see it.
Data Integrity Information is modified only in appropriate ways by persons authorized to change it.
Nonrepudiation A person cannot perform an action and then later deny performing the action.
Availability Apps, services, and hardware are ready when needed and perform acceptably.

Authentication
The authentication service is concerned with ensuring that a communication is authentic. In the case
of a single message, such as a warning or an alarm signal, the function of the authentication service is
to ensure the recipient that the message is from the source that it claims to be from. In the case of an
ongoing interaction, such as the connection of a client to a server, two aspects are involved. First, at the
time of connection initiation, the service ensures that the two entities are authentic—that is, that each is
the entity that it claims to be. Second, the service must ensure that the connection is not interfered with
 1.3 Security Services 11

in such a way that a third party can masquerade as one of the two legitimate parties for the purpose of
unauthorized transmission or reception.
X.800 defines two specific authentication services:

■ Peer entity authentication: Provides for the corroboration of the identity of a peer entity
in an association. Two entities are considered peers if they implement the same protocol in
different systems. Peer entity authentication is provided for use at the establishment of, or
at times during the data transfer phase of, a connection. It attempts to provide confidence
that an entity is not performing either a masquerade or an unauthorized replay of a previ-
ous connection.
■ Data origin authentication: Provides for the corroboration of the source of a data unit. It does
not provide protection against the duplication or modification of data units. This type of service
supports applications like electronic mail, where there are no ongoing interactions between the
communicating entities.

Access Control
Access control is the ability to limit and control the access to host systems and applications via commu-
nications links. To achieve this, each entity trying to gain access must first be identified, or authenti-
cated, so that access rights can be tailored to the individual.

Data Confidentiality
Confidentiality is the protection of transmitted data from passive attacks. With respect to the content
of a data transmission, several levels of protection can be identified. The broadest service protects all
user data transmitted between two users over a period of time. For example, when a logical network
connection is set up between two systems, this broad protection prevents the release of any user data
transmitted over the connection.
The other aspect of confidentiality is the protection of traffic flow from analysis. This requires that an
attacker not be able to observe the source and destination, frequency, length, or other characteristics of
the traffic on a communications facility.

Data Integrity
A connection-oriented integrity service—one that deals with a stream of messages—ensures that
messages are received as sent, with no duplication, insertion, modification, reordering, or replays. The
destruction of data is also covered under this service. Thus, the connection-oriented integrity service
addresses both message stream modification and denial of service. On the other hand, a connectionless
integrity service—one that deals with individual messages without regard to any larger context—
generally provides protection against message modification only.
12 CHAPTER 1 Security and Cryptography Concepts

We can make a distinction between service with and without recovery. Because the integrity service
relates to active attacks, we are concerned with detection rather than prevention. If a violation of
integrity is detected, then the service may simply report this violation, and some other portion of
software or human intervention is required to recover from the violation. Alternatively, there
are mechanisms available to recover from the loss of integrity of data, as we will review subsequently.
The incorporation of automated recovery mechanisms is, in general, the more attractive alternative.

Nonrepudiation
Nonrepudiation prevents either a sender or a receiver from denying a transmitted message. Thus, when
a message is sent, the receiver can prove that the alleged sender in fact sent the message. Similarly,
when a message is received, the sender can prove that the alleged receiver in fact received the message.

Availability Service
Availability means that a system or a system resource is accessible and usable upon demand by an
authorized system entity, according to performance specifications for the system; that is, a system is
available if it provides services according to the system design whenever users request them. A variety
of attacks can result in loss of or reduction in availability. Some of these attacks are amenable to auto-
mated countermeasures, such as authentication and encryption, whereas others require some sort of
physical action to prevent or recover from loss of availability of elements of a distributed system.
X.800 treats availability as a property to be associated with various security services. However, it
makes sense to call out specifically an availability service—that is, a service that protects a system to
ensure its availability. This service addresses the security concerns raised by denial-of-service attacks.
It depends on proper management and control of system resources and thus depends on access control
service and other security services.

1.4 Security Mechanisms

Part b of Figure 1.2 lists the most important security mechanisms, which include:

■ Cryptographic algorithms: Sections 1.5 through 1.9 cover this topic.

■ Data integrity: This category covers a variety of mechanisms used to ensure the integrity of a
data unit or stream of data units.
■ Digital signature: Data appended to, or a cryptographic transformation of, a data unit that allows a
recipient of the data unit to prove the source and integrity of the data unit and protect against forgery.
■ Authentication exchange: A mechanism intended to ensure the identity of an entity by means
of information exchange.
■ Traffic padding: The insertion of bits into gaps in a data stream to frustrate traffic analysis attempts.
 1.5 Cryptographic Algorithms 13

■ Routing control: A control that enables selection of particular physically or logically secure routes
for certain data and allows routing changes, especially when a breach of security is suspected.
■ Notarization: The use of a trusted third party to ensure certain properties of a data exchange.
■ Access control: A variety of mechanisms that enforce access rights to resources.

1.5 Cryptographic Algorithms

The NIST Computer Security Glossary ([Link] provides the following
definitions:

Cryptography: 1. The discipline that embodies the principles, means, and methods for the
transformation of data in order to hide their semantic content, prevent their unauthorized
use, or prevent their undetected modification. 2. The discipline that embodies the prin-
ciples, means, and methods for the providing information security, including confidenti-
ality, data integrity, non-repudiation, and authenticity.
Cryptographic algorithm: A well-defined computational procedure, pertaining to cryp-
tography, that takes variable inputs, often including a cryptographic key, and produces an
output.

Cryptography is an essential component in the secure storage and transmission of data and in the
secure interaction between parties. Sections 1.5 through 1.10 provide brief technical introductions
to important aspects of the use of cryptography and cryptographic algorithms. For a more detailed
treatment, see Cryptography and Network Security: Principles and Practice [STAL20].
Cryptographic algorithms can be divided into three categories, as illustrated in Figure 1.4 and described
in the list that follows:

Keyless Single-Key Two-Key

Cryptographic Block cipher Asymmetric
hash function symmetric encryption
encryption

Pseudo- Digital
random number Stream cipher signature
generator symmetric
encryption Key
exchange
Message
authentication User
code authentication

FIGURE 1.4 Cryptographic Algorithms

14 CHAPTER 1 Security and Cryptography Concepts

■ Keyless: An algorithm that does not use any keys during cryptographic transformations.
■ Single-key: An algorithm in which the result of a transformation is a function of the input data
and a single key, known as a secret key.
■ Two-key: An algorithm in which, at various stages of the calculation, two different but related
keys are used, referred to as the private key and the public key.

Keyless Algorithms
Keyless algorithms are deterministic functions that have certain properties that are useful for cryptography.
One important type of keyless algorithm is the cryptographic hash function. A hash function turns a
variable amount of text into a small, fixed-length value called a hash value, hash code, or digest. A
cryptographic hash function has additional properties that make it useful as part of another crypto-
graphic algorithm, such as a message authentication code or a digital signature.
A pseudorandom number generator produces a deterministic sequence of numbers or bits that has
the appearance of being a truly random sequence. Although the sequence appears to lack any definite
pattern, it will repeat after a certain sequence length. Nevertheless, for some cryptographic purposes,
this apparently random sequence is sufficient.

Single-Key Algorithms
Single-key cryptographic algorithms depend on the use of a secret key. This key may be known to a
single user; for example, this is the case when protecting stored data that is only going to be accessed
by the data creator. Commonly, two parties share the secret key so that communication between the
two parties is protected. For certain applications, more than two users may share the same secret key. In
this case, the algorithm protects data from those outside the group who share the key.
Encryption algorithms that use a single key are referred to as symmetric encryption algorithms. With
symmetric encryption, an encryption algorithm takes as input some data to be protected and a secret
key and produces an unintelligible transformation on that data. A corresponding decryption algorithm
uses the transformed data and the same secret key to recover the original data.
Another form of single-key cryptographic algorithm is the message authentication code (MAC). A
MAC is a data element associated with a data block or message. The MAC is generated by a cryp-
tographic transformation involving a secret key and, typically, a cryptographic hash function of the
message. The MAC is designed so that someone in possession of the secret key can verify the integrity
of the message. Thus, the MAC algorithm takes as input a message and a secret key and produces the
MAC. The recipient of the message plus the MAC can perform the same calculation on the message;
if the calculated MAC matches the MAC accompanying the message, this provides assurance that the
message has not been altered.
 1.6 Symmetric Encryption 15

Two-Key Algorithms
Two-key algorithms involve the use of two related keys. A private key is known only to a single user
or entity, whereas the corresponding public key is made available to a number of users. Encryption
algorithms that use two related keys are referred to as asymmetric encryption algorithms. Asymmetric
encryption can work in two ways:

■ An encryption algorithm takes as input some data to be protected and the private key and pro-
duces an unintelligible transformation on that data. A corresponding decryption algorithm uses
the transformed data and the corresponding public key to recover the original data. In this case,
only the possessor of the private key can have performed the encryption, and any possessor of
the public key can perform the decryption.
■ An encryption algorithm takes as input some data to be protected and a public key and produces
an unintelligible transformation on that data. A corresponding decryption algorithm uses the
transformed data and the corresponding private key to recover the original data. In this case,
any possessor of the public key can have performed the encryption, and only the possessor of
the private key can perform the decryption.

Asymmetric encryption has a variety of applications. One of the most important is the digital signature
algorithm. A digital signature is a value computed with a cryptographic algorithm and associated with
a data object in such a way that any recipient of the data can use the signature to verify the data’s
origin and integrity. Typically, the signer of a data object uses the signer’s private key to generate the
signature, and anyone in possession of the corresponding public key can verify the validity of that
signature.
Asymmetric algorithms can also be used in two other important applications. Key exchange is the
process of securely distributing a symmetric key to two or more parties. User authentication is the
process of authenticating that a user attempting to access an application or a service is genuine and, simi-
larly, that the application or service is genuine.

1.6 Symmetric Encryption

Symmetric encryption, also referred to as secret-key encryption, is a cryptographic scheme in which
encryption and decryption are performed using the same key. A symmetric encryption scheme has five
ingredients, as illustrated in Figure 1.5:
16 CHAPTER 1 Security and Cryptography Concepts

Adversary

Encryp-
Data X Encryption ted Decryption X Data
block Algorithm data Algorithm block
block
Y = E(K, X)

Secret key K
shared by
sender and
recipient
Secure Channel
Key
Source

FIGURE 1.5 Model of Symmetric Cryptosystem

■ Plaintext: The original message or data block that is fed into the algorithm as input.
■ Encryption algorithm: The algorithm that performs various substitutions and transformations
on the plaintext.
■ Secret key: An input to the encryption algorithm. The exact substitutions and transformations
performed by the algorithm depend on the key.
■ Ciphertext: The scrambled message produced as output. It depends on the plaintext and the
secret key. For a given data block, two different keys will produce two different ciphertexts.
■ Decryption algorithm: The inverse of the encryption algorithm. It uses the ciphertext and the
secret key to produce the original plaintext.

There are two requirements for secure use of symmetric encryption:

■ A strong encryption algorithm is required. At a minimum, the algorithm should be such that an
opponent who knows the algorithm and has access to one or more ciphertexts would be unable to
decipher the ciphertext or figure out the key. This requirement is usually stated in a stronger form:
The opponent should be unable to decrypt ciphertext or discover the key even if he or she is in
possession of a number of ciphertexts together with the plaintext that produced each ciphertext.
 1.7 Asymmetric Encryption 17

■ The sender and receiver must have obtained copies of the secret key in a secure fashion and
must keep the key secure. If someone can discover the key and knows the algorithm, all com-
munication using this key is readable.

The generation and distribution of secret keys are essential elements of a symmetric cryptography
scheme. Typically, a key generation algorithm generates a random number and derives a secret key
from that number. For two parties to communicate, there are a number of possibilities for key distri-
bution, including:

■ One party generates the key and securely transfers it to the other party.
■ The two parties engage in a secure key exchange protocol that enables them to jointly generate
a key known only to the two parties.
■ A third party generates the key and securely transfers it to the two communicating parties.

Figure 1.5 illustrates the first alternative. One way to establish a secure channel of communication is
if the two parties already share an older secret key, and the party that generates the key can encrypt the
new key with the older key. Another alternative is the use of public-key cryptography to encrypt the
key. Public-key cryptography is discussed subsequently.
Figure 1.5 also indicates the existence of a potential adversary that seeks to obtain the plaintext. It
is assumed that the adversary can eavesdrop on the encrypted data and knows the encryption and
decryption algorithms that were used.
There are two general approaches an adversary can use to attack a symmetric encryption scheme. One
approach is cryptanalysis. Cryptanalytic attacks rely on the nature of the algorithm plus perhaps some
knowledge of the general characteristics of the plaintext or even some sample plaintext/ciphertext
pairs. This type of attack exploits the characteristics of the algorithm to attempt to deduce a specific
plaintext or to deduce the key being used. If an attack succeeds in deducing the key, the effect is cata-
strophic: All future and past messages encrypted with that key are compromised. The second method,
known as the brute-force attack, involves trying every possible key on a piece of ciphertext until an
intelligible translation into plaintext is obtained. On average, half of all possible keys must be tried
to achieve success. Thus, a secure symmetric encryption scheme requires an algorithm that is secure
against cryptanalysis and a key of sufficient length to defeat a brute-force attack.

1.7 Asymmetric Encryption

Public-key cryptography, also called asymmetric cryptography, involves the use of two separate keys,
in contrast to symmetric encryption, which uses only one key. The use of two keys has profound conse-
quences in the areas of confidentiality, key distribution, and authentication. A public-key encryption
scheme has six ingredients (see part a of Figure 1.6; compare with Figure 1.5):
18 CHAPTER 1 Security and Cryptography Concepts

Alice's
public key
ring
Bob’s private key
Joy
Ted
Mike Bob
Data Bob's Encrypted Data
block public key block block

Encryption Decryption
Algorithm Algorithm

Alice Bob

(a) Public-key encryption/decryption (Alice encrypts block for Bob only)

Bobs's
public key
ring
Alice’s private key
Joy
Ted
Mike Alice
Data Encrypted Data
Alice’s
block block block
public key

Encryption Decryption
Algorithm Algorithm

Alice Bob

(b) Public-key encryption/decryption (Alice authenticates block for any recipient)

FIGURE 1.6 Model of Asymmetric Cryptosystem

■ Plaintext: This is the readable message or data block that is fed into the algorithm as input.
■ Encryption algorithm: The encryption algorithm performs various transformations on the
plaintext.
■ Public key and private key: This is a pair of keys that have been selected so that if one is used
for encryption, the other is used for decryption. The exact transformations performed by the
encryption algorithm depend on the public or private key that is provided as input.
 1.7 Asymmetric Encryption 19

■ Ciphertext: This is the scrambled block produced as output. It depends on the plaintext and the
key. For a given message, two different keys will produce two different ciphertexts.
■ Decryption algorithm: This algorithm accepts the ciphertext and the matching key and pro-
duces the original plaintext.

The essential steps shown in part a of Figure 1.6 are as follows:

1. Each user generates a pair of keys to be used for the encryption and decryption of messages.

2. Each user places one of the two keys in a public register or another accessible file. This is the
public key. The companion key is kept private. As part a of Figure 1.6 suggests, each user main-
tains a collection of public keys obtained from others.
3. If Alice wishes to send a confidential message to Bob, Alice encrypts the message using Bob’s
public key.
4. When Bob receives the message, he decrypts it using his private key. No other recipient can
decrypt the message because only Bob knows Bob’s private key.

The process works (produces the correct plaintext on output) regardless of the order in which the
pair of keys is used. With this approach, all participants have access to public keys, and private keys
are generated locally by each participant and therefore need never be distributed. As long as a user’s
private key remains protected and secret, incoming communication is secure. At any time, a system can
change its private key and publish the companion public key to replace its old public key. Table 1.2
summarizes some of the important aspects of symmetric and asymmetric encryption.

TABLE 1.2 Symmetric and Asymmetric Encryption

Symmetric Encryption Asymmetric Encryption

Needed to Work: Needed to Work:
The same algorithm with the same secret key is One algorithm is used for encryption and a related
used for encryption and decryption. algorithm for decryption, with a pair of keys, known as
the public key and the private key. The two keys can
be used in either order, one for encryption and one for
decryption.
The sender and receiver must share the algo- The sender and receiver must each have a unique
rithm and the secret key. public/private key pair.
Needed for Security: Needed for Security:
The key must be kept secret. The private key must be kept secret.
It must be impossible or at least impractical to It must be impossible or at least impractical to deci-
decipher a message if the key is kept secret. pher a message if the private key is kept secret.
Knowledge of the algorithm plus samples of Knowledge of the algorithm plus the public key plus
ciphertext must be insufficient to determine samples of ciphertext must be insufficient to deter-
the key. mine the private key.
20 CHAPTER 1 Security and Cryptography Concepts

As with symmetric encryption, asymmetric key generation involves the use of a random number. In this
case, the key generation algorithm computes a private key from a random number and then computes
a public key as a function of the private key. Without knowledge of the private key, it is infeasible to
calculate the public key. On the other hand, knowledge of the public key does not enable calculation
of the private key.
Public-key encryption can be used in another way, as illustrated in part b of Figure 1.6. Suppose that Alice
wants to send a message to Bob and, although it isn’t important that the message be kept secret, she wants
Bob to be certain that the message is indeed from her. In this case, Alice uses her private key to encrypt
the message. When Bob receives the ciphertext, he finds that he can decrypt it with Alice’s public key,
thus proving that the message must have been encrypted by Alice: No one else has Alice’s private key,
and therefore no one else could have created a ciphertext that could be decrypted with Alice’s public key.
As with symmetric encryption algorithms, the security of public-key encryption is a function of the
strength of the algorithm and the length of the private key. Public-key cryptographic algorithms are
considerably slower than symmetric algorithms for a given data block length. Accordingly, public-
key cryptography is almost always limited to use with small blocks of data, such as a secret key or, as
discussed subsequently, a hash value.

1.8 Cryptographic Hash Functions

A hash function takes an input of arbitrary length and maps it to a fixed-length data block that is typi-
cally shorter than the input data block. This is therefore a many-to-one function; that is, multiple input
blocks produce the same output. The output is known as the hash value or hash digest. A cryptographic
hash function, also known as a secure hash function, is a hash function with specific properties that are
useful for various cryptographic algorithms, as explained subsequently. Secure hash functions are an
essential element of many security protocols and applications. To be useful for security applications, a
hash function called H must have the properties indicated in Table 1.3.

TABLE 1.3 Requirements for Cryptographic Hash Function H

Requirement Description
Variable input size H can be applied to a block of data of any size.
Fixed output size H produces a fixed-length output.
Efficiency H(x) is relatively easy to compute for any given x, making both hardware
and software implementations practical.
Preimage resistant (one-way For any given hash value h, it is computationally infeasible to find y
property) such that H(y) = h.
Second preimage resistant For any given block x, it is computationally infeasible to find y ≠ x with
(weak collision resistant) H(y) = H(x).
Collision resistant (strong col- It is computationally infeasible to find any pair (x, y) such that
lision resistant) H(x) = H(y).
Pseudorandomness Output of H meets standard tests for pseudorandomness; that is, the
output appears to be a random sequence of bits.
 1.8 Cryptographic Hash Functions 21

Figure 1.7 indicates two common ways in which hash functions are used. Part a of Figure 1.7 illus-
trates the use of a hash function to ensure the data integrity of a block of data, generally referred to
as message authentication. The two important aspects of message authentication are to verify that the
contents of the message have not been altered and that the source is authentic. The hash function can
also verify a message’s timeliness (to see if it has been artificially delayed and replayed) and sequence
relative to other messages flowing between two parties by including timestamps and sequence numbers
in the message.

Source A Destination B
Message

Message

Message
H

Ksecret Ksecret Compare

H

E D

(a) Message authentication using symmetric encryption

Message

Message

Message

V
Valid/
Kprivate invalid
H
Kpublic
S

(b) Digital signature using public-key encryption

E = encryption algorithm S = signing algorithm

D = decryption algorithm V = verifying algorithm
H = hash function

FIGURE 1.7 Uses for a Secure Hash Function

Message authentication using a hash value proceeds as follows. First, generate a hash value for the
source message. Next, encrypt the hash value using a secret key shared by a cooperating partner. Then,
transmit the message plus encrypted hash value to the destination. The recipient decrypts the incoming
encrypted hash value, generates a new hash value from the incoming message, and compares the two
hash values. If only the receiver and the sender know the identity of the secret key, and if the received
code matches the calculated code, then:

■ The receiver is assured that the message has not been altered. If an attacker alters the mes-
sage but does not alter the code, then the receiver’s calculation of the code will differ from the
22 CHAPTER 1 Security and Cryptography Concepts

received code. For a secure hash function, it is infeasible for an attacker to alter the message in
such a way that the hash value is not altered.
■ The receiver is assured that the message is from the alleged sender. Because no one else knows
the secret key, no one else could prepare a message with a proper code.
■ If the message includes a sequence number (as is used with TCP), then the receiver can be assured
of the proper sequence because an attacker cannot successfully alter the sequence number.

A second important use for hash functions is in the digital signature process, explained next.

1.9 Digital Signatures

NIST FIPS 186-4 (Digital Signature Standard) defines a digital signature as follows:

The result of a cryptographic transformation of data that, when properly implemented,

provides a mechanism for verifying origin authentication, data integrity, and signatory
non-repudiation.

Thus, a digital signature is a data-dependent bit pattern, generated by an agent as a function of a

file, message, or other form of data block. Another agent can access the data block and its associated
signature and verify that (1) the data block has been signed by the alleged signer, and (2) the data block
has not been altered since the signing. Further, the signer cannot repudiate the signature.
Part b of Figure 1.7 provides a simplified illustration of the digital signature process. Suppose that Bob
wants to sign a document or message. Although it is not important that the message be kept secret, he
wants others to be certain that the message is indeed from him. For this purpose, Bob uses a secure
hash function to generate a hash value for the message. Together, that hash value and Bob’s private
key serve as input to a digital signature generation algorithm that produces a short block that func-
tions as a digital signature. Bob sends the message with the signature attached. Any other user can
calculate a hash value for the message. The user then inputs that hash value, the attached signature, and
Bob’s public key to a digital signature verification algorithm. If the algorithm returns the result that
the signature is valid, the user is assured that the message must have been signed by Bob. No one else
has Bob’s private key; therefore, no one else could have created a signature that could be verified for
this message with Bob’s public key. In addition, it is impossible to alter the message without access to
Bob’s private key, so the message is authenticated both in terms of source and in terms of data integrity.
The message also has the feature of nonrepudiation. Bob cannot deny having signed the message
because no one else could have done so.
Digital signatures are widely used for a number of purposes, including:

■ Digitally signing email messages to authenticate the sender

■ Digitally signing software programs to authenticate the source of the program and to counter
the threat of software tampering
 1.10 Practical Considerations 23

■ Verifying the authorship or origin of digital data

■ Ensuring the integrity of digital data against tampering
■ Authenticating online entities

1.10 Practical Considerations

This section examines two practical aspects of the use of cryptographic algorithms: the selection of
specific algorithms and the accompanying key lengths and implementation considerations.

Selection of Cryptographic Algorithms and Key Lengths

As processor speeds and capacity have increased, and as cryptographic algorithms are subjected to
increased scrutiny, algorithms that were once considered secure have been abandoned. Similarly, key
lengths and hash value lengths that were once considered secure are now too weak for secure use.
Accordingly, security managers should take care to choose algorithms and lengths to achieve a desired
level of security. A useful source of guidance for algorithm selection is FIPS 140-2A (Approved
Security Functions for FIPS PUB 140-2) and for key and hash length is SP 800-131A (Transitioning
the Use of Cryptographic Algorithms and Key Lengths). Similar recommendations are in the ENISA
report Algorithms, Key Size and Protocol Report [ECRY18].
For symmetric encryption, NIST recommends the use of the Advanced Encryption Standard (AES),
with a key length of 128, 192, or 256 bits. AES is widely accepted worldwide and has become the
standard symmetric encryption algorithm.
For the hash function, NIST recommends one of two NIST standard hash functions: SHA-2 or SHA-3.
The approved hash lengths for both functions range from 224 to 512 bits. The structure and functions
used for SHA-3 are substantially different from those of SHA-2. Thus, if weaknesses are discovered in
either SHA-2 or SHA-3, users have the option to switch to the other standard. SHA-2 has held up well,
and NIST considers it secure for general use. So for now SHA-3 is a complement to SHA-2 rather than a
replacement for it. The relatively compact nature of SHA-3 may make it useful for “embedded” or smart
devices that connect to electronic networks but are not themselves full-fledged computers. Examples
include sensors in a building-wide security system and home appliances that can be controlled remotely.
For digital signatures, NIST recommends three alternative digital signature algorithms:

■ Digital Signature Algorithm (DSA) with length of 2048 bits

■ RSA algorithm with 2048 bits
■ Elliptic-Curve Digital Signature Algorithm with length of 224 bits

SP 800-131A also includes recommendations for random bit generation algorithms, message authenti-
cation codes, key agreement algorithms, and key encryption algorithms.
24 CHAPTER 1 Security and Cryptography Concepts

Implementation Considerations
SP 800-12 (An Introduction to Information Security) lists the following as important management
considerations for implementing cryptography within an organization:

■ Selecting design and implementation standards: It is almost always advisable not to rely on
a proprietary cryptographic algorithm, especially if the algorithm itself is secret. Standardized
algorithms, such as AES, SHA, and DSS, have been subject to intense scrutiny by the profes-
sional community, and managers can have a high degree of confidence that the algorithms
themselves, used with the recommended lengths, are secure. NIST and other organizations
have developed numerous standards for designing, implementing, and using cryptography and
for integrating it into automated systems. Managers and users of systems should choose the
appropriate cryptographic standard based on cost-effectiveness analysis, trends in the standard’s
acceptance, and interoperability requirements.
■ Deciding between hardware, software, and firmware implementations: The trade-offs
among security, cost, simplicity, efficiency, and ease of implementation need to be studied by
managers acquiring various security products meeting a standard.
■ Managing keys: Key management is the process of administering or managing cryptographic
keys for a cryptographic system or application. It involves the generation, creation, protection,
storage, exchange, replacement, and use of keys and enables selective restriction for certain
keys. In addition to access restriction, key management also involves the monitoring and
recording of each key’s access, use, and context. A key management system also includes key
servers, user procedures, and protocols, including cryptographic protocol design. This complex
topic is beyond the scope of this book; see Effective Cybersecurity: A Guide to Using Best
Practices and Standards [STAL19] for a discussion.
■ Security of cryptographic modules: A cryptographic module contains the cryptographic
algorithm(s), certain control parameters, and temporary storage facilities for the key(s) being
used by the algorithm(s). The proper functioning of cryptography requires the secure design,
implementation, and use of the cryptographic module. This includes protecting the mod-
ule against tampering. A useful tool is the NIST Cryptographic Module Validation Program
(CMVP), which validates vendor offerings using independent accredited laboratories. The
validation is against the security requirements in FIPS 140-2 (Security Requirements for Cryp-
tographic Modules). FIPS 104-2 provides a detailed set of requirements at four security levels,
against which vendor hardware, firmware, and software offerings can be evaluated.

Lightweight Cryptographic Algorithms

Two recent areas of strong interest in the field of cryptography are lightweight cryptography and post-
quantum cryptography. It is likely that, in the coming years, a number of new algorithms in both areas will
be widely deployed. In essence, lightweight cryptography is focused on developing algorithms that, while
secure, minimize execution time, memory usage, and power consumption. Such algorithms are suitable for
small embedded systems such as those in wide use in the Internet of Things. Work on lightweight cryptog-
raphy is almost exclusively devoted to symmetric (secret key) algorithms and cryptographic hash functions.
 1.11 Public-Key Infrastructure 25

In 2018, NIST announced a project to solicit designs for lightweight cryptographic algorithms. NIST is
planning to develop and maintain a portfolio of lightweight algorithms and modes that are approved for
limited use. Each algorithm in the portfolio will be tied to one or more profiles, which consist of algo-
rithm goals and acceptable ranges for metrics. NISTIR 8114 (Report on Lightweight Cryptography)
indicates that the initial focus is the development of symmetric encryption and secure hash functions.
NIST has issued a preliminary set of two profiles for these algorithms: one for implementations in both
hardware and software and one for hardware-only implementations.

Post-Quantum Cryptographic Algorithms

Post-quantum cryptography is an area of study that arises from the concern that quantum computers
would be able to break currently used asymmetric cryptographic algorithms. Recent research demon-
strates feasible ways to break the commonly used asymmetric algorithms. Thus, work on post-quantum
cryptography is devoted to developing new asymmetric cryptographic algorithms.
There is no single widely accepted alternative to the existing algorithms currently in use, and
researchers are exploring a number of mathematical approaches. An indication of the interest shown
in these approaches is found in the submissions to the NIST effort at post-quantum standardization.
As reported in NISTIR 8105 (Report on Post-Quantum Cryptography), NIST hopes to standardize a
number of algorithms that can be used to replace or complement existing asymmetric schemes. See
[STAL20] for a study of lightweight and post-quantum cryptographic algorithms.

1.11 Public-Key Infrastructure

A public-key infrastructure (PKI) supports the distribution and identification of public encryption
keys, enabling users and computers to both securely exchange data over networks such as the Internet
and verify the identity of the other party. A PKI is used to bind public keys to entities, enable other
entities to verify public key bindings, revoke such bindings, and provide other services that are critical
to managing public keys.
Before providing an overview of PKI, this section introduces the concept of public-key certificates.

Public-Key Certificates
A public-key certificate is a set of data that uniquely identifies an entity. The certificate contains
the entity’s public key and other data and is digitally signed by a trusted party, called a certification
authority, thereby binding the public key to the entity.
Public-key certificates are designed to provide a solution to the problem of public-key distribution.
Typically, in a public-key scheme, multiple users need to have access to the public key of a given entity
A, whether to encrypt data to send to A or to verify a digital signature signed by A. Each holder of a
public/private key pair could simply broadcast its public key for anyone to read. The problem with this
approach is that it would be easy for some attacker X to impersonate A and to broadcast X’s public key
26 CHAPTER 1 Security and Cryptography Concepts

improperly labeled as A’s public key. To counter this, it would be possible to set up some trusted central
authority that would interact with each user to authenticate and then maintain a copy of A’s public key.
Any other user could then consult the trusted central authority over a secure, authenticated communi-
cation channel to obtain a copy of the key. It should be clear that this solution would not scale efficiently.
An alternative approach is to rely on public-key certificates that can be used by participants to exchange
keys without contacting a public-key authority, in a way that is as reliable as if the keys were obtained
directly from a public-key authority. In essence, a certificate consists of a public key plus an identifier
of the key owner, and the whole block is signed by a trusted third party. Typically, the third party is a
certification authority (CA), such as a government agency or a financial institution, that is trusted by
the user community. A user can present his or her public key to the authority in a secure manner and
obtain a certificate. The user can then publish the certificate. Anyone needing this user’s public key can
obtain the certificate and verify that it is valid by way of the attached trusted signature. A participant
can also convey its key information to another by transmitting its certificate. Other participants can
verify that the certificate was created by the authority.
Figure 1.8 illustrates the overall scheme for generation of a public-key certificate. The certificate for
Bob’s public key includes unique identifying information for Bob, Bob’s public key, identifying infor-
mation about the CA, and certificate information, such as expiration date. This information is then signed
by computing a hash value of the information and generating a digital signature using the hash value and
the CA’s private key. Bob can then either broadcast this certificate to other users or attach the certificate
to any document or data block he signs. Anyone who needs to use Bob’s public key can be assured that
the public key contained in Bob’s certificate is valid because the certificate is signed by the trusted CA.

Bob's ID
Unsigned certificate: information
contains user ID,
user's public key Bob's public key
H

H CA information
Certificate
information Verify algorithm
S V indicates whether
the signature is
Generate hash Signed certificate valid
code of unsigned
certificate

Use hash code of Supply CA's public key

unsigned certificate to the verify algorithm
with CA's private key
to form signature

Create signed Use certificate to

digital certificate verify Bob's public key

FIGURE 1.8 Public-Key Certificate Use

The standard ITU-T X.509 (The Directory: Public-Key and Attribute Certificate Frameworks) has
become universally accepted for formatting public-key certificates.
 1.11 Public-Key Infrastructure 27

PKI Architecture
A PKI architecture defines the organization and interrelationships among CAs and PKI users. PKI
architectures satisfy the following requirements:

■ Any participant can read a certificate to determine the name and public key of the certificate’s owner.
■ Any participant can verify that the certificate originated from the certification authority and is
not counterfeit.
■ Only the certification authority can create and update certificates.
■ Any participant can verify that the certificate is currently valid.

Figure 1.9 provides a typical architecture for a PKI, the essential components of which are defined in
the list that follows:

Public key

Private key Certificates

and certificate Repository
revocation
lists (CRLs)
Request
for issuing
certificate Certification
Registration
Authority (CA)
Authority (RA)

CA’s
Public key
Bob’s
Bob’s
public key
public key
certificate
certificate
Application
for
certificate

Signing
Party
Bob

Bob’s Document
Public key
public key signed with Relying
Private key certificate Bob’s private Party Alice
key

FIGURE 1.9 PKI Scenario

28 CHAPTER 1 Security and Cryptography Concepts

■ End entity: An end user, a device (such as a router or server), a process, or any item that can
be identified in the subject name of a public-key certificate. End entities can also be consumers
of PKI-related services and, in some cases, providers of PKI-related services. For example, a
registration authority is considered to be an end entity from the point of view of the certification
authority.
■ Certification authority (CA): An authority trusted by one or more users to create and
assign public-key certificates. Optionally the certification authority may create the subjects’
keys. A CA digitally signs a public-key certificate, which effectively binds the subject name
to the public key. CAs are also responsible for issuing certificate revocation lists (CRLs).
A CRL identifies certificates previously issued by the CA that are revoked before their
expiration date. A certificate could be revoked because the user’s private key is assumed to
be compromised, the user is no longer certified by this CA, or the certificate is assumed
to be compromised.
■ Registration authority (RA): An optional component that can be used to offload many of the
administrative functions that a CA ordinarily assumes. The RA is normally associated with
the end entity registration process. This includes the verification of the identity of the end entity
attempting to register with the PKI and obtain a certificate for its public key.
■ Repository: Any method for storing and retrieving PKI-related information, such as public-key
certificates and CRLs. A repository can be an X.500-based directory with client access via
Lightweight Directory Access Protocol (LDAP). It also can be something simple, such as
a means for retrieval of a flat file on a remote server via File Transfer Protocol (FTP) or
Hypertext Transfer Protocol (HTTP).
■ Relying party: Any user or agent that relies on the data in a certificate in making decisions.

Figure 1.9 illustrates the interaction of the various components. Consider a relying party Alice that
needs to use Bob’s public key. Alice must first obtain in a reliable, secure fashion a copy of the public
key of the CA. This can be done in a number of ways, depending on the particular PKI architecture
and enterprise policy. If Alice wishes to send encrypted data to Bob, Alice checks with the repository
to determine whether Bob’s certificate has been revoked, and if not, she obtains a copy of Bob’s certif-
icate. Alice can then use Bob’s public key to encrypt data sent to Bob. Bob can also send to Alice a
document signed with Bob’s private key. Bob may include his certificate with the document or assume
that Alice already has or can obtain the certificate. In either case, Alice first uses the CA’s public key
to verify that the certificate is valid and then uses Bob’s public key (obtained from the certificate) to
validate Bob’s signature.
Rather than using a single CA, an enterprise may need to rely on multiple CAs and multiple reposi-
tories. CAs can be organized in a hierarchical fashion, with a root CA that is widely trusted signing the
public-key certificates of subordinate CAs. Many root certificates are embedded in web browsers, so
they have built-in trust of those CAs. Web servers, email clients, smartphones, and many other types
of hardware and software also support PKI and contain trusted root certificates from the major CAs.
 Chapter 2
Information Privacy Concepts
Learning Objectives
After studying this chapter, you should be able to:
■ Explain the difference between privacy by design and privacy engineering
■ Understand how privacy-related activities fit into the system development life cycle
■ Define privacy control
■ Discuss the areas of overlap between security and privacy and the areas that are distinct
to either security or privacy
■ Explain the trade-off between privacy and utility
■ Explain the distinction between privacy and usability

This chapter provides a roadmap for the remainder of the book, introducing the key information
privacy concepts and indicating how they relate to one another. The chapter begins by defining key
terms in the field of information privacy. Then, Sections 2.2 and 2.3 introduce the concepts of privacy
by design and privacy engineering. Sections 2.4 through 2.6 deal with the relationship between privacy
and security, the trade-off between privacy and utility, and the concept of usable privacy.

2.1 Key Privacy Terminology

The term privacy is used frequently in ordinary language as well as in philosophical, political, and
legal discussions. However, there is no single definition or analysis or meaning of the term; a good
survey of this topic is the privacy entry in the Stanford Encyclopedia of Philosophy [DECE18]. Two
general characteristics of privacy are the right to be left alone—that is, free from being observed or
disturbed—and the ability to control the information released about oneself.

32
 2.1 Key Privacy Terminology 33

This book is concerned with a concept of privacy referred to as information privacy. ITU-T Recom-
mendation X.800 (Security Architecture for Open Systems Interconnection) defines privacy as the right
of individuals to control or influence what information related to them may be collected and stored and
by whom and to whom that information may be disclosed. A U.S. National Research Council report
(At the Nexus of Cybersecurity and Public Policy: Some Basic Concepts and Issues) [CLAR14] indi-
cates that in the context of information, the term privacy usually refers to making ostensibly private
information about an individual unavailable to parties that should not have that information. Privacy
interests attach to the gathering, control, protection, and use of information about individuals.
Information privacy generally pertains to what is known as personally identifiable information (PII),
as opposed to, say, video surveillance. PII is information that can be used to distinguish or trace an
individual’s identity. NIST SP 80-122 (Guide to Protecting the Confidentiality of Personally Identi-
fiable Information) gives the following examples of information that might be considered PII:

■ Name, such as full name, maiden name, mother’s maiden name, or alias
■ Personal identification number, such as Social Security number (SSN), passport number,
driver’s license number, taxpayer identification number, patient identification number, and
financial account or credit card number
■ Address information, such as street address or email address
■ Asset information, such as Internet Protocol (IP) or media access control (MAC) address or
other host-specific persistent static identifier that consistently links to a particular person or to a
small, well-defined group of people
■ Telephone numbers, including mobile, business, and personal numbers
■ Personal characteristics, including photographic images (especially of the face or other distin-
guishing characteristic), x-rays, fingerprints, or other biometric image or template data
(e.g., retinal scan, voice signature, facial geometry)
■ Information identifying personally owned property, such as vehicle registration number or title
number and related information
■ Information about an individual that is linked or linkable to one of the above (e.g., date of birth,
place of birth, race, religion, weight, activities, geographic indicators, employment information,
medical information, education information, financial information)

In dealing with the privacy of PII, two new concepts have emerged: privacy by design (PbD) and privacy
engineering. The goal of privacy by design is to take privacy requirements into account throughout the
system development process, from the conception of a new IT system through detailed system design,
implementation, and operation. ISO 29100 (Information Technology—Security Techniques—Privacy
Framework) views PbD as the practice of considering privacy safeguarding measures at the time of the
design of the system; that is, designers should consider privacy compliance during the design phase for
systems processing PII rather than address compliance only at a subsequent stage.
66 CHAPTER 3 Information Privacy Requirements and Guidelines

■ Individual participation: As with the principle of openness, this principle implies that there
is a user-friendly process for individual participation. Participation includes the rights to obtain
confirmation of whether the data controller has one’s personal data; have one’s personal data
communicated to him or her; and challenge data relating to him or her and have data modified
or erased, if appropriate.
■ Accountability: This principle dictates that the individual (data controller) who is in charge of
determining how personal data are going to be processed and used will be held responsible for
ensuring that the data are processed in an authorized, fair, and legitimate manner. Of course,
breaches of privacy will raise issues of accountability not only for the data controller but for the
organization as a whole. Accountability refers to accountability supported by legal sanctions, as
well as to accountability established by codes of conduct and contractual obligations.

3.4 Privacy Regulations

A number of national governments have introduced laws and regulations intended to protect indi-
vidual privacy in information processing. There is a great deal of overlap and similarity across different
nations. This section provides two regional examples that are by far the most influential and that
encompass virtually all of the requirements imposed in other nations. Part VI, “Legal and Regulatory
Requirements,” examines these two examples in detail.

European Union
One of the most comprehensive initiatives is the European Union (EU) General Data Protection Regu-
lation (GDPR), approved by the EU Parliament in 2016, with an effective enforcement date of May
2108. The GDPR is designed to harmonize data privacy laws across Europe, to protect and empower
the data privacy of all EU citizens, and to reshape the way organizations, both public and private,
across the region approach data privacy.
The presentation “10 Key Facts Businesses Need to Note About the GDPR” from the 2016 European
Identity & Cloud Conference [KINA16] summarizes important aspects of GDPR that organizations
that do business in Europe need to be aware of:

■ The GDPR applies to all companies worldwide that process personal data of EU residents,
both EU citizens and non-citizens. Any company that works with information relating to EU
residents must comply with the requirements of the GDPR, making it the first global data
protection law. This aspect alone is contributing significantly to all companies around the world
taking data privacy more seriously.
■ The GDPR widens the definition of personal data compared to prior regulations by EU member
states. As a result, parts of IT that have been unaffected by data protection laws in the past will
need attention from businesses to ensure their compliance with the new regulation.
 3.4 Privacy Regulations 67

■ The GDPR tightens the rules for obtaining valid consent to using personal information. Having
the ability to prove valid consent for using personal information is likely to be one of the big-
gest challenges presented by the GDPR. The GDPR states that the consent of the data subject
means any freely given, specific, informed, and unambiguous indication of his or her wishes
by which the data subject, either by a statement or by a clear affirmative action, signifies agree-
ment to personal data relating to him or her being processed.
■ The GDPR requires public authorities processing personal information to appoint a data protec-
tion officer (DPO), as well as other entities, when core activities require regular and systematic
monitoring of data subjects on a large scale or consist of processing on a large scale of special
categories of data.
■ The GDPR mandates data protection impact assessments. Data controllers must conduct
assessments where privacy breach risks are high in order to minimize risks to data subjects.
This means before organizations can implement projects involving personal information,
they must conduct a privacy risk assessment and work with the DPO to ensure that they are
in compliance as projects progress. Chapter 11 covers data protection impact assessments in
detail.
■ The GDPR requires organizations to notify the local data protection authority of a data breach
within 72 hours of discovering it. This means organizations need to ensure that they have
technologies and processes in place that will enable them to detect and respond to a data
breach.
■ The GDPR introduces the right to be forgotten. Also known as data erasure, the right to be
forgotten entitles the data subject to have the data controller erase his/her personal data, cease
further dissemination of the data, and potentially have third parties halt processing of the data.
The conditions for erasure include the data no longer being relevant to original purposes for
processing or a data subject withdrawing consent. This means organizations will have to get
fresh consent before they can alter the way they are using the data they have collected. It also
means organizations have to ensure that they have the processes and technologies in place to
delete data in response to requests from data subjects.
■ The GDPR requires that privacy be included in systems and processes by design. At its core,
privacy by design—referred to as data protection by design in the GDPR—calls for the inclu-
sion of data protection from the onset of the designing of systems rather than an addition.

The GDPR is an important landmark in the evolving integration of privacy in cybersecurity. Even orga-
nizations unaffected by this regulation should be aware of its provision and consider them in designing
their own privacy controls. Chapter 14 examines the GDPR in detail.

U.S. Privacy Laws and Regulations

There is no single law or regulation covering privacy in the United States. Rather, a collection of
federal privacy laws cover various aspects of privacy. Some U.S. privacy laws apply only to federal
68 CHAPTER 3 Information Privacy Requirements and Guidelines

agencies and contractor companies working under federal contract. Others impose mandates on private
organizations as well as government agencies and departments. These include:

■ The Privacy Act of 1974: Specifies the rules that a federal agency must follow to collect, use,
transfer, and disclose an individual’s PII.
■ The Fair and Accurate Credit Transaction Act of 2003 (FACTA): Requires entities engaged
in certain kinds of consumer financial transactions (predominantly credit transactions) to be
aware of the warning signs of identity theft and to take steps to respond to suspected incidents
of identity theft.
■ The Health Insurance Portability and Accountability Act of 1996 (HIPAA): Requires cov-
ered entities (typically medical and health insurance providers and their associates) to protect
the security and privacy of health records.
■ The Family Educational Rights and Privacy Act of 1974 (FERPA): Protects students and
their families by ensuring the privacy of student educational records.
■ The Gramm Leach Bliley Act of 1999 (GLBA): Imposes privacy and information security
provisions on financial institutions; designed to protect consumer financial data.
■ Federal Policy for the Protection of Human Subjects: Published in 1991 and codified in sep-
arate regulations by 15 federal departments and agencies, outlines the basic ethical principles
(including privacy and confidentiality) in research involving human subjects.
■ The Children’s Online Privacy Protection Act (COPPA): Governs the online collection of
personal information from children under the age of 13.
■ The Electronic Communications Privacy Act: Generally, prohibits unauthorized and inten-
tional interception of wire and electronic communications during the transmission phase and
unauthorized accessing of electronically stored wire and electronic communications.

In addition, there are numerous state laws and regulations that impact business. By far, the most
important of these is the California Consumer Privacy Act (CCPA). Chapter 15 examines the most
important federal privacy laws as well as the CCPA.

3.5 Privacy Standards

The management, design, and implementation of privacy safeguards in information systems is complex
and difficult. A wide variety of technologies are involved. These include cybersecurity mechanisms,
such as cryptography, network security protocols, operating system mechanisms, database security
schemes, and malware identification, which are discussed in Part III, “Technical Security Controls
for Privacy.” Also included are privacy-specific mechanisms, which are described in Part IV, “Privacy
Enhancing Technologies.” The areas of concern are broad, including stored data, data communications,
human factors, physical asset and property security, and legal and regulatory concerns. And there is
 Chapter 14
The EU General Data Protection
Regulation

Learning Objectives
After studying this chapter, you should be able to:
■ Understand the organization of the GDPR and its supporting guideline documents
■ Explain the principles of the GDPR
■ Explain the rights of data subjects defined in the GDPR
■ Summarize the roles of the controller and processor
■ Present an overview of the data protection impact assessment process

The General Data Protection Regulation (GDPR) is a European Commission regulation for the protec-
tion of data in the European Union (EU). The European Commission published the GDPR on April 27,
2016, and it entered into force on May 25, 2018. The GDPR builds on the foundations of and repeals
its predecessor, Directive 95/46/EC, which had provided the basis for EU member states’ data protec-
tion laws. As a regulation instead of a directive, the GDPR enters directly into force without the need
for implementation in law by the member states. This regulation makes data protection stronger and
more specific in some areas while expanding the rights of consumers over their data. Thus, while many
of the provisions look familiar, taken together, they radically change the impact of data protection
within the EU.
The GDPR is enforceable in all the nations in the European Economic Area (EEA), which consists of
the 28 member states of the EU, plus Iceland, Liechtenstein, and Norway. The GDPR also regulates the
flow of personal data outside the EU. Its main objective is to protect the privacy of citizens of the EU
and unify the data regulation rules of the EU’s member nations. Its rules apply to private organiza-
tions and government agencies, including the police and military. Mandates in the GDPR apply to all
personal data of EU citizens, whether or not the organization collecting the data in question is located

412
 14.1 Key Roles and Terms in the GDPR 413

within the EU, as well as all people whose data is stored within the EU, whether or not they are actually
EU citizens.

Note
The official designation of the GDPR is Regulation (EU) 2016/679 of the European Parliament and
of the Council of 27 April 2016.

This chapter focuses on areas of the GDPR that are of particular interest to information privacy
designers and implementers. The first three sections provide an overview, discussing key roles and
terms in the GDPR, the structure of the GDPR document, and the objectives and scope of the GDPR.
Sections 14.4 through 14.6 focus on the rights of data subjects, including a description of the principles
used to define these rights, restrictions on collecting and processing certain types of personal data, and
the specific rights of data subjects that are protected by the GDPR. The remaining sections detail the
operational and management requirements imposed by the GDPR. Section 14.7 looks at the roles and
responsibilities of the controller, the processor, and the data protection officer. Section 14.8 deals with
the GDPR requirements for a data protection impact assessment.

14.1 Key Roles and Terms in the GDPR

A number of actors or roles are defined in the GDPR, including the following:

■ Natural person: A human being.

■ Legal person: A non-human entity, such as a corporation, partnership, or sole proprietorship,
that is recognized as having privileges and obligations, such as having the ability to enter into
contracts, to sue, and to be sued. Sometimes the term legal person encompasses natural per-
sons and well as non-human entities, but the GDPR limits the term legal person to non-human
entities.
■ Data subject: An identified or identifiable natural person, which is one who can be identified,
directly or indirectly, in particular by reference to an identifier such as a name, an identifica-
tion number, location data, an online identifier, or one or more factors specific to the physical,
physiological, genetic, mental, economic, cultural, or social identity of that natural person.
■ Controller: The natural or legal person, public authority, agency, or other body that determines
the purposes and means of processing personal data, regardless of whether such data are col-
lected, stored, processed, or disseminated by that party or by an agent on its behalf.
■ Processor: The natural or legal person, public authority, agency, or other body responsible for
processing personal data on behalf of and in accordance with the instructions of a controller.
The controller and the processor may be the same entity.
414 CHAPTER 14 The EU General Data Protection Regulation

■ Third party: A natural or legal person, public authority, agency, or body other than the data
subject, controller, processor, and persons who, under the direct authority of the controller or
processor, are authorized to process personal data.
■ Data protection officer (DPO): An independent member of the privacy team who reports
directly to senior management. The responsibilities of the DPO include:
■ Assisting the controller or the processor in monitoring internal compliance with the
GDPR
■ Providing advice where requested regarding to the data protection impact assessment and
monitoring its performance
■ Cooperating with the supervisory authority and acting as a contact point. A supervisory
authority is a government entity with the authority to enforce the GDPR
■ Prioritizing activities and focusing efforts on issues that present increased data protection
risks
■ Creating inventories and holding a register of processing operations based on information
provided by the various departments in the organization responsible for the processing of
personal data
■ Supervisory authority: An independent public authority established by a EU member state
that is responsible for monitoring the application of the GDPR. Some, but not all, EU mem-
ber states refer to this body as a data protection authority (DPA), although this term does not
appear in the GDPR.

The GDPR uses some terms that differ from those that were heretofore common in the information
privacy literature and regulations. Table 14.1 shows GDRP terms and equivalent terms in common use.

TABLE 14.1 Key GDPR Terms

GDPR Term Equivalent Term

Data controller PII controller
Data processor PII processor
Data protection Information privacy
Data protection by default Privacy by default
Data protection by design Privacy by design
Data protection impact assessment Privacy impact assessment
Data protection officer Chief privacy officer, privacy leader
Data subject PII principal
Personal data PII
Personal data breach Privacy breach, privacy violation
 14.2 Structure of the GDPR 415

14.2 Structure of the GDPR

The GDPR consists of a set of 99 articles, organized into 11 chapters. These 99 articles are the specific
rules put forth in the regulation. In addition, the GDPR document includes 173 recitals, which provide
commentary and additional explanation of the GDPR. Each recital, with the exception of recital 172, is
associated with one or more specific articles.

Note
In the English-language version of the regulation, the articles occupy 57 pages, and the recitals
occupy 31 pages.

Table 14.2 summarizes the topics covered by each chapter and indicates the corresponding articles and
recitals.

TABLE 14.2 Structure of the GDPR

Chapter Description Articles Recitals

1: General Provisions Defines the regulation objectives, 1–4 1–37
scope with respect to personal data,
territorial scope, and terminology.
2: Principles Discusses how an organization 5–11 38–57
should treat personal data and how
the person who is processing the
data has to demonstrate compli-
ance. This chapter also brings in
consent, categories for personal
data, and when processing does not
require identification.
3: Rights of the Data Explains the rights of the person 12–23 58–73
Subject whose data are handled by the pro-
cessor, controller, or someone who
receives the data.
4: Controller and Deals with a number of procedural 24–43 13, 39, 74–100
Processor issues. Covers the roles of control-
ler, processor, and data protection
officer. Mandates use of data protec-
tion by design and data protection
impact assessment. Also outlines
requirements for notification of a
personal data breach.
5: Transfer or Personal Deals with transfer of personal data 44–50 101–116
Data to Third Countries or to a third country (a country outside
International Organizations the EU or EAA) or to an international
organization.
416 CHAPTER 14 The EU General Data Protection Regulation

Chapter Description Articles Recitals

6: Independent Focuses on requirements and man- 51–59 117–132
Supervisory Authorities dates for EU member states.
7: Cooperation and Discusses how supervisory authori- 60–76 124–128,
Consistency ties can remain consistent and 130–131,
cooperate with one another. This 133–140
chapter also defines the purpose of
the European Data Protection Board
and discusses its purpose.
8: Remedies, Liability and Reviews the rights of data subjects 77–84 141–152
Penalties and how they proceed with
complaints. This chapter also
covers penalties for processors and
controllers.
9: Provisions Relating Discusses how member states can 85–91 153–165
to Specific Processing provide exemptions, conditions, or
Situations rules in relation to specific process-
ing activities.
10: Delegated Acts and Discusses the EU Commission’s 92–93 166–170
Implementing Acts power to adopt delegated acts and
the process in which that occurs.
This refers to acts that are not spe-
cifically part of EU legislation.
11: Final Provisions Discusses how the EU Commission 94–99 102, 171, 173
must report on the regulation every
four years. This chapter also dis-
cusses the differences between pre-
vious directives and the GDPR.

To provide more detailed guidance, the Article 29 Data Protection Working Party issued a number of
documents. This advisory body was made up of a representative from the data protection authority of
each EU member state, the European Data Protection Supervisor, and the European Commission. In
2018, it was replaced by the European Data Protection Board (EDPB) under the GDPR. The EDPB has
issued additional guidance documents. With respect to the focus of this book, the most relevant docu-
ments are the following:

■ Guidelines 2/2019 on the Processing of Personal Data Under Article 6(1)(b) of the GDPR in
the Context of the Provision of Online Services to Data Subjects
■ Guidelines 3/2018 on the Territorial Scope of the GDPR (Article 3)
■ Guidelines on Transparency Under Regulation 2016/679 (wp260rev.01)
■ Guidelines on Automated Individual Decision-Making and Profiling for the Purposes of Regu-
lation 2016/679 (wp251rev.01)
 14.3 GDPR Objectives and Scope 417

■ Guidelines on Personal Data Breach Notification Under Regulation 2016/679 (wp250rev.01)

■ Guidelines on Consent under Regulation 2016/679 (wp259rev.01)
■ Guidelines on the Lead Supervisory Authority (wp244rev.01)
■ Guidelines on Data Protection Officers (“DPOs”) (wp243rev.01)
■ Guidelines on the Right to “Data Portability” (wp242rev.01)
■ Guidelines on Data Protection Impact Assessment (DPIA) (wp248rev.01)

14.3 GDPR Objectives and Scope

Chapter 1 of the GDPR lays out the objectives for this regulation and the scope of its applicability.

Objectives
The GDPR has the following key objectives:

■ Provide the fundamental right to the protection of personal data for every individual
■ Harmonize the protection of fundamental rights and freedoms of natural persons in
respect of processing activities and to ensure the free flow of personal data between mem-
ber states
■ Balance privacy rights against other fundamental rights, in accordance with the principle of
proportionality
■ Define a strong and more coherent data protection framework in the EU, backed by strong
enforcement, given the importance of creating the trust that will allow the digital economy to
develop across the internal market
■ Enable, to the extent possible, natural persons to have control of their own personal data
■ Ensure consistent and homogenous application of the rules for the protection of the funda-
mental rights and freedoms of natural persons with regard to the processing of personal data
throughout the European Union
■ Strengthening and setting out in detail the rights of data subjects and the obligations of those
who process and determine the processing of personal data, as well as equivalent powers for
monitoring and ensuring compliance with the rules for the protection of personal data and
equivalent sanctions for infringements in the member states
■ Take account of the specific needs of micro, small, and medium-sized enterprises in the appli-
cation of the GDPR
418 CHAPTER 14 The EU General Data Protection Regulation

Scope of the GDPR

The GDPR defines both a material scope and a territorial scope for its application. These terms can be
defined as follows:

■ Material scope: The actions covered by a particular law or regulation. In the context of this
chapter, material scope refers to the types of processing of personal data that are covered by the
GDPR.
■ Territorial scope: The jurisdictional reach of a law or regulation. In the context of this chapter,
territorial scope refers to what physical locations of enterprises and data subjects are covered by
the GDPR.

Material Scope
The GDPR applies to the processing of personal data wholly or partly by automated means and to the
processing other than by automated means of personal data that form part of a filing system or are
intended to form part of a filing system. Recital 15 clarifies this as follows:

In order to prevent creating a serious risk of circumvention, the protection of natural per-
sons should be technologically neutral and should not depend on the techniques used. The
protection of natural persons should apply to the processing of personal data by automated
means, as well as to manual processing.

The GDPR defines personal data as:

Any information relating to an identified or identifiable natural person (data subject); an

identifiable natural person is one who can be identified, directly or indirectly, in particular
by reference to an identifier such as a name, an identification number, location data, an
online identifier or to one or more factors specific to the physical, physiological, genetic,
mental, economic, cultural or social identity of that natural person.

The regulation defines a filing system as “any structured set of personal data which are accessible
according to specific criteria, whether centralized, decentralized or dispersed on a functional or geo-
graphical basis.” One example is chronologically ordered sets of manual records containing personal
data; in this case, records are accessible by date.
There are four key elements to the GDPR definition of personal data:

■ Any information: In essence, the GDPR considers any data that can be used to identify
an individual as personal data. It includes, for the first time, things such as genetic, mental,
 14.3 GDPR Objectives and Scope 419

cultural, economic, and social information. There may be a wide variety in terms of the nature
and content of the information, as well as its technical format.
■ Relating to: This phrase implies that the regulation applies to information that relates to an
individual on the basis of its content, purpose, or result. This phrase also covers information
that may have an impact on the way in which an individual is treated or evaluated.
■ Identified or identifiable: Recital 26 states that to determine whether a person is identifiable,
account should be taken of all the means reasonably likely to be used to identify the person.
To ascertain whether means are reasonably likely to be used to identify the natural person,
account should be taken of all objective factors, such as the costs of and the amount of time
required for identification, taking into consideration the available technology at the time of the
processing and technological developments. Thus, personal data include not only data that by
itself identifies a person but also data that, together with other available information and means,
can be used to identify the person. The principles of protection do not apply to data rendered
anonymous in such a way that the data subject is no longer identifiable.
■ Natural person: Personal data apply to natural persons, not to non-human legal persons.

In the United States and many other countries, a somewhat restrictive view is taken of PII, which fre-
quently focuses on whether the data is actually linked to an identified person. However, EU privacy
laws and regulations, culminating in the GDPR, are more expansive and broadly define PII to encom-
pass all data that can be used to make a person identifiable. As pointed out in “Reconciling Personal
Information in the United States and European Union,” from the California Law Review [SCHW14],
in the EU interpretation, “even if the data alone cannot be linked to a specific individual, if it is rea-
sonably possible to use the data in combination with other information to identify a person, then the
information is PII.”
The regulation does not apply in certain cases, such as in some cases related to national security, for-
eign policy, or certain law enforcement activities of member states. In addition, the regulation does not
apply to a natural person in the course of a purely personal or household activity.

Territorial Scope
The processing of EU residents’ data anywhere in the world is subject to the GDPR. The GDPR applies
to organizations that are established solely outside EU territory if they “(i) offer goods or services to
EU residents; or (ii) monitor the behavior of EU residents.” Table 14.3 illustrates the territorial scope
requirement. Note that a business unit within an organization needs to consider whether any other
entity—whether in the same organization or another organization—with which it collaborates in the
processing of data is covered by the GDPR.
420 CHAPTER 14 The EU General Data Protection Regulation

TABLE 14.3 Territorial Scope of the GDPR

Who Doing What Whose Processed Where For What

Personal Data
Data controller or Processing Belonging to Within or outside the In the context of
processor estab- personal data natural persons EU the activities of the
lished in the EU establishment
Data controller or Processing Belonging to Within or outside the Related to offering
processor estab- personal data natural persons EU goods/services (paid
lished outside who are in the or free) OR Relating to
the EU EU monitoring the behavior
of natural persons tak-
ing place within the EU
Data controller Processing Belonging to Outside the EU, but Diplomatic mission or
established out- personal data natural persons member state law consular position
side the EU applies because of
public international law

The subject of territorial scope is a complex one. Two useful sources are the EDPB document
Guidelines 3/2018 on the Territorial Scope of the GDPR (Article 3) and “What Does Territorial Scope
Mean Under the GDPR?” from The Privacy Advisor [KISH18].

14.4 GDPR Principles

The GDPR is built on a foundation of principles and rights (see Figure 14.1). The principles drive
the development of an organization’s privacy policies and controls, and the rights of data subjects
define the constraints on the collection, processing, and storage of personal data. This section examines
GDPR principles, and Section 14.5 discusses rights of data subjects.

GDPR

Informed
Fair, lawful Purpose about data Access
transparent limitation collection
Data Accuracy Rectification Erasure
minimization

Data retention Data Restrict Data

periods security processing portability
Decision
Accountability Object making and
profiling

Principles Rights

FIGURE 14.1 Foundations of GDPR

 14.4 GDPR Principles 421

Chapter 2 of the GDPR defines a set of principles that govern the processing of personal data. These
principles are similar to the fair information practice principles (FIPPs) set forth by the OECD (see
Table 3.2 in Chapter 3, “Information Privacy Requirements and Guidelines”) and consist of the
following:

■ Fair, lawful, and transparent processing: The requirement to process personal data fairly and
lawfully is extensive. It includes, for example, an obligation to tell data subjects what their per-
sonal data will be used for.
■ Purpose limitation: Personal data collected for one purpose should not be used for a new,
incompatible, purpose. Further processing of personal data for archiving, scientific, historical,
or statistical purposes is permitted, subject to appropriate laws and regulations.
■ Data minimization: Subject to limited exceptions, an organization should process only per-
sonal data that it actually needs to process in order to achieve its processing purposes.
■ Accuracy: Personal data must be accurate and, where necessary, kept up to date. Every reason-
able step must be taken to ensure that personal data that are inaccurate are either erased or recti-
fied without delay.
■ Storage limitation: Personal data must be kept in a form that permits identification of data
subjects for no longer than is necessary for the purposes for which the data were collected or
for which they are further processed. Data subjects have the right to erasure of personal data, in
some cases sooner than the end of the maximum retention period.
■ Integrity and confidentiality: Technical and organizational measures must be taken to protect
personal data against accidental or unlawful destruction or accidental loss, alteration, unauthor-
ized disclosure, or access
■ Accountability: The controller is obliged to demonstrate that its processing activities are com-
pliant with the data protection principles.

This section elaborates on some key aspects of these principles.

Fairness
Organization should collect and process data in the spirit of fairness to the data subjects. The GDPR
does not explicitly indicate what is meant by the term fair. The EDPB document Guidelines on
Automated Individual Decision-Making and Profiling for the Purposes of Regulation 2016/679
(wp251rev.01) discusses the potential for profiling to be unfair and create discrimination, such as by
denying people access to employment opportunities, credit, or insurance or by targeting them with
excessively risky or costly financial products. Under the GDPR, profiling is any automated evalua-
tion of a natural person, especially when the goal of the processing is predictive or used for targeting
purposes. Profiling works by creating derived or inferred data about individuals that has not been
422 CHAPTER 14 The EU General Data Protection Regulation

provided directly by the data subjects themselves. Individuals have different levels of comprehen-
sion and may find it challenging to understand the complex techniques involved in profiling and
automated decision-making processes.
The document gives the example of a data broker that sells consumer profiles to financial compa-
nies without consumer permission or knowledge of the underlying data. The data broker puts indi-
viduals into categories based on their personal data and may put them at an unwarranted financial
disadvantage.
In general, processing that may cause injury to an individual or a group of individuals may be unfair.
For example, a resume-aggregating service may collect and use an individual’s gender as a factor
in matching applicants to employers, but the algorithm used may be inadvertently biased against
women.
Another aspect of the principle of fairness relates to the reasonable expectations of the data subject.
This includes the data subjects understanding the possible adverse consequences of personal data that
are processed for a given purpose (Recital 47). It also includes that the controller should have due
regard to the relationship and potential effects of imbalance between the data subject and the controller
with respect to processing personal data for purposes other than the original one (Recital 50).

Lawful
The GDPR requires an organization to identify a legal basis for processing at the time of collection,
before processing occurs, and must furnish the data subject with both the purpose of the processing and
its legal basis at the time data are collected. Article 6 lays out six different legal bases that satisfy the
lawfulness requirement. At least one of the following must apply:

■ The data subject has given informed consent to the processing of his or her personal data for
one or more specific purposes. Note that consent is only one alternative for lawfulness. WP259
(Guidelines on Consent Under Regulation 2016/679) provides a detailed discussion of this
topic.
■ Processing is necessary for the performance of a contract to which the data subject is party or in
order to take steps at the request of the data subject prior to entering into a contract.
■ Processing is necessary for compliance with a legal obligation to which the controller is
subject.
■ Processing is necessary in order to protect the vital interests of the data subject or of another
natural person.
■ Processing is necessary for the performance of a task carried out in the public interest or in the
exercise of official authority vested in the controller.
 14.5 Restrictions on Certain Types of Personal Data 423

■ Processing is necessary for the purposes of the legitimate interests pursued by the controller
or by a third party, except where such interests are overridden by the interests or fundamental
rights and freedoms of the data subject, which require protection of personal data, in particular
where the data subject is a child.

Transparency
Transparency implies that the organization provide data subjects with information about the processing
of their personal data that complies with the following rules:

■ It must be concise, intelligible, and easily accessible.

■ Clear and plain language must be used.
■ The requirement for clear and plain language is of particular importance when providing infor-
mation to children.
■ It must be in writing or by other means, including, where appropriate, by electronic means.
■ Where requested by the data subject, it may be provided orally.
■ It must be provided free of charge.

The EDPB document Guidelines on Transparency Under Regulation 2016/679 (wp260rev.01) gives a
number of examples that satisfy the transparency requirement, including discussing privacy notices on
websites. A link to this privacy statement/notice should be clearly visible on each page of a website
under a commonly used term (such as Privacy, Privacy Policy, or Data Protection Notice). Positioning
or color schemes that make a text or link less noticeable or hard to find on a web page are not consid-
ered easily accessible.

14.5 Restrictions on Certain Types of

Personal Data
The GDPR imposes two special restrictions on the types of personal data that may be collected and
processed: children’s personal data and special categories of personal data.

Children’s Personal Data

The GDPR provides enhanced protection for children under the age of 16 because, compared to adults,
they may be less aware of the risks, consequences, and safeguards concerned and of their rights in
relation to the processing of personal data. Article 8 of the GDPR applies when both of the following
conditions are met:

■ The processing is based on consent (i.e., the processing is based on the first of the alternative
conditions for lawfulness described in Section 14.4).
64 Roger Taylor

II. Describing the Harms from Loss of Privacy

Ideas of privacy and the harms associated with a loss of privacy have changed over
time and in response to technological developments. In antiquity, private matters
were those areas of life over which the public and the state had limited or no legiti-
mate remit. Aristotle’s distinction between the household and the state is often
cited as an early formulation of this view.1 A more developed idea in the same vein
is John Stuart Mill’s view that there are areas of life where the intrusion of law or
public censure can only reduce human happiness—areas where the individual’s
autonomy and individuality should not just be recognised but encouraged.2 It
remains an important idea today and has been used in court to limit government
interference in matters of family planning and sexual relations.3
The idea that privacy was about control over information developed in response
to new technology. In 1890, Brandeis and Warren’s proposals for a right to privacy4
under US law was prompted by fears of: ‘the too enterprising press, the photogra-
pher, or the possessor of any other modern device for recording or reproducing
scenes or sounds’ which had, they said, created circumstances in which ‘gossip
is no longer the resource of the idle and of the vicious, but has become a trade’.
Brandeis and Warren sought a legal basis for protection against the sharing of
information that falls short of being slanderous but where the subject has a right
to protection from ‘the effect of the publication upon his estimate of himself and
upon his own feeling’.
Seventy years later, William Prosser reviewed the legal use of privacy5 and found
that press intrusion remained a central concern for the US courts. He also identi-
fied another issue. Alongside protection from intrusion, embarrassment, or being
placed in a false light, he found that the courts had recognised the right to be pro-
tected against the ‘appropriation of someone’s name or likeness’.
Prosser’s examples of this include the use of someone’s pictures in an advertise-
ment without permission or adopting a name in order to fraudulently pose as
someone’s relative. Identity theft was a rare event in his day but with the rise of
digital technology, it has become a constant and daily concern for anyone engaged
in online activity.
Lack of privacy has been linked to a variety of problems throughout history.
However, these problems have little else that connects them. The danger of the
state legislating about my sex life, the danger of press intrusion and the danger of
my credit card details being stolen online have little in common apart from the link

1 Aristotle Politics.
2 John Stuart Mill, On Liberty (1869).
3 e.g. Griswold v. Connecticut (1965) 381 U.S. 479 on contraception or Roe v. Wade (1973) 410 U.S.

113 on abortion both reference privacy.

4 Samuel D. Warren and Louis D. Brandeis, ‘The Right to Privacy’ (1890) Harvard Law Review,

Vol. 4, No. 5, pp. 193–220.

5 W Prosser, ‘Privacy’ (1960) California Law Review 48: 383–423.
 No Privacy without Transparency 65

to privacy. For that reason, the mechanism used to protect against these harms—
constitutional limitations on the legislature, press regulation, data protection
laws—have nothing in common apart from their connection to the idea of privacy.
The rise of digital technology and artificial intelligence is creating a new set of
potential harms that can arise from the misuse of personal information. The fact
that these concerns are discussed under the heading of ‘privacy’ does not imply
that the remedy will have anything in common with the mechanisms we have used
to protect against previous dangers.
Facebook stated in evidence to the FTC in 2010 ‘Given the vast differences
between Justice Brandeis’s conception of privacy and the way the concept applies
to users on the social web, privacy cannot be viewed in one static way across every
interaction that a user might have. Instead, an effective framework for privacy on
the social web must focus on users’ expectations, which depends on the nature and
context of the relationships that users have with the companies and other services
with which they interact’.6

A. Public Perceptions of the Privacy Related Harm

Public views of the dangers of sharing information with online services have been
extensively researched in the US, Europe and elsewhere. In testimony to a con-
gressional inquiry,7 Professor Alan Westin summarises the US polling evidence as
follows: ‘we have concern about privacy, but also a desire to enjoy the benefits of a
consumer society, and the question is, how do Americans divide in those balances
between those two values?’
Polling in the UK has yielded similar conclusions—that people are concerned
about sharing data and the risks to data security; that they want risks minimised
but recognise they are a necessary evil; and that the justification for taking these
risks is the degree of personal benefit that results.8
The benefits the public wish to see are not just personal. Many are both public
as well as personal, for example better public services or crime prevention; and
some primarily public, such as research.9 But personal benefit was what people
were most interested in. For example, one survey found ‘more tailored services’

6 Facebook, ‘Response to the Federal Trade Commission preliminary FTC staff report ‘protecting

consumer privacy in an era of rapid change: a proposed framework for Businesses and Policymakers’
(2011) available at: [Link]
ftc-staff-report-protecting-consumer-privacy-era-rapid-change-proposed-framework/00413-58069.
pdf [Accessed 2 Feb. 2017].
7 US Congress Subcommitee on Commerce, Trade and Consumer Protection of the Committee on

Energy and Commerce, ‘Opinion Surveys: What consumers have to say about information privacy’
(2001).
8 ScienceWise, ‘Big Data Public views on the collection, sharing and use of personal data by govern-

ment and companies’ (2014).

9 Wellcome Trust, ‘Summary Report of Qualitative Research into Public Attitudes to Personal Data

and Linking Personal Data’ (2013).

66 Roger Taylor

was the most popular justification for data sharing with ‘public benefit’ coming
second with half as many responses.10
The specific benefits identified in public opinion surveys include better and/
or cheaper services and products (both from government and companies),11
more tailored/personalised services and communications,12 preventing crime
and exposing dishonesty13 and transactional convenience.14 The dangers are loss
of control over data leading to either privacy invasion (people knowing things
you would wish them not to) or economic harms through identity theft, fraud
or other misuse of data;15 nuisance marketing and poorly targeted advertising;16
and discrimination whether by government or commercial organisations such as
insurers.17 Worries about these dangers were exacerbated by a sense that data con-
trollers were not to be trusted or were not being open about how data was being
used.18
This balancing of the benefits against the risks is often described in terms of a
rational ‘trade-off ’ that the public are willing to make.19 However, many surveys
and commentators have pointed out that public attitudes often appear to reflect
irrational and contradictory viewpoints rather than a rational trade-off between
competing priorities.
The ‘privacy paradox’20 refers to the fact that people in surveys express strong
levels of concern about lack of control over their private data while at the same
time showing a strong appetite for products such as social media or store cards
that depend, in most cases quite transparently, on the individuals sharing per-
sonal data.
Evidence of contradictory opinions can also be found within the survey data.
A UK survey found that receiving more personalised services and recommen-
dations was the most common reason for favouring company use of personal data

10 Deloitte, ‘Data Nation 2012: our lives in data’ (2012) available at: [Link]

content/dam/Deloitte/uk/Documents/deloitte-analytics/[Link].
11 Lee Rainie and M Duggan, ‘Privacy and Information Sharing’ (2015) Pew Research Center. Availa-

ble at: [Link] Deloitte 2012,

12 Wellcome Trust ‘Summary Report of Qualitative Research into Public Attitudes to Personal Data

and Linking Personal Data’; Deloitte 2012 (n 11), Lee Rainie (n 12).
13 Wellcome Trust; ‘Summary Report of Qualitative Research into Public Attitudes to Personal Data

and Linking Personal Data’; Deloitte ‘Data Nation 2012: our lives in data’; Daniel Cameron, Sarah Pope
and Michael Clemence ‘Dialogue on Data’ (2014) Ipsos MORI Social Research Institute.
14 Wellcome Trust, ‘Summary Report of Qualitative Research into Public Attitudes to Personal Data

and Linking Personal Data’; Rainie, ‘Privacy and Information Sharing’).

15 Wellcome Trust, ‘Summary Report of Qualitative Research into Public Attitudes to Personal Data

and Linking Personal Data’; Deloitte, ‘Data Nation 2012: our lives in data’;; Rainie, ‘Privacy and Infor-
mation Sharing’).
16 Wellcome Trust, ‘Summary Report of Qualitative Research into Public Attitudes to Personal Data

and Linking Personal Data’; Rainie, ‘Privacy and Information Sharing’).

17 Wellcome Trust, ‘Summary Report of Qualitative Research into Public Attitudes to Personal Data

and Linking Personal Data’; Rainie, ‘Privacy and Information Sharing’).

18 Deloitte, ‘Data Nation 2012: our lives in data’; Rainie, ‘Privacy and Information Sharing’).
19 Data & Marketing Association, ‘Data privacy: What the consumer really thinks’ (2015).
20 Susan Barnes, ‘A privacy paradox: Social networking in the United States’ (2006) First

Monday, 11(9).
 No Privacy without Transparency 67

even though people were more than twice as likely to be dissatisfied as satisfied
with the way that companies used browsing data to personalise communications.21
Another found that 41% of people agreed that: ‘Organisations I interact
with clearly explain why they collect and share data about me’. But, in the same
­survey, most people said that they would prefer not to share data because they
‘don’t know what happens with it’. The authors described these findings as a ‘clear
contradiction’.22 The same survey found that loss or theft of data was the number
one concern and yet the institutions the public most trusted to hold data (govern-
ment and public services) had the worst record for data breaches.
The contradiction is especially stark in research by the Annenberg School of
Communications,23 which found that 55% of US citizens disagreed (38% of them
strongly) that ‘It’s okay if a store where I shop uses information it has about me to
create a picture of me that improves the services they provide for me’. But when
asked if they would take discounts in exchange for allowing their supermarket to
collect information about their grocery purchases, 43% said yes. This included
many people who had disagreed with the first statement.
These apparent contradictions may reflect, as some have suggested, a lack of under-
standing. Surveys of the US public find low levels of appreciation of how privacy
policies work24 and, in particular, the way in which companies share anonymised
data to generate user profiles which predict behaviours or characteristics.25
An alternative explanation, supported by the Annenberg research, is that con-
sumers are resigned to the current way in which data sharing works but believe
they are being offered a poor deal. They are theoretically happy to engage in data
sharing and recognise it can be of benefit. But rather than engaging in a rational
weighing of risks and benefits, they are frustrated by the fact that they have insuf-
ficient information to make an informed judgement. They suspect they are being
offered a bad bargain—that there is a better deal that could be achieved but which
no-one is putting on the table. Surveys consistently find high levels of distrust:
public suspicion that their data is being used in ways that are not disclosed; aware-
ness that this may affect them adversely; and a sense that they do not have suffi-
cient control over what goes on.26
To an individual faced by a system which they believe is unfairly rigged against
them, but where they believe there is probably still a net benefit in participating,

21 Deloitte, ‘Data Nation 2012: our lives in data’;

22 Deloitte, ‘Data nation 2014: Putting customers first’ (2014) available at: [Link]
com/content/dam/Deloitte/uk/Documents/deloitte-analytics/[Link].
23 Joseph Turow, Michael Hennessy and Nora A. Draper, ‘The Tradeoff Fallacy: How ­ Marketers
are Misrepresenting American Consumers and Opening Them Up to Exploitation’ University of
­Pennsylvania (2015).
24 Pew Research Center, ‘What Internet Users Know About Technology and the Web’ (2014).
25 J Thurow, ‘The Tradeoff Fallacy: How Marketers are Misrepresenting American Consumers and

Opening Them Up to Exploitation’.

26 Commission (EC), ‘Special Eurobarometer 359: Attitudes on Data Protection and Electronic

Identity in the European Union’ (2011); Mary Madden and Lee Rainie, ‘Americans’ Attitudes About
Privacy, Security and Surveillance’ (2015) Pew Research Center.
68 Roger Taylor

the rational response is to insist that the terms of trade are unreasonable, but to
take part none-the-less. This is the behaviour we observe.
Such behaviour is not paradoxical or contradictory. It is rational and consist-
ent with a world in which promises not to share personal data still leave room
for companies to trade detailed anonymised records which are then used to infer
with varying degrees of accuracy highly personal things, such as whether or not
someone is pregnant.27 The observed behaviour is rational and consistent with a
situation in which the public are being offered a data ‘trade-off ’, but are denied
the means to assess whether or not it is beneficial.28 As one research participant
said about sharing data with companies: ‘none of them have ever told me how
I benefit’.29

B. Insecure Use and Imprecise Use of Data

There are two elements of the way the discourse is framed in surveys and policy
discussion which can exacerbate this sense of powerlessness. First, there is the
role of informed consent and the reliance on a mechanism in which individuals
exercise personal control over how their data is used. This approach is of limited
value if the individual is faced with a set of data-sharing options all of which are
sub-optimal.
Second, there is the focus on legal control over the purpose or uses to which
personal data is applied. Such control can be ineffective if the problem is not the
purpose to which the data is being put but the manner in which it is used for that
purpose. To explore this possibility, we can define two quite distinct problems that
users can encounter with the use of their personal data—the first we call insecure
use, the second imprecise use.
1. Insecure use of data. This causes harms through unauthorised or illegal use
whether that be through loss or theft of data or use by data controllers out-
side of areas for which they have legal authority. Harms here would include
identity theft and fraud or sharing with third parties without permission and
could result in financial loss, nuisance marketing or discrimination.
2. Imprecise use of data. This is use of data within legally authorised purposes,
but in a manner that none-the-less harms the data subject through the
poor quality of the application e.g. personalisation algorithms that produce
advertising of no interest to the data subject; medical algorithms that have a

27 Charles Duhigg, ‘How companies learn your secrets’ New York Times (Feb 16 2012) [Link]

[Link]/2012/02/19/magazine/[Link].
28 Dara Hallinan and Michael Friedewald, ‘Public Perception of the Data Environment and

Information Transactions: A Selected-Survey Analysis of the European Public’s Views on the Data
Environment and Data Transactions’ (2012) Communications & Strategies, No. 88, 4th Quarter 2012,
pp. 61–78.
29 Jamie Bartlett, The Data Dialogue (Demos 2012).
 No Privacy without Transparency 69

high error rate in diagnosis; financial algorithms that make inaccurate risk
assessments; or security algorithms that have low precision in identifying
threats. These problems can also result in financial loss, nuisance marketing
or discrimination.
There are examples in the public opinion surveys of harms that are as likely to
arise from imprecise use of data for a desired purpose as from unauthorised use
of data. For example, ‘more tailored and personalised services or recommenda-
tions’ is cited in one survey as one of the primary benefits from sharing data,30
while in another ‘nuisance marketing’ and the inappropriate ‘targeting’ of indi-
viduals by companies was seen as principle risk.31 While nuisance marketing may
be manageable to some degree through limiting the purposes for which data is
used, nuisance marketing may equally arise as the result of imprecise targeting of
communication and advertising to people who are actively seeking such targeting
as a benefit. If my only remedy is to define ever more precisely the information
I wish to receive, I may still fail and find I am pestered because I do not control the
way in which such options are framed. Even if I succeed in adequately defining the
content, frequency and style of communications I wish to receive, it will be a pyr-
rhic victory since I will have had to perform exactly the work that the personalisa-
tion algorithm claimed to be able to do for me—which was the original reason for
agreeing to share data. What I require is a reliable way to assess the precision of the
algorithm before consenting.
A similar tension can be found in other areas. The use of personal data to identify
fraud, to unearth dishonesty and to stop people cheating has received support in
surveys while at the same time, people expressed concern that use of data might lead
to ‘discrimination’.32 This does not just refer to discrimination against protected
characteristics, but refers to any unfair difference in treatment such as rejection of
credit or insurance, rejection of benefits claims, or differential pricing. The issue at
stake here is not whether in principle it is a good idea to use data for these purposes.
It is a question of whether data used in this way is done well or poorly. When using
data profiling to determine whether to accept or reject insurance risks or benefits
claims, the difference between discrimination and unearthing dishonesty is not a
difference in purpose, approach or generic consequence. The difference is the preci-
sion of the risk estimates and propensity scores generated by the algorithms.
The potential harm from insecure use features prominently in consumer sur-
veys. Harm from imprecise use of data is less often identified as a specific category
of risk. However, this may reflect the structure of the survey questions which typi-
cally present loss or theft of data as a separate category, rather than a clear public
view about the relative risks presented by these two issues.

30 Deloitte, ‘Data Nation 2012: our lives in data’.

31 Wellcome Trust, ‘Summary Report of Qualitative Research into Public Attitudes to Personal Data
and Linking Personal Data’.
32 Wellcome Trust, ‘Summary Report of Qualitative Research into Public Attitudes to Personal Data

and Linking Personal Data’.

70 Roger Taylor

There is substantial evidence of the potential harm that can arise from data-
driven systems which are designed to do something the public regard as beneficial,
but do so with insufficient precision. Health applications have, in particular, been
subjected to a degree of scrutiny and found wanting. Applications that aim to
treat psychological illnesses were highly variable in their effectiveness and were,
in some cases, based on weak scientific evidence with the risk that they might
be doing ‘more harm than good’.33 Three out of four apps designed to diagnose
melanoma were found to wrongly categorise 30% of melanomas or more as
‘unconcerning’.34 Diagnosis and triage apps have been found to perform poorly in
general.35 Wearable technology to support weight loss has been found to diminish
the impact of weight loss programmes.36
Data-driven applications designed to provide information may also be doing
their customers a disservice. If I use an online media platform that promises to
make me better informed, I risk, instead, being provided with a stream of infor-
mation that leaves me less well informed37 but more emotionally secure in the
correctness of my own beliefs.38 The harm here does not relate to unauthorised
use of data. I want my personal data to be used to identify relevant information.
However, the execution may fall short of what I hoped for in ways that are harmful
and which I have no way of discerning.
There is, additionally, evidence of websites using personal information to engage
in price discrimination against customers. This can be regarded as a form of lack
of precision, since the customer is sharing data online in the hope of accessing
keener pricing but is instead subjected to an algorithm which identifies them as an
appropriate target for higher prices. Although evidence of this is not widespread,
it does occur and there is potential for it to increase.39
In summary, there is substantial evidence that a significant risk of sharing data
with automated decision-making systems is lack of precision. It is not possible to
estimate whether the risks associated with imprecise use are greater or less than
the risks associated with insecure use. However, the relative risk of imprecision
increases to the extent that personal data is used more widely to drive automated
decisions by intelligent machines. And while it is true that with further data

33 S Leigh, S Flatt, ‘App-based psychological interventions: friend or foe?’ (2015) Evidence-Based

Mental Health 18:97–99.

34 JA Wolf, JF Moreau et al. ‘Diagnostic Inaccuracy of Smartphone Applications for Melanoma

Detection’ (2013) JAMA Dermatol. 149(4):422–426. doi:10.1001/jamadermatol.2013.2382.

35 Hannah L Semigran, Jeffrey A Linder, Courtney Gidengil and Ateev Mehrotra, ‘Evaluation of

symptom checkers for self diagnosis and triage: audit study’ (2015) BMJ 351:h34800.
36 JM Jakicic JM, KK Davis et al ‘ Effect of Wearable Technology Combined With a Lifestyle

Intervention on Long-term Weight Loss The IDEA Randomized Clinical Trial’, (2016) JAMA (11):
1161–1171. doi:10.1001/jama.2016.12858.
37 David Lazer, ‘The rise of the social algorithm’ (2015) Science Vol. 348, Issue 6239, pp. 1090–1091

DOI: 10.1126/science.aab1422.
38 Eli Pariser, The Filter Bubble: What the Internet Is Hiding From You (Viking 2012).
39 The White House (Executive Office of the President of the United States), Big data and differential

pricing (2015).
 No Privacy without Transparency 71

gathering and testing we would expect the precision of data-driven algorithms to

increase, it is also true that there are strong incentives within markets and society
that will encourage increasingly imprecise and harmful algorithms. Transparency
is important because such algorithms can operate harmfully across populations
without the harm being evident. These issues, explored in detail in section 4 below,
make it plausible that in the long run, we might expect the dangers of ‘rogue
algorithms’ behaving in legal ways that result in widespread, unintended harm to
be as great a threat as insecure processing of data.

III. How Does Data Protection Protect against

Insecure and Imprecise Use of Data?

The need to control how data is used has been central to data protection from the
start. The US HEW Fair Information Practices40 established the principle that data
subjects should know what data was collected and how it was used; they should
be able to correct data; and they should be assured that it would not be used for
any other purpose without consent. The OECD41 built on this, emphasising that
data collection and processing must be limited and lawful; that data processing
should for a specific limited purpose; that data subjects are entitled to know what
data is collected, how it is used and to review and correct information; and that
data should not be used for any other purpose except by consent or legal authority.
These same principles inform EU data protection regulations including the GDPR
under which data processing is illegal unless it falls under one of the specified cat-
egories of use; that it should be proportional to such use; that data subjects have
rights to be informed, to correct data and, where appropriate, limit use through
withholding of consent.42
This framework was developed prior to the widespread use of automated
decision-making systems and is designed to ensure secure use of data, as defined
above. It is not designed to protect against the imprecise use of data in automated
decision-making.
Where the logic of any such decision system is static and sufficiently simple to
be disclosed and understood, a description of the use of the data might be suf-
ficient to enable data subjects, citizens and regulators to assess the likely precision

40 Department of Health, Education and Welfare (US), Report of the Secretary’s Advisory Committee

on Automated Personal Data Systems, Records, Computer, and the Rights of Citizens (1973).
41 OECD Recommendation of the council concerning guidelines governing the protection of privacy and

transborder flows of personal data (1980).

42 REGULATION (EU) 2016/679 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL

of 27 April 2016 on the protection of natural persons with regard to the processing of personal data
and on the free movement of such data, and repealing Directive 95/46/EC [2016] OL J 119/12 (GDPR).
72 Roger Taylor

of the approach and any risks that might result. This may be helpful in informing
consent decisions or political opinions. However, with more sophisticated deci-
sion algorithms this is not possible.

A. The GDPR

New rights and protections afforded by the GDPR do not remedy this deficit. The
regulations are designed to protect the Fundamental Rights and Freedoms defined
in the EU charter. The rights specifically referenced (in recital 4) include
‘In particular the respect for private and family life, home and communications, the pro-
tection of personal data, freedom of thought, conscience and religion, freedom of expres-
sion and information, freedom to conduct a business, the right to an effective remedy
and to a fair trial, and cultural, religious and linguistic diversity.’

The two fundamental rights most frequently referenced in GDPR are Article 8
rights to data protection (e.g. recitals 39, 65, 71) and Article 25 rights to non-
discrimination (e.g. recital 71). Processing that does not have legal authority and
processing with legal authority that results in discrimination against protected
characteristics are clearly identified as breaching the regulations.
Some of the language used suggests there may be broader protections against
the adverse consequences of data processing. In particular, recitals 75 and 85 pro-
vide a list of risks including the following:
where the processing may give rise to discrimination, identity theft or fraud, financial loss,
damage to the reputation, loss of confidentiality of personal data protected by professional
secrecy, unauthorised reversal of pseudonymisation, or any other significant economic or
social disadvantage;

The reference to data processing that gives rise to ‘any other significant economic
or social disadvantage’ might suggest an intention to provide wide scope for pro-
tection against legal processing that performs poorly with negative results for the
data subject. This is listed as an additional issue over and above discrimination or
unauthorised use.
Recital 71 may also appear to address the question of precision in algorithmic
decision-making:
In order to ensure fair and transparent processing in respect of the data subject, taking into
account the specific circumstances and context in which the personal data are processed, the
controller should use appropriate mathematical or statistical procedures for the profiling,
implement technical and organisational measures appropriate to ensure, in particular, that
factors which result in inaccuracies in personal data are corrected and the risk of errors is
minimised …

However, it is far from clear that imprecise propensity scores could be regarded
as ‘inaccuracies’ in personal data any more than a record of a diagnosis given by
 No Privacy without Transparency 73

a doctor would be regarded as incorrect personal data on the grounds that the
doctor had a poor record of accurate diagnosis. The reference to ‘risk of errors’
would seem to apply to this same sense of ‘inaccuracies’ in data. An organisation
that was assiduous in ensuring the correct recording of the output of a relatively
imprecise algorithm would appear to be justified in claiming it was minimising
the risk of error under this definition. Any such claim would fall short of what the
public would expect ‘minimising the risk of error’ to mean.
The supposed new right to an ‘explanation’ with regard to automated decision-
making (Art. 13-15 and 22) does not resolve the problem. It is true that data sub-
jects must be informed of any ‘consequences’ of data processing. However close
analysis43 finds that this requirement does not go further than the requirements
of some existing data protection regimes and implies nothing more than a generic
explanation of processing: for example, that the consequence of a credit check is
that you may or may not get a loan. It does not protect against the risk that such
an algorithm is imprecise with the result that it produces credit scores that unfairly
penalise data subjects.
The right not to be subjected to automated decision-making (Art. 22) is also
of no help if I want to benefit from automated decision-making but only to do so
secure in the knowledge that the algorithms used are sufficiently precise and not
harmful.
Finally, there are some welcome clarifications to your rights of data access
(Art.15). But, as described in more detail in section 3 below, data about yourself
can rarely, if ever, provide a basis for querying the precision and accuracy of a
complex decision-making algorithm since such an assessment requires knowl-
edge of how the algorithm operates at a population level, not at an individual
level.
The lack of clear steps to address imprecision means that the GDPR falls short of
the ambition of recital 4 that ‘The processing of personal data should be designed
to serve mankind’. It leaves ample room for poor quality processing that complies
with the law and yet results in nuisance marketing, poor medical advice, unde-
served credit ratings, rejected insurance applications or information flows that
distort perceptions and mislead.
In passing, it is worth noting that the illegality of discrimination against
protected characteristics but the lack of protection against the broader impact of
imprecise algorithms has the potential to produce peculiar results. For example, if
an algorithm designed to identify low priced insurance systematically performed

43 Sandra Wachter, Brent Mittelstadt and Luciano Floridi, ‘Why a Right to Explanation of Automated

Decision-Making Does Not Exist in the General Data Protection Regulation’ (2016). International
Data Privacy Law, Forthcoming. Available at SSRN: [Link] Against:
Dimitra Kamarinou, Christopher Millard and Jatinder Singh, Machine Learning with Personal Data,
this volume.
74 Roger Taylor

worse for people of a particular ethnicity it might be in breach of the regulations.

However, if it performed poorly for all customers it would not. If a travel
recommendation service produced targeted communications that are deemed an
invasive nuisance for women but did not do this for men, it might be in breach.
But if men and women were equally annoyed by its communications it would
likely not be.

B. Transparency, Consent and Fair Processing

Transparency and informed consent are central features of data protection regimes
around the world, providing the basis for fair processing of data in the absence of
any more specific legal permission.44 Discussions of the value of consent often
assume that it allows a user to assess the risks and benefits of a particular agree-
ment to share data. In the words of the Canadian regulator: ‘being informed about
and understanding an organization’s policies and practices allow individuals to
provide meaningful consent. Individuals should be able to understand the risks
and benefits of sharing their personal information with the organization and be in
a position to freely decide whether to do so’.45
The gap between this intent and the reality has been widely noted. Criticism of
consent has ‘reached a crescendo on both sides of the Atlantic’ and ‘perhaps more
surprisingly the critique of notice and consent has more recently been echoed by
regulators, industry and privacy advocates’.46
Much of the attention has focussed on the complexity of the information users
are expected to understand; the imbalance in power between organisations seek-
ing consent and individuals; the broad nature of the consents sought and the non-
obvious nature of what these consents might enable.47 It has also been observed
that the reliance on privacy notices as a form of consumer protection risks giving
false assurance and undermining effective consumer protection.48 These problems
are further exacerbated by the increasing number of objects that automatically
and continuously collect data making the point at which consent should be sought
less clear.49

44 Eg US Privacy Act, 1974; EU Data Protection Directive art 7; GDPR art 6; Asia Pacific Economic

Cooperation Privacy Framework.

45 [Link]

gl_oc_201405/.
46 Fred H Cate, ‘Big data consent and the future of data protection’ in Cassidy R. Sugimoto, Hamid

R. Ekbia, Michael Mattioli (eds), Big Data Is Not a Monolith (MIT press 2016).
47 Ibid.
48 Omri Ben-Shahar and Carl Schneider, More Than You Wanted to Know: The Failure of Mandated

Disclosure (Princeton University Press 2014).

49 S Dritsas et al. ‘Protecting privacy and anonymity in pervasive computing: trends and perspec-

tives’ (2006) Telematics and Informatics 23 196–210; E Luger and T Rodden, ‘Terms of Agreement:
 No Privacy without Transparency 75

This has prompted calls to rely less on consent in which the individual is
expected to assess the acceptability of the risk/benefit trade off, and to instead put
more weight on regulation and accountability regimes in which service providers
take on the responsibility for ensuring such trade-offs fall within parameters set
by law and regulation.50
The GDPR has responded to that need by placing greater emphasis on the
duties of the data controller to demonstrate compliance and giving greater powers
on regulators to intervene. The requirement to keep audit trails of data processing
could, perhaps, provide a mechanism whereby regulators could examine the
question of the precision of algorithmic decision-making. However, in the
broader context of the regulations, the purpose of such powers would seem to
be to ensure that data is processed securely and is done so in a way that does not
infringe fundamental rights. It falls short of securing rights to information about
the precision of decision-making algorithms.
To illustrate the regulatory gap this creates, we can compare the use of
consent for medical intervention with the use of consent under data protection
regulations. With the former, there is typically an explicit requirement that the
patient be informed not only about the nature of the procedure and the rationale
but also about the risks that it presents. This does not refer simply to the risks of
the procedure going wrong or the doctor doing something that the patient had
not wanted. It refers also to the risks that arise if the procedure goes entirely as
intended.
It is also of note that in the literature on medical ethics, there is strong
recognition that consent and oversight are not alternatives but complementary
activities. There is a clear understanding that consent only operates effectively
within a context of trust established by effective regulation of those same risks that
patients are expected to accept as part of informed consent. Consent to treatment
is to a large degree based on trust in the individuals, professions and institutions
of medicine.51 In this context, trust has been defined as ‘a willing dependency on
another’s actions’ which ‘is limited to the area of need and is subject to overt and
covert testing. The outcome of trust is an evaluation of the congruence between
the expectations of the trusted person and actions.’52
The accountability mechanisms of medical regulation by professions and
governments, along with a medical culture that recognises the importance

Rethinking Consent for Pervasive Computing’ (2013) Interacting with Computers, 25(3); Richard
Gomer, MC Schraefel and Enrico Gerding, ‘Consenting Agents: Semi-Autonomous Interactions for
Ubquitous Consent’ (2014) UbiComp [Link]
50 Cate (n 47).
51 Kennet Calman, ‘Communication of risk: choice, consent, and trust’ (2002) The Lancet,
­Volume 360, Issue 9327, 166–168.
52 JE Hupcey, J Penrod, JM Morse and C Mitcham, ‘An exploration and advancement of the concept

of trust’ (2001) Journal of Advanced Nursing, 36: 282–293. doi:10.1046/j.1365-2648.2001.01970.x.

76 Roger Taylor

of scientific inquiry, ethics and care, provide the ‘overt and covert testing’ that
support the development of trust. An analogous accountability regime in privacy
regulation would aim to make consent a meaningful assessment of the congruence
between our expectations of what users of personal data are doing and what is
in fact occurring. Data protection regulation will not be able to achieve this if it
does not address risks of imprecise use of data—risks that the public regard as
significant issues for data protection.

C. Privacy vs Consumer Protection

One possible explanation for the focus on use based consent, rather than risk
based consent, in data protection regulations would be a view that risks of unau-
thorised use are matters relevant to privacy and risks relating to authorised use
should be viewed as consumer protection issues. In this view, privacy regulation
should concern itself primarily with preventing information being used illegally,
beyond consent or without due care to security. The question of whether use of
personal data within legal consented services is beneficial or harmful is matter for
consumer protection organisations.
This same view might take comfort from the view that market competition
might be expected to drive imprecise decision systems out of the market in favour
of more precise mechanisms. We will outline in the next section why market forces
are likely in many cases to favour less precision rather than more.
The arguments against separating consumer protection issues from data pro-
tection issues are practical. First, there is the consideration that this distinction
does not map well to the way in which the public think about the risks of data use
as described in Section 1 above.
Second, the practical mechanisms to address imprecision are the same as those
used to address insecure use of data. Consent and transparency around use of
data are unlikely to cease being important parts of any regulatory regime. In that
context, separating the risks of imprecise use from insecure use is confusing and
cumbersome.
Thirdly, the regulatory mechanism to ensure transparency about the precision
of decision-making systems will need to address questions of ownership and
control of the underlying data sets on which those systems operate. The skills
and expertise to police the various ways in which automated decision-making
can harm individuals do not divide neatly into those relevant to ‘consumer’
issues as opposed to those relevant to a more restricted definition of ‘privacy’
issues.
It is true that consumer protection research mechanisms can be of value. This
includes conducting research among the users of an application or putting an
application through a range of scenarios. This last approach was used by the EU
 No Privacy without Transparency 77

to investigate whether Google was distorting search results in favour of its own
shopping service.53
However, these approaches have limitations when applied to sophisticated AI
driven surveillance systems which continuously generate information about the
quality and nature of their decisions. While it is technically possible to gather
information about the quality of these systems without access to the data on which
they run, this approach has the disadvantage of being economically inefficient and
inherently less reliable.
It therefore makes sense to explore how privacy regulation can address the ques-
tion of risks and benefits as a totality considering both risks to security and risks of
imprecision within the same framework of regulations. The next section sets out
in more detail the challenges this creates.

IV. Measuring the Benefits and Risks of Data-driven

Automated Decision-making (Surveillance)

A simplified model of a dynamic surveillance system—an automated intelligent

data-driven decision-making system—is used to present a view of how informa-
tion about the risks and benefits of such systems can be generate from user data
and to highlight how individual and corporate rights over information interact
with the potential to generate this information.
Automated decision systems can operate according to fixed algorithms, but
much of the power of digital technology comes from the ability of surveillance
systems to operate dynamically, continually improving and optimising their
algorithms. Surveillance capitalism—the ability to collect data about customers,
segment audiences, predict behaviour and tailor products or offers to different
segments—has become a primary driver of value creation in industries such as
retailing, finance and the media.54 Governments use similar techniques of digital
surveillance for policing and national security and there are ambitions to greatly
expand the use of digital surveillance techniques to improve public services such
as health and education. This model can be applied equally to search engines such
as Google, advertising driven media such as Facebook, and web services such as
Amazon or Spotify as it can to potential future applications of data-driven
­decision-making such as AI-driven diagnostics or HR systems.

53 European Commission press release Antitrust: Commission fines Google €2.42 billion for abusing

dominance as search engine by giving illegal advantage to own comparison shopping service 27 June 2017.
54 Shoshana Zuboff, ‘Big other: surveillance capitalism and the prospects of an information civiliza-

tion’, (2015) Journal of Information Technology Vol 30, 75–89.

78 Roger Taylor

A. Model Surveillance System

We characterise digital surveillance systems with a five step model as follows.55

Step 1: Define A surveillance system must first define a propensity of interest
(i.e. tendency to purchase a particular product, respond to a drug, or commit a
crime) and find an association with attributes (data) known about the target pop-
ulation. Examples include estimates of the likelihood of involvement in criminal
activity based on communications metadata; estimates of the likelihood of buying
a product based on browsing history; or estimates of the likelihood of responding
to medication based on blood pressure readings and family history. The associa-
tions are normally based on historic data or research studies, but can be based on
any a priori beliefs.
Step 2: Identify A surveillances system must be able to collect and process
attribute data across a population and categorise individuals according to propen-
sities, typically defining a series of segments by clustering on the basis of similar
attributes and propensities. Examples might include ‘terrorist threat’, ‘white van
man’ or ‘hypertensive’.
Step 3: Intervene The surveillance organisation must then have the ability to
systematically intervene across a population according to the segment an indi-
vidual is in: wire-tapping some and not others; advertising to some and not others;
recommending medical treatments to one group and not to the other.
Many non-digital surveillance systems stop at this point and operate using static
segments and static rules defining the interventions for each. There are two fur-
ther steps that characterise dynamic surveillance system which data-driven digital
systems are particularly well suited to. The two steps are:
Step 4: Observe outcomes Surveillance systems can collect information about
the outcomes across the surveilled population. The outcome should relate to
the propensity of interest—e.g. was the individual identified as a threat pros-
ecuted for criminal activity; did the individual identified as a prospect buy the
product; did the patient given a diagnosis respond positively to treatment. Ide-
ally outcomes are collected equally for the whole population regardless of the
intervention but this is often not possible. For example, ideally, the system will
monitor the future blood pressure and stroke rate among treated and untreated
people whether defined as hypertensive or not; it will measure the purchasing
behaviour of those to whom a promotional message was sent and those to whom
it was not.
Step 5: Test error rate Outcome data can then be used to test the sensitivity and
specificity of categories/segments by identifying how well propensity estimates
forecast real behaviour and looking for attributes that correlate with unexpected
behaviour. If a cost is assigned to inaccurate intervention (where the outcome is

55 This model is based on a model presented in Roger Taylor and Tim Kelsey Transparency and the

Open Society (Policy Press, 2016).

 No Privacy without Transparency 79

not as predicted or accuracy is no higher than random) and a benefit to accurate

intervention, a net benefit of the surveillance can be calculated.

Model
surveillance
system Define/re-define
signature (i.e.
attributes of
target category)

Identify members
Test error rate of
with relevant
signature/target
attributes

Monitoring
Intervention

Observe outcomes
for category
members compared Observe/intervene
to non- with target group
members/other
categories

B. Estimating the Net Benefit of a Surveillance System

To estimate the net benefit of a surveillance system we need to know how often it
incorrectly estimates a propensity and intervenes in a way that is non-beneficial or
harmful or fails to intervene when it would be beneficial. We need to know both
its false positive rate and its false negative rate along with the costs associated with
each type of error.
Such estimates do not exist in the public domain for most surveillance systems,
but healthcare is one area where they do exist. The results show that information
about the purpose of surveillance does not provide reliable information about
the benefit of such a system. Breast screening programmes have been assumed
to be beneficial based on estimates from past studies. Meta-analysis of the out-
comes from breast screening suggest that it may be causing more harm than good
80 Roger Taylor

because the likelihood that it will recommend unnecessary tests is more costly to
people than the likelihood it will detect cancer earlier than would have occurred
without screening.56 A description of the purposes of breast screening or the way
the data was used could never reveal this.
Information about false positives and negatives is equally useful in assessing the value
of a surveillance system that makes recommendations regarding news, diet, investment,
or exercise regimes. Before consenting to an application that segments the population
on the basis of their exercise regime and heart rate to make exercise recommendations,
I would be wise to ask the extent to which people who follow its advice see improved
heart health as opposed to suffering heart attacks as compared to those who do not.

C. Risks of Surveillance Systems Resulting in Net Harm

There are reasons to believe that, even with the best intentions, surveillance
systems have the potential for significant harm. The example of breast cancer
screening shows how even in a relatively transparent and closely regulated area
of activity, it is possible that surveillance systems intended to protect people may
be harmful. Judging whether the harm that results from false negatives and false
positives outweighs the benefit of correct categorisation is not something that can
be done reliably from cursory examination. It relies on repeated interrogation of
the impact across populations.
There is an additional problem in market driven situations. Market competi-
tion may incentivise algorithms that make users happy but this can be wholly
consistent with harming the individuals concerned. Algorithms will typically be
optimised against a measure that is at best a proxy for the benefit that the data
subject wishes to receive. For example, an application making recommendations
about my exercise regime based on information about my heart rate and my
exercise may be optimised to produce the greatest improvement in heart health
or it may be optimised to produce the highest resubscription rate by users. It
might be assumed that if users like it, it is doing them good. However, it is equally
possible that users are delighted by recommendations that are damaging to their
health.
In a similar way, concerns about filter bubbles can be characterised as a mis-
match between a customer desire to be kept informed and the aim of the algo-
rithm to keep the customer happy as measured by their tendency to click on links.
The latter may mean hiding information from them that displeases them.
Finally, even if an algorithm is calibrated against exactly the outcome that the
data subject is interested in, the optimal level of false positives and false nega-
tives for the operator of a surveillance system is likely to differ from the socially
optimal level that the data subject would choose. Take for example, a commercial

56 PC Gotzsche and K Jorgensen, ‘Screening for breast cancer with mammography’, Cochrane

­Database of Systematic Reviews 2013, No 6. Art No: CD001877. DOI: 10.1002/14651858.CD001877.

 No Privacy without Transparency 81

surveillance system designed to help people identify the most suitable product at
the lowest price. The data subject’s interests are met by doing just that. The inter-
ests of the operator of the system would be met by identifying the combination
of product and price that yields the optimum combination of customer loyalty
and profit margin. The risks of misaligned incentives become troubling when
applied to the promotion of potentially addictive products such as gambling,
loans or alcohol.
As a result, it is unlikely that the GDPR will achieve its ambition of ensuring
that: ‘The processing of personal data should be designed to serve mankind.’
Indeed, given the likely spread of AI decision-making systems to a wide range
of mechanisms from self-driving cars and medical diagnostics to share trad-
ing and employment decisions, there is a risk that without stronger transpar-
ency the processing of personal data will be a significant cause of harm to
mankind.

V. How Might Regulators Ensure Reliable Information

about the Impact of Surveillance Systems be Generated?

We can identify three steps that could help in enabling accurate assessment of
the risks and benefits of data-driven surveillance systems. First, establishing
independent rights to access data for audit and assurance will be of great value.
This step has been recommended by a number of commentators including, for
example, Wachter57 who suggests that regulations should ‘allow for examination
of automated decision-making systems, including the rationale and circumstances of
specific decisions, by a trusted third party. … The powers of Supervisory Authorities
could be expanded in this regard.’
This might allow for a meaningful explanation of the consequences of data pro-
cessing from an unconflicted source. It is unclear the extent to which the authors
are recommending third parties be allowed access to raw data, but the implication
is that they would have such access since it is proposed as a mechanism to allow
scrutiny without compromising commercial confidentiality.
This approach is of value because the data held within a surveillances system
provides a unique insight into how the system is operating which it would not be
possible to replicate through external testing of a system. Requirements placed on
organisations to produce analyses of impact according to fixed regulatory formu-
lae run the risk of prompting gaming more than transparency.
However, the success of this approach would depend on the level of data access
and the analytical competence of the third party. There is a risk that if this right

57 Wachter, ‘Why a Right to Explanation of Automated Decision-Making Does Not Exist in the

General Data Protection Regulation’.