CS561T

Machine Learning Systems Design

ML and Data Systems Fundamentals

Course Slides have been adapted from

CS 329S (Chip Huyen, 2022)
[Link]
Agenda
1. ML systems fundamentals
2. Decoupling objectives
3. Data engineering 101

2
1. ML Systems Fundamentals

3
Business and ML Objectives
● Data scientists get excited about improving the model’s accuracy and
might spend a ton of resources—data, compute, and engineering
time—to achieve that.
● Most corporations don’t care about the fancy ML metrics, unless it
moves some business metrics.
● The sole purpose of businesses, according to the Nobel-winning
economist Milton Friedman, is to maximize profits for shareholders.
● The ultimate goal of any project within a business is,
○ To increase profits, either directly or indirectly:
○ directly such as increasing sales (conversion rates) and cutting costs;
○ indirectly such as higher customer satisfaction and increasing time spent on a
website.

4
Business and ML Objectives
● For an ML project to succeed within a business organization, it’s crucial to tie
the performance of an ML system to the overall business performance.
● What business performance metrics is the new ML system supposed to
influence,
● Example:
○ the amount of ads revenue,
○ the number of monthly active users?
● The effect of an ML project on business objectives can be hard to reason
about.
● For example,
○ An ML model that gives customers more personalized solutions can make them
happier, which makes them spend more money on your services.
○ The same ML model can also solve their problems faster, which makes them
spend less money on your services.

5
Business and ML Objectives
● To gain a definite answer on the question of how ML metrics influence
business metrics, experiments are often needed.
● Many companies do that with experiments like A/B testing and choose the
model that leads to better business metrics, regardless of whether this
model has better ML metrics.
● Sometimes even rigorous experiments might not be sufficient to understand
the relationship between an ML model’s outputs and business metrics.

6
Business and ML Objectives
● Example

7
Business and ML Objectives
● Many companies like to say that they use ML in their systems because
“being AI-powered” alone already helps them attract customers, regardless
of whether the AI part does anything useful.
● When evaluating ML solutions through the business lens, it’s important to
be realistic about the expected returns.
● There are many companies that have seen payoffs from ML.
● For example,
○ ML has helped Google search better, sell more ads at higher prices, improve
translation quality, and build better Android applications.
○ But this gain hardly happened overnight. Google has been investing in ML for
decades.

8
Returns on investment in ML
Returns on investment in ML depend a lot on the maturity stage of adoption.

9
Requirements for ML Systems
• The system should continue to perform the correct function at
Reliability the desired level of performance even in the face of adversity
• ML systems might grow in
• complexity,
Scalability
• traffic volume,
• model count, etc.
Code should be documented.
Code, data, and artifacts should be versioned.
Models should be sufficiently reproducible so that even when the original
authors are not around, other contributors can have sufficient contexts to
Maintainability build.
When a problem occurs, different contributors should be able to work
together to identify the problem and implement a solution without finger-
pointing.

The system should have some capacity for both discovering

Adaptability aspects for performance improvement and allowing updates
without service interruption.

10
ML in production: expectation
1. Collect data
2. Train model
3. Deploy model
4. .

11
ML in production: reality
1. Choose a metric to optimize model biases against one group ->
2. Collect data revert to older version
3. Train model 13. Get more data, train more, do more
4. Realize many labels are wrong -> testing
relabel data 14. Deploy model
5. Train model 15. Pray
6. Model performs poorly on one class -> 16. Model performs well but revenue
collect more data for that class decreasing
7. Train model 17. Cry
8. Model performs poorly on most recent 18. Choose a different metric
data -> collect more recent data 19. Start over
9. Train model
10. Deploy model
11. Dream about $$$
12. Wake up at 2am to complaints that

12
ML in production: reality Step 15 and 17 are essential

1. Choose a metric to optimize model biases against one group ->

2. Collect data revert to older version
3. Train model 13. Get more data, train more, do more
4. Realize many labels are wrong -> testing
relabel data 14. Deploy model
5. Train model [Link]
6. Model performs poorly on one class -> 16. Model performs well but revenue
collect more data for that class decreasing
7. Train model
[Link]
8. Model performs poorly on most recent
18. Choose a different metric
data -> collect more recent data
19. Start over
9. Train model
10. Deploy model
11. Dream about $$$
12. Wake up at 2am to complaints that

13
Process of developing an ML system

14
Project considerations
1. Framing
2. Objectives
3. Constraints
4. Phases

15
Framing the problem

Task type

Regression Classification

Binary Multiclass Multilabel

Low cardinality High cardinality

16
Multiclass vs. multilabel

Task type

Regression Classification

Binary Multiclass Multilabel

0 0 0 1 0 1 0 1

A label can belong to A label can belong to

only one class multiple classes
17
How to handle multilabel tasks

Multilabel problem solution

A multiclass problem A set of multiple binary

problems
0 1 0 1

Model 1: Model 2:
…
Does this Does this
belong to belong to
class 1? class 2?

18
Multilabel is harder than multiclass

Multilabel problem solution

1. How to create ground truth

labels?
2. How to decide decision
A multiclass problem A set of multiple binary boundaries

problems
0 1 0 1

Model 1: Model 2: …
Does this Does this
belong to belong to
class 1? class 2?
19
Multilabel: decision boundaries

Multilabel problem solution

A multiclass problem A set of multiple binary

0 1 2 3
problems
0.45 0.33 0.2 0.02

Poll:
Which classes should this
Model 1: Model 2: …
example belong to?
1. 0 Does this Does this
2. 0, 1 belong to belong to
3. 0, 1, 2 class 1? class 2?
20
A problem can be framed as different task types
Problem: predict the app users will most likely open next

Classification

0.072 0.15 … … 0.067 0.154

INPUT

User’s features Environment

time, location, etc.

0.2 App 0

OUTPUT 0.02 App 1

… …

0.04

21
A problem can be framed as different task types
Problem: predict the app users will most likely open next

Classification

INPUT 0.072 0.15 … … 0.067 0.154 Every time an app is

added/removed, you have to
User’s features Environment retrain your model
time, location, etc.

0.2 App 0

OUTPUT 0.02 App 1

… …

0.04

22
 Framing can make the problem easier/harder
Problem: predict the app users will most likely open next

Regression
OUTPUT

0.072 0.15 … … 0.067 0.154

INPUT 0 0.03 App 0

User’s features Environment App’s features

time, location, etc.

INPUT 1 0.072 0.15 … … 0.067 0.154 0.06 App 1

INPUT … 0.072 0.15 … … 0.067 0.154 0.25 App …

23
 Framing can make the problem easier/harder
Very common framing for
Problem: predict the app users will most likely open next recommendations / ads CTR

Regression
OUTPUT

0.072 0.15 … … 0.067 0.154

INPUT 0 0.03 App 0

User’s features Environment App’s features

time, location, etc.

INPUT 1 0.072 0.15 … … 0.067 0.154 0.06 App 1

INPUT … 0.072 0.15 … … 0.067 0.154 0.25 App …

24
Project objectives
● ML objectives
● Business objectives

25
Project objectives
● ML objectives
○ Performance
How to evaluate
○ Latency accuracy/F1/etc. without ground
○ etc. truth labels?

26
Project objectives
● ML objectives
○ Performance
○ Latency
○ etc.
● Business objectives
○ Cost
○ ROI
○ Regulation & compliance

27
Project objectives
● ML objectives
○ Performance
○ Latency
○ etc.
● Business objectives
○ Cost
○ ROI
○ Regulation & compliance

28
Business objectives
How can this ML project increase profits directly or indirectly?

● Directly: increasing sales (ads, conversion rates), cutting costs

● Indirectly: increasing customer satisfaction, increasing time spent on a
website

29
ML <-> business: can be tricky

ML model gives customers more personalized solutions

customers’ problems
customers happier
solved faster

customers spending customers spending

more money less money
30
ML <-> business: mapping
● Baselines
○ Existing solutions, simple solutions, human experts, competitors solutions, etc.

31
ML <-> business: mapping
● Baselines
● Usefulness threshold
○ Self-driving needs human-level performance. Predictive texting doesn’t.

32
ML <-> business: mapping
● Baselines
● Usefulness threshold
● False negatives vs. false positives
○ Covid screening: no false negative (patients with covid shouldn’t be classified as no covid)
○ Fingerprint unlocking: no false positive (unauthorized people shouldn’t be given access)

33
ML <-> business: mapping
● Baselines
● Usefulness threshold
● False negatives vs. false positives
● Interpretability
○ Does the ML system need to be interpretable? If yes, to whom?

34
ML <-> business: mapping
● Baselines
● Usefulness threshold
● False negatives vs. false positives
● Interpretability
● Confidence measurement (how confident it is about a prediction)
○ Does it need confidence measurement?
○ Is there a confidence threshold? What to do with predictions below that threshold—discard
it, loop in humans, or ask for more information from users?

35
ML <-> business: mapping
● Baselines
○ Existing solutions, simple solutions, human experts, competitors solutions, etc.
● Usefulness threshold
○ Self-driving needs human-level performance. Predictive texting doesn’t.
● False negatives vs. false positives
○ Covid screening: no false negative (patients with covid shouldn’t be classified as no covid)
○ Fingerprint unlocking: no false positive (unauthorized people shouldn’t be given access)
● Interpretability
○ Does it need to be interpretable? If yes, to whom?
● Confidence measurement (how confident it is about a prediction)
○ Does it need confidence measurement?
○ Is there a confidence threshold? What to do with predictions below that threshold—discard
it, loop in humans, or ask for more information from users?

36
Constraints: time & budget
● Time
○ Rule of thumb: 20% time to get initial working system, 80% on iterative development
● Budget
○ Data, resources, talent

Time/budget tradeoffs

● Use more (powerful) machines

● Hire more people to label data faster
● Run more experiments in parallel
● Buy existing solutions
37
Constraints: privacy
● Annotation
○ Can data be shipped outside organizations for annotation?
● Storage
○ What kind of data are you allowed to store? How long can you store it?
● Third-party solutions
○ Can you share your data with a 3rd party (e.g. managed service)?
● Regulations
○ What regulations do you have to conform to?

38
Technical constraints
● Competitors
● Legacy systems

39
Four phases of ML adoption

40
 Phase 1: Before ML

“If you think that machine learning will give you a 100% boost, then a
heuristic will get you 50% of the way there.”

Martin Zinkevich, Google

41
 42
[Link]
 Phase 2: Simplest ML models
Start with a simple model that allows visibility into its working to:

● validate hypothesis
● validate pipeline

43
 Phase 3: Optimizing simple models
● Different objective functions
● Feature engineering
● More data
● Ensembling

44
Phase 4: Complex ML models

45
2. Decoupling objectives

46
Decoupling objectives
Possible high-level goals when building a ranking system for newsfeed?

1. minimize the spread of misinformation

2. maximize revenue from sponsored content
3. maximize engagement

Which goal would you choose?

47
 Side note: ethics of maximizing engagement

Facebook Employee Raises Powered by ‘Really Dangerous’ Algorithm That Favors Angry Posts (SFist, 2019) 48
The Making of a YouTube Radical (NYT, 2019)
Goal: maximize engagement
Step-by-step objectives:

1. Filter out spam

2. Filter out NSFW content
3. Rank posts by engagement: how likely users will click on them

49
Wholesome newsfeed
Goal: maximize users’ engagement while minimizing the spread of extreme views
and misinformation

Step-by-step objectives:

1. Filter out spam

2. Filter out NSFW content
3. Filter out misinformation
4. Rank posts by quality
5. Rank posts by engagement: how likely users will click on them

50
Decoupling objectives
Goal: maximize users’ engagement while minimizing the spread of extreme views
and misinformation

Step-by-step objectives:

1. Filter out spam

2. Filter out NSFW content
3. Filter out misinformation
4. Rank posts by quality
5. Rank posts by engagement: how likely users will click on it

How to rank posts by both

quality & engagement? 51
Multiple objective optimization (MOO)
● Rank posts by quality
○ Predict posts’ quality
○ Minimize quality_loss: difference between predicted quality and true quality

● Rank posts by how likely users will click on it

○ Predict posts’ engagement
○ Minimize engagement_loss: difference between predicted clicks and true clicks

52
One model optimizing combined loss
● Rank posts by quality
○ Predict posts’ quality
○ Minimize quality_loss: difference between predicted quality and true quality

● Rank posts by how likely users will click on it

○ Predict posts’ engagement
○ Minimize engagement_loss: difference between predicted clicks and true clicks

loss = 𝛼 quality_loss + 𝛽 engagement_loss

Train one model to minimize this combined loss

Tune 𝛼 and 𝛽 to meet your need

Side note 1: check out Pareto optimization if you

want to learn about how to choose 𝛼 and 𝛽
53
 One model optimizing combined loss
● Rank posts by quality
○ Predict posts’ quality
○ Minimize quality_loss: difference between predicted quality and true quality

● Rank posts by how likely users will click on it

○ Predict posts’ engagement
○ Minimize engagement_loss: difference between predicted clicks and true clicks

loss = 𝛼 quality_loss + 𝛽 engagement_loss

Train one model to minimize this combined loss

Side note 2: this is quite common, e.g. style transfer

54
A Neural Algorithm of Artistic Style (Gatys et al, 2017)
One model optimizing combined loss
● Rank posts by quality
○ Predict posts’ quality
○ Minimize quality_loss: difference between predicted quality and true quality

● Rank posts by how likely users will click on it

○ Predict posts’ engagement
○ Minimize engagement_loss: difference between predicted clicks and true clicks

loss = 𝛼 quality_loss + 𝛽 engagement_loss

Train one model to minimize this combined loss

Every time you want to tweak 𝛼 and

𝛽, you have to retrain your model!
55
Multiple models: each optimizing one objective
● Rank posts by quality
○ Predict posts’ quality
○ Minimize quality_loss: difference between predicted quality and true quality

● Rank posts by how likely users will click on it

○ Predict posts’ engagement
○ Minimize engagement_loss: difference between predicted clicks and true clicks

Mq: optimizes quality_loss

Me: optimizes engagement_loss

Rank posts by 𝛼 Mq(post) + 𝛽 Me(post)

Now you can tweak 𝛼 and 𝛽 without retraining models

56
Decouple different objectives
● Easier for training:
○ Optimizing for one objective is easier than optimizing for multiple objectives
● Easier to tweak your system:
○ E.g. 𝛼 % model optimized for quality + 𝛽 % model optimized for engagement
● Easier for maintenance:
○ Different objectives might need different maintenance schedules
■ Spamming techniques evolve much faster than the way post quality is perceived
■ Spam filtering systems need updates more frequently than quality ranking systems

57
3. Data Engineering 101

58
Data engineering 101
● Data sources
● Data formats
● Data models
● Data storage engines & processing

59
Data sources
● User generated
● Systems generated
● Internal databases: users, inventory, customer relationships
● Third-party data

60
Data sources

Users generated data Systems generated data

User inputs Logs, metadata, predictions

Easily mal-formatted Easier to standardize

Need to be processed ASAP OK to process periodically

(unless to detect problems ASAP)

Can grow very large very quickly

● Many tools to process & analyze logs:

Logstash, DataDog, Logz, etc.
● OK to delete when no longer useful

Users’ behavioral data (clicks, time spent, etc.) is often

system-generated but is considered user data
61
 Third-party data: creepy but fascinating
● Types of data
○ social media, income, job
● Demographic group
○ men, age 25-34, work in tech
● More available with Mobile Advertiser ID
● Useful for learning features
○ people who like A also like B

62
[Link]/audience-data
The end of tracking IDs …

63
 Or is this?

64
TikTok wants to keep tracking iPhone users with state-backed workaround | Ars Technica
How to store your data?
Storing your data is only interesting if you want to access it later

● Storing data: serialization

● Unloading data: deserialization

65
How to store your data?

Data formats are

agreed upon standards
to serialize your data so that
it can be transmitted & reconstructed later

66
Data formats: questions to consider
● How to store multimodal data?
○ {‘image’: [[200,155,0], [255,255,255], ...], ‘label’: ‘car’, ‘id’: 1}
● Access patterns
○ How frequently the data will be accessed?
● The hardware the data will be run on
○ Complex ML models on TPU/GPU/CPU

67
Data formats

Format Binary/Text Human-readable Example use cases

JSON Text Yes Everywhere

Row-major CSV Text Yes Everywhere

Column-major Parquet Binary No Hadoop, Amazon Redshift

Avro Binary primary No Hadoop

Protobuf Binary primary No Google, TensorFlow (TFRecord)

Pickle Binary No Python, PyTorch serialization

68
 Row-major vs. column-major

Column-major:
● stored and retrieved column-by-column
● good for accessing features

Column 1 Column 2 Column 3

Row-major: Sample 1 ... ... ...
● stored and retrieved row-
by-row Sample 2 ... ... ...
● good for accessing samples
Sample 3 ... ... ...

69
 Row-major vs. column-major: DataFrame vs. ndarray

Pandas DataFrame: column-major

● accessing a row much slower than
accessing a column and NumPy

NumPy ndarray: row-major by

default
● can specify to be column-based

70
[Link]
Text vs. binary formats

Text files Binary files

Examples CSV, JSON Parquet

Pros Human readable Compact

Store the number 1000000? 7 characters -> 7 bytes If stored as int32, only 4 bytes

71
Data models
● Describe how data is represented
● Two main paradigms:
○ Relational model
○ NoSQL

72
 Relational model (est. 1970)
● Similar to SQL model
● Formats: CSV, Parquet

Tuple (row):
unordered
Column 1 Column 2 Column 3 .... Heading

Column:
unordered 73
Relational model: normalization
What if we change “Banana Press” to “Pineapple Press”?

Title Author Format Publisher Country Price Original Book

Harry Potter J.K. Rowling Paperback Banana Press UK $20 Relation
Harry Potter J.K. Rowling E-book Banana Press UK $10

Sherlock Holmes Conan Doyle Paperback Guava Press US $30

The Hobbit J.R.R. Tolkien Paperback Banana Press US $30

Sherlock Holmes Conan Doyle Paperback Guava Press US $15

74
Relational model: normalization
Title Author Format Publisher ID Price Updated Book
Harry Potter J.K. Rowling Paperback 1 $20 Relation
Harry Potter J.K. Rowling E-book 1 $10

Sherlock Holmes Conan Doyle Paperback 2 $30

The Hobbit J.R.R. Tolkien Paperback 1 $30

Sherlock Holmes Conan Doyle Paperback 2 $15

Publisher ID Publisher Country Publisher

1 Banana Press UK Relation
2 Guava Press US

75
Relational model: normalization
Title Author Format Publisher ID Price Pros:
Harry Potter J.K. Rowling Paperback 1 $20 ● Less mistakes
(standardized spelling)
Harry Potter J.K. Rowling E-book 1 $10
● Easier to update
Sherlock Holmes Conan Doyle Paperback 2 $30 ● Easier localization
The Hobbit J.R.R. Tolkien Paperback 1 $30

Sherlock Holmes Conan Doyle Paperback 2 $15

Publisher ID Publisher Country Cons:

● Slow to join across
1 Banana Press UK
multiple large tables
2 Guava Press US

76
Relational Model & SQL Model
● SQL model slightly differs from relational model
○ e.g. SQL tables can contain row duplicates. True relations can’t.
● SQL is a query language
○ How to specify the data that you want from a database
● SQL is declarative
○ You tell the data system what you want
○ It’s up to the system to figure out how to execute
■ Query optimization

77
SQL
● SQL is an essential data scientists’ tool

78
Problems with SQL
● What if we add a new column?
● What if we change a column type?

79
 SQL to NoSQL

80
[Link]
NoSQL: No SQL -> Not Only SQL
● Document model
● Graph model

81
NoSQL
● Document model
○ Central concept: document
○ Relationships between documents are rare
● Graph model
○ Central concept: graph (nodes & edges)
○ Relationships are the priority

82
Document model: example
● Book data in the document model
● Each book is a document

83
Graph model
type: country
name: USA

within

type: state
type: country
name:
name: France
California
within
within within

type: city type: city type: city

name: Paris name: Palo Alto born_in name: Stanford

lives_in born_in lives_in

type: person coworker type: person friend type: person

name: Kinbert Chou name: Megan Leszczynski name: Chloe He

84
Graph model Query: show me everyone
who was born in the USA?
● Easy in graph
● Difficult in SQL
type: country
name: USA

within

type: state
type: country
name:
name: France
California
within
within within

type: city type: city type: city

name: Paris name: Palo Alto born_in name: Stanford

lives_in born_in lives_in

type: person coworker type: person friend type: person

name: Kinbert Chou name: Megan Leszczynski name: Chloe He

85
Structured vs. unstructured data

Structured Unstructured
Schema clearly defined Whatever

Easy to search and analyze Fast arrival (e.g. no need to clean up first)

Can only handle data with specific schema Can handle data from any source

Schema changes will cause a lot of trouble No need to worry about schema changes

Data warehouses Data lakes

86
Structured vs. unstructured data

Structured Unstructured
Structure is assumed at write Structure is assumed at read

87
Data Storage Engines & Processing

Databases optimized for

Transactional Analytical
processing processing

88
OnLine Transaction Processing (OLTP)
● Transactions: tweeting, ordering a Lyft, uploading a new model, etc.
● Operations:
○ Insert when generated
○ Occasional update/delete

89
OnLine Transaction Processing
● Transactions: tweeting, ordering a Lyft, uploading a new model, etc.
● Operations:
○ Inserted when generated
○ Occasional update/delete
● Requirements
○ Low latency
○ High availability

90
OnLine Transaction Processing
● Transactions: tweeting, ordering a Lyft, uploading a new model, etc.
● Operations:
○ Inserted when generated
○ Occasional update/delete
● Requirements
○ Low latency
○ High availability See ACID:
Atomicity,
○ ACID not necessary
Consistency,
■ Atomicity: all the steps in a transaction fail or succeed as a group
Isolation,
● If payment fails, don’t assign a driver
Durability
■ Isolation: concurrent transactions happen as if sequential
● Don’t assign the same driver to two different requests that happen at the same time

91
OnLine Transaction Processing
● Transactions: tweeting, ordering a Lyft, uploading a new model, etc.
● Operations:
○ Inserted when generated
○ Occasional update/delete
● Requirements
○ Low latency
○ High availability
● Typically row-major

INSERT INTO RideTable(RideID, Username, DriverID, City, Month, Price)

Row VALUES ('10', 'memelord', '3932839', 'Stanford', 'July', '20.4');

92
OnLine Analytical Processing (OLAP)
● How to get aggregated information from a large amount of data?
○ e.g. what’s the average ride price last month for riders at Stanford?
● Operations:
○ Mostly SELECT

93
OnLine Analytical Processing
● Analytical queries: aggregated information from a large amount of data?
○ e.g. what’s the average ride price last month for riders at Stanford?
● Operations:
○ Mostly SELECT
● Requirements:
○ Can handle complex queries on large volumes of data
○ Okay response time (seconds, minutes, even hours)

94
OnLine Analytical Processing
● Analytical queries: aggregated information from a large amount of data?
○ e.g. what’s the average ride price last month for riders at Stanford?
● Operations:
○ Mostly SELECT
● Requirements:
○ Can handle complex queries on large volumes of data
○ Okay response time (seconds, minutes, even hours)
● Typically column-major

SELECT AVG(Price)
Column FROM RideTable
WHERE City = 'Stanford' AND Month = 'July';

95
OLTP & OLAP are outdated terms

96
Decoupling storage & processing
● OLTP & OLAP: how data is stored is also how it’s processed
○ Same data being stored in multiple databases
○ Each uses a different processing engine for different query types
● New paradigm: storage is decoupled from processing
○ Data can be stored in the same place
○ A processing layer on top that can be optimized for different query types

97
 Decoupling storage & processing

98
[Link]
99
ETL (Extract, Transform, Load)

Extract,
Transform,
OLTP Load OLAP

Transform: the meaty part

● cleaning, validating, transposing, deriving values, joining from multiple
sources, deduplicating, splitting, aggregating, etc.

100
 Often done in batch

101
[Link]
ETL -> ELT

Structured -> unstructured -> structured

want more flexibility tools & infra
standardized

ETL -> ELT -> ETL

102
Machine Learning Systems Design
Next class: Training Data

[Link] | Chip Huyen