CS561T

Machine Learning Systems Design

Course Slides have been adapted from

CS 329S (Chip Huyen, 2022)
[Link]
 Lecture 1:
Understanding ML production

2
Agenda
1. Course overview
2. ML research vs. ML production
3. ML systems vs. traditional software
4. ML production myths

3
1. Course overview

4
2025: ML is in almost every aspect of our lives

Face Machine
Photo unlocking
Translation
editing
Fraud
Recommendation detection

ETA

Self driving
Search cars
Smart AI
compose Smart
assistant security
cameras

5
 Enterprise use cases

6
See course website: [Link]
 AI value creation by 2030

13 trillion USD
Most of it will be outside the
consumer internet industry

We need more people from

non-CS background in AI!

7
Why ML Systems Design?
● ML algorithms is the less problematic part.
● The hard part is to how to make algorithms work with other parts to solve
real-world problems.

8
Why ML Systems Design?
● ML algorithms is the less problematic part.
● The hard part is to how to make algorithms work with other parts to solve
real-world problems.
● 60/96 failures caused by non-ML components

9
ML Systems Design

System

Interface

Data ML algorithms

Infrastructure
Most ML
Hardware courses/books

10
What’s machine learning systems design?
The process of defining the interface, algorithms, data, infrastructure, and
hardware for a machine learning system to satisfy specified requirements.

11
What’s machine learning systems design?
The process of defining the interface, algorithms, data, infrastructure, and
hardware for a machine learning system to satisfy specified requirements.

reliable, scalable, maintainable, adaptable

12
The questions this class will help answer …
● You’ve trained a model, now what?
● What are different components of an ML system?
● How to do data engineering?
● How to engineer features?
● How to evaluate your models, both offline and online?
● What’s the difference between online prediction and batch prediction?
● How to serve a model on the cloud? On the edge?
● How to continually monitor and deploy changes to ML systems?
● …

13
This class will not teach ...
● Machine learning/deep learning algorithms
● Computer systems
● UX design

14
Machine learning: expectation
BLADE RUNNER 2049 - Official Trailer

There are still pages left in this story. Watch the NEW trailer for #BladeRunner2049, in theaters October 6.
--
Thirty years after the events of the first film, a new blade runner, LAPD Officer K (Ryan Gosling), unearths a long-buried secret that has the potential to plunge what’s left of society into chaos. K’s discovery leads him on a quest to find Rick Deckard (Harrison Ford), a former LAPD blade runner who has been missing for 30 years.

From executive producer Ridley Scott and director Denis Villeneuve, #BladeRunner2049 stars Ryan Gosling, Harrison Ford, Ana De Armas, MacKenzie Davis, Sylvia Hoeks, Lennie James, Carla Juri, Robin Wright, Dave Bautista and Jared Leto.
--
Follow #BladeRunner2049 on social media:
[Link]
[Link]
[Link]
[Link]

This class won’t teach

you how to do this

15
 Machine learning: reality
AI Camera Mistakes Soccer Ref’s Bald Head For Ball

AI Camera Mistakes Soccer Ref’s Bald Head For Ball

Hello World.
I'm Imagination.
In this video, I'm going to talk about how an AI Camera Mistakes Soccer Ref’s Bald Head For Ball.

Technology and sports have a fairly mixed relationship already. Log on to Twitter during a soccer match (or football as it's properly known*) and as well as people tweeting ambiguous statements like "YESSS" and "oh no mate" to about 20,000 inexplicable retweets, you'll likely see a lot of complaints about the video assistant referee (VAR) and occasionally goal-line technology not doing its job.

Fans of Scottish football team Inverness Caledonian Thistle FC experienced a new hilarious technological glitch during a match last weekend, but in all honesty, you'd be hard-pressed to say it didn't improve the viewing experience dramatically.
The club announced a few weeks ago it was moving from using human camera operators to cameras controlled by AI. The club proudly announced at the time the new "Pixellot system uses cameras with in-built, AI, ball-tracking technology" and would be used to capture HD footage of all home matches at Caledonian Stadium, which would be broadcast directly to season-ticket holders' homes.

The AI camera appeared to mistake the man's bald head for the ball for a lot of the match, repeatedly swinging back to follow the linesman instead of the actual game. Many viewers complained they missed their team scoring a goal because the camera "kept thinking the Lino bald head was the ball," and some even suggested the club would have to provide the linesman with a toupe or hat.
With no fans allowed in the stadium due to Covid-19 restrictions, the fans of Inverness Caledonian Thistle FC and their opponents Ayr United could only watch via the cameras, and so were treated to mostly a view of the linesman's head instead of any exciting moments of the match that were occurring off-camera, though some fans saw this as a bonus given the usual quality of performance.

The cognitive capabilities of current architectures are very limited, using only a simplified version of what intelligence is capable of. For instance, the human mind has come up with ways to reason beyond measure and logical explanations to different occurrences in life. What would have been otherwise straightforward, an equivalently difficult problem may be challenging to solve computationally as opposed to using the human mind. This gives rise to two classes of models: structuralist and functionalist. The structural models aim to loosely mimic the basic intelligence operations of the mind such as reasoning and logic. The functional model refers to the correlating data to its computed counterpart.

The overall research goal of artificial intelligence is to create technology that allows computers and machines to function intelligently. The general problem of simulating (or creating) intelligence has been broken down into sub-problems. These consist of particular traits or capabilities that researchers expect an intelligent system to display.

That's all for today.

Thanks for watching.

You’ll likely build something like

this (buggy but cool)

16
Prerequisites
● Knowledge of CS principles and skills
● Understanding of ML algorithms
● Familiar with at least one framework such as TensorFlow, PyTorch, JAX
● Familiarity with basic probability theory

17
Syllabus

18
Grading

19
Honor code: permissive but strict - don’t test us ;)
● OK to search, ask in public about the systems we’re studying. Cite all the
resources you reference.
○ E.g. if you read it in a paper, cite it. If you ask on Quora, include the link.
● NOT OK to ask someone to do assignments/projects for you.
● OK to discuss questions with classmates. Disclose your discussion partners.
● NOT OK to copy solutions from classmates.
● OK to use existing solutions as part of your projects/assignments. Clarify
your contributions.
● NOT OK to pretend that someone’s solution is yours.
● OK to publish your final project after the course is over (we encourage that!)
● NOT OK to post your assignment solutions online.
● ASK the course staff if unsure!

20
Course staff

21
2. ML research vs. ML production

22
What is Machine Learning?
● Machine learning is an approach to learn complex patterns from
existing data and use these patterns to make predictions on unseen
data.
● Learn: the system has the capacity to learn.
○ For an ML system to learn, there must be something for it to learn from.
○ For example, if you want to build an ML system to learn to predict the rental
price for Airbnb listings, you need to provide a dataset where each input is a
listing with relevant characteristics (square footage, number of rooms,
neighborhood, amenities, rating of that listing, etc.) and the associated output is
the rental price of that listing.
● Complex patterns: there are patterns to learn, and they are complex
○ ML has been very successful with tasks with complex patterns such as object
detection and speech recognition.
○ Whether a pattern exists might not be obvious, or if patterns exist, your dataset
or ML algorithms might not be sufficient to capture them.
23
What is Machine Learning?
● Machine learning is an approach to learn complex patterns from
existing data and use these patterns to make predictions on unseen
data.
● Learn: the system has the capacity to learn.
○ For an ML system to learn, there must be something for it to learn from.
○ For example, if you want to build an ML system to learn to predict the rental
price for Airbnb listings, you need to provide a dataset where each input is a
listing with relevant characteristics (square footage, number of rooms,
neighborhood, amenities, rating of that listing, etc.) and the associated output is
the rental price of that listing.
● Complex patterns: there are patterns to learn, and they are complex
○ ML has been very successful with tasks with complex patterns such as object
detection and speech recognition.
○ Whether a pattern exists might not be obvious, or if patterns exist, your dataset
or ML algorithms might not be sufficient to capture them.
24
What is Machine Learning?
● Existing data: data is available, or it’s possible to collect
data
○ Because ML learns from data, there must be data for it to learn from.
○ It’s also possible to launch an ML system without data, but they will
learn from incoming data in production.
○ Without data and without continual learning, many companies follow a
“fake-it-till-you make it” approach:
■ Launching a product that serves predictions made by humans,
instead of ML models, with the hope of using the generated data to
train ML models later.

25
What is Machine Learning?

26
What is Machine Learning?
● Predictions: ML models make predictions, so they can only solve problems that
require predictive answers.

1. Classification Problems (Predicting Categories)

○ Email Spam Detection – Predict whether an email is spam or not spam.
○ Credit Risk Prediction – Predict if a loan applicant will default or repay.
○ Medical Diagnosis – Predict whether a patient has disease A, disease B, or healthy.
○ Sentiment Analysis – Predict if a product review is positive, negative, or neutral.
○ Image Recognition – Predict what object is in an image (e.g., cat, dog, car).

2. Regression Problems (Predicting Numeric Values)

○ House Price Prediction – Predict the price of a house based on size, location, etc.
○ Stock Price Forecasting – Predict the next day’s stock closing price.
○ Sales Forecasting – Predict future product sales from past trends.
○ Weather Forecasting – Predict temperature or rainfall for the next week.
○ Energy Consumption Prediction – Predict electricity demand in smart grids.

27
What is Machine Learning?
3. Time Series Prediction (Predicting Sequential Data)
○ Traffic Flow Prediction – Predict the number of cars on a highway at a given time.
○ Demand Forecasting – Predict product demand (e.g., retail or food delivery).
○ Disease Outbreak Prediction – Predict the spread of infections over time.
○ Cryptocurrency Forecasting – Predict Bitcoin or Ethereum price trends

4. Ranking / Recommendation Problems

○ Movie Recommendation – Predict which movies a user will like.
○ Search Ranking – Predict the order of relevance of search results.
○ Ad Click Prediction – Predict the probability of a user clicking on an ad.
○ Job Matching – Predict best job postings for a candidate.

28
What is Machine Learning?
● As predictive machines (e.g., ML models) are becoming more effective, more
problems are being reframed as predictive problems.

● Problems Reframed as Predictive Problems

○ Healthcare → Early disease detection, hospital readmission prediction
○ Finance → Credit scoring, fraud detection
○ Retail & Marketing → Customer churn prediction, product recommendations, dynamic
pricing
○ Transportation → Ride demand forecasting, traffic prediction
○ Manufacturing → Predictive maintenance, defect detection
○ Education → Student performance and dropout prediction
○ Language Generation → LLMs predict the next word (token) in a sequence given previous
context.
○ Question Answering → Predict the most probable answer to a user query.
○ Code Completion → Predict the next line or function in a code snippet.
○ Dialogue Systems → Predict the next appropriate conversational turn.

29
What is Machine Learning?
● Unseen data: The patterns your model learns from existing
data are only useful if unseen data also share these
patterns.
○ A model to predict whether an app will get downloaded on Christmas
2020 won't perform very well if it's trained on data from 2008, when
the most popular app on the App Store was Koi Pond.
● In technical terms, it means your unseen data and training
data should come from similar distributions.

30
Traditional Programming Vs ML

31
Different Machine Learning Models

32
Machine Learning: Some Recent Applications
Application /
Year What it does Impact
Product
Deep learning–based face Pioneered large-scale face
DeepFace recognition system with recognition, enabled photo
2014
(Meta/Facebook) >97% accuracy on LFW tagging, but sparked privacy
dataset. debates.
Google Neural Huge quality leap in
Replaced phrase-based
Machine translations across 100+
2016 Google Translate with
Translation languages; set new standard
seq2seq + attention model.
(GNMT) for MT.
Enabled rapid prototyping in
2022 (v1), Text-to-image diffusion
Imagen (Google design/marketing;
updates model, high-fidelity and
DeepMind) democratized image creation,
2023–2024 photorealistic outputs.
but raised copyright concerns.

33
Machine Learning: Some Recent Applications
Application /
Year What it does Impact
Product
Next-gen text-to-image Gave designers & brands
generation with modes advanced visual generation
Flux / Flux 1.1 Pro 2024–2025
(“Ultra”, “Raw”), higher tools; reduced creative costs,
resolution and realism. accelerated content pipelines.
Image generation model Increased competition in
Janus-Pro outperforming DALL-E 3 & generative AI; boosted local
2025
(DeepSeek, China) Stable Diffusion in innovation ecosystems;
benchmarks. applied in advertising, media.
Early preventive care; could
AI/ML tool predicting Type-
2024–2025 reduce long-term costs of
NHS Aire-DM (UK) 2 diabetes risk from ECG
(trial rollout) diabetes treatment in public
data up to 13 years ahead.
health.

34
Machine Learning: Some Recent Applications
Application /
Year What it does Impact
Product
Deep learning model Improved accuracy of 6-month
FaceAge (UCL &
estimating biological age survival predictions (80% vs ~61%
Royal Marsden 2025
from facial images to predict
by clinicians); aids personalized
Hospital, UK)
cancer survival. treatment.
Covariant Improved adaptability in
Multimodal deep learning for
Robotics 2023– warehouses, automated item
warehouse robots (vision,
Foundation 2024 picking/sorting; reduced labor
text, sensor data).
Model costs, faster operations.
Quantum machine learning ~20% better performance vs
Quantum Kernel
(QKAR) applied to classical ML in small-data
ML for Chip 2025
semiconductor design (GaN regimes; accelerates chip R&D
Design
HEMTs). efficiency.

35
Machine Learning Systems
● A machine learning system is a complete setup that uses
○ data + algorithms + computing resources
○ to automatically learn patterns, make predictions, or take decisions
○ without being explicitly programmed with rules.
● It’s more than just an algorithm — it includes everything needed to train,
deploy, and use the model in the real world.

36
Different components of an ML system.
• An ML system is not just about the model.
• It’s an end-to-end ecosystem: from business goals → data &
infrastructure → model development → deployment & user interaction.

37
Different components of an ML system.
● 1. ML System Users
○ These are the end users or stakeholders who interact with the ML system.
○ They care about outcomes: predictions, recommendations, insights.
○ Example: Customers using Netflix recommendations, doctors using AI diagnostics.
● 2. Business Requirements
○ Defines what problem the ML system should solve.
○ Example: Fraud detection, customer churn prediction, image classification.
○ It ensures the system is aligned with business objectives (profitability, efficiency, customer
experience).
● 3. ML System Developers
○ These are the engineers, data scientists, and ML practitioners who design, train, and deploy
the system.
○ They work on everything: infrastructure, algorithms, data pipelines, deployment.

38
Different components of an ML system.
● 4. Infrastructure
○ The foundation layer of the ML system.
○ Includes hardware (servers, GPUs), software (ML frameworks like TensorFlow, PyTorch), and
cloud platforms (AWS, GCP, Azure).
○ Ensures scalability, reliability, and security.
● 5. Data
○ The core resource of ML systems.
○ Includes raw data collection, cleaning, preprocessing, and storage.
○ Without quality data, models cannot perform well (“Garbage In, Garbage Out”).
● 6. Feature Engineering
○ Process of transforming raw data into meaningful features for the model.
○ Example:
■ From date of birth → Age
■ From text → Word embeddings
○ Helps improve model accuracy and efficiency.

39
Different components of an ML system.
● 7. ML Algorithms
○ The mathematical models that learn patterns from data.
○ Examples: Linear Regression, Random Forests, Deep Neural Networks.
○ Choice of algorithm depends on task (classification, regression, clustering, etc.).
● 8. Evaluation
○ Measuring how well the ML model performs.
○ Uses metrics: accuracy, F1-score, ROC-AUC, RMSE, etc.
○ Helps avoid overfitting and ensures generalization.
● 9. Deployment, Monitoring, Updating of Logics
○ Once trained and validated, the model is deployed into production (e.g., API, mobile app,
embedded system).
○ Monitoring: Track performance over time (drift detection, accuracy decay).
○ Updating: Retrain with new data to keep the system relevant.

40
 ML research vs. ML production

Research Production

Objectives Model performance* Different stakeholders have

different objectives

41
“*” It’s actively being worked. See Utility is in the Eye of the User: A Critique of NLP Leaderboards (Ethayarajh and Jurafsky, EMNLP 2020)
Stakeholder objectives
ML team
highest accuracy

42
Stakeholder objectives
ML team Sales
highest accuracy sells more ads

43
Stakeholder objectives
ML team Sales Product
highest accuracy sells more ads fastest inference

44
Stakeholder objectives
ML team Sales Product Manager
highest accuracy sells more ads fastest inference maximizes profit

45
 Example: Restaurant Recommender App
Revenue is generated by charging restaurants a 10% service fee on each order.
ML team Sales Product Infrastructure engineers Manager
highest accuracy sells more ads fastest inference prioritize improving maximizes profit
platform
• Want a model • Wants a model • Notices that • As the traffic • Wants to
that recommends that recommends every increase in grows, this team maximize the
restaurants that the more latency leads to a has been woken margin,
users will most expensive drop in orders up in the middle • One way to
likely order from, restaurants since through the of the night achieve this
• Believe they can these restaurants service, because of might be to let go
do so by using a bring in more • They want a problems with of the ML team.
more complex service fees. model that can scaling their
model with more return the existing system,
data. recommended • Want to hold off
restaurants in on model updates
less than 100 ms. to prioritize
improving the ML
platform.
46
Important Takeaways
● When developing an ML project, it’s important for ML engineers to understand
requirements from all stakeholders involved and how strict these requirements
are.
● Production having different requirements from research is one of the reasons
why successful research projects might not always be used in production.
● Example:
○ The BellKor’s Pragmatic Chaos used ensembling to win Netflix Prize (2006–2009) of $1M
for movie recommendation.
○ Yet it’s not widely used in production.
○ Ensembling combines multiple learning algorithms to obtain better predictive performance
than could be obtained from any of the constituent learning algorithms alone.
○ While it can give your ML system a small performance improvement, ensembling tends to
make a system too complex to be useful in production, e.g., slower to make predictions or
harder to interpret the results.

47
Leaderboard-style ML
● More comprehensive utility function
○ Model performance (e.g. accuracy)
○ Latency
○ Prediction cost
○ Interpretability
○ Robustness
○ Ease of use (e.g. OSS tools, community support)
○ Hardware requirements
● Adaptive to different use cases
○ Instead of a leaderboard for each dataset/task, the leaderboard adapts to each company’s
needs
● Dynamic datasets
○ Realistic distribution shifts with different types of shifts

48
Computational priority

Research Production

Objectives Model performance Different stakeholders have different

objectives

Computational priority Fast training, high throughput Fast inference, low latency

generating predictions

49
Latency matters

● 100ms delay can hurt conversion rates by 7% (Akamai study ‘17)

● 30% increase in latency costs 0.5% conversion rate ([Link] ‘19)
● 53% phone users will leave a page that takes >3s to load (Google ‘16)

50
● Latency: time to move a leaf
● Throughput: how many leaves in 1 sec

51
● Real-time: low latency = high throughput
● Batched: high latency, high throughput

52
Latency matters

53
ML in research vs. in production

Research Production

Objectives Model performance Different stakeholders have different

objectives

Computational priority Fast training, high throughput Fast inference, low latency

Data Static Constantly shifting

54
Data
Research Production

● Clean ● Messy
● Static ● Constantly shifting
● Mostly historical ● Historical + streaming data
data ● Biased, and you don’t know
how biased
● Privacy + regulatory
concerns

55
56
ML in research vs. in production

Research Production

Objectives Model performance Different stakeholders have different

objectives

Computational priority Fast training, high throughput Fast inference, low latency

Data Static Constantly shifting

Fairness Good to have (sadly) Important

57
Fairness
● During the research phase, a model is not yet used on people, so it’s
easy for researchers to put off fairness as an afterthought.
● You or someone in your life might already be a victim of biased
mathematical algorithms without knowing it.
○ Your loan application might be rejected because the ML algorithm picks on your zip
code, which embodies biases about one’s socioeconomic background.
○ Your resume might be ranked lower because the ranking system employers use picks
on the spelling of your name.
○ Your mortgage might get a higher interest rate because it relies partially on credit
scores, which favor the rich and punish the poor.
● Other examples of ML biases in the real world are in
○ Predictive policing algorithms
○ Personality tests administered by potential employers
○ College rankings.

58
Some more examples

59
ML in research vs. in production

Research Production

Objectives Model performance Different stakeholders have different

objectives

Computational priority Fast training, high throughput Fast inference, low latency

Data Static Constantly shifting

Fairness Good to have (sadly) Important

Interpretability* Good to have Important

60
Interpretability

61
Interpretability

Result from a class poll

62
Interpretability
● Since most ML research is still evaluated on a single objective, model
performance, researchers aren’t incentivized to work on model
interpretability.
● However, interpretability isn’t just optional for most ML use cases in
the industry, but a requirement.
○ First, interpretability is important for users, both business leaders and end users,
to understand why a decision is made so that they can trust a model and detect
potential biases mentioned previously.
○ Second, it’s important for developers to be able to debug and improve a model.
○ As of 2019, only 19% of large companies are working to improve the
explainability of their algorithms.

63
ML in research vs. in production

Research Production

Objectives Model performance Different stakeholders have different

objectives

Computational priority Fast training, high throughput Fast inference, low latency

Data Static Constantly shifting

Fairness Good to have (sadly) Important

Interpretability Good to have Important

64
ML Research Vs Corporates
● It’s OK to know only the academic and research side of ML.
● Most companies can’t afford research unless it leads to short-term business applications.
● Oftentimes, new models require a massive amount of data and tens of millions of dollars
in compute alone.
● Example: GPU Usage Across OpenAI Models
Parameters GPUs Used
Model Year GPU Type Notes
(approx.) (estimated/known)
Early experiment; trainable on single-node
GPT-1 2018 117M Tens of GPUs NVIDIA V100
clusters.
Required larger clusters, but still manageable
GPT-2 2019 1.5B Hundreds of GPUs NVIDIA V100
by research labs.
Trained on a Microsoft Azure supercomputer
GPT-3 2020 175B ~10,000 GPUs NVIDIA V100
with 400 Gbps InfiniBand.
~1T (est.) / 20,000–25,000 Trained for several months; estimated $50–
GPT-4 2023 NVIDIA A100
MoE design GPUs 100M compute cost.
Not disclosed
50,000+ GPUs Trained on new Azure AI supercomputer;
GPT-5 2025 (likely multi- NVIDIA H100
(est.) estimated cost in the hundreds of millions USD.
trillion / MoE)
65
ML Research Vs Corporates
● As ML research and off-the-shelf models become more accessible, more
people and organizations would want to find applications for them, which
increases the demand for ML in production.
● Most ML-related jobs will be, and already are, in productionizing ML.

66
3. ML systems vs. traditional software

67
 Separation of Concerns is a design principle for
Traditional software separating a computer program into distinct sections
such that each section addresses a separate concern

● Code and data are separate

○ Inputs into the system shouldn’t change the underlying code

68
Image by Arda Cetinkaya
ML systems

● Code and data are tightly coupled

○ ML systems are part code, part data
● Not only test and version code, need to test and version data too

the hard part

69
 Test and version data

● Extremely hard to
ensure correctness in time

70
Back to the Future: Solving the time-travel problem in machine learning - Tecton
ML systems: version data
● Line-by-line diffs like Git doesn’t work with datasets
● Can’t naively create multiple copies of large datasets
● How to merge changes?

71
Difference between SWE and ML Production
Aspect Traditional Software Engineering ML Production
Explicitly programmed rules and algorithms by
Logic Source Learned from historical data through model training.
developers.
Probabilistic – same input may yield slightly different
Deterministic – same input always gives same
Determinism results (due to randomness, retraining, or model
output.
updates).
Trained model + pipeline (data, features, code,
Primary Artifact Codebase is the main deliverable.
weights).
Validation via statistical metrics (accuracy, F1,
Testing Unit, integration, regression tests with expected
precision/recall, AUC, RMSE, etc.). Edge cases are
Approach outputs.
harder to define.
Logic errors, unhandled exceptions, performance Model drift, concept drift, bias, fairness issues,
Failure Modes
bottlenecks. overfitting/underfitting.
Traceable through logs, stack traces, and Harder – errors may come from data quality, distribution
Debugging
breakpoints. shifts, or training instability.
Code-centric: design → implement → test → Data-centric: collect → clean → feature engineer →
Workflow
deploy. train → validate → deploy → monitor → retrain.
Data sources, data pipelines, ML frameworks
Dependencies Libraries, APIs, frameworks, CI/CD pipelines. (TensorFlow, PyTorch, Scikit-learn), GPUs/TPUs,
MLflow/Kubeflow, feature stores.

72
Difference between SWE and ML Production
Aspect Traditional Software Engineering ML Production
Change Changes come from new data, retraining,
Changes come from developer edits to code.
Management hyperparameter tuning, feature engineering.
Continuous – models degrade over time due to
Maintenance Mostly static, occasional bug fixes or updates. changing data distributions → retraining pipelines are
essential.
Scalability Scaling training workloads, serving predictions at low
Scaling code execution and system performance.
Concerns latency, distributed data handling.
Monitor accuracy, drift detection, bias/fairness metrics,
Monitoring Monitor uptime, performance, error rates, logs.
prediction latency, model confidence.
Team Software engineers, QA testers, DevOps Data scientists, ML engineers, MLOps engineers,
Composition engineers. domain experts, data engineers.
Versioning of code + data + model weights (DVC,
Versioning Source code version control (Git).
MLflow, Model Registry).
Model serving via APIs, batch inference, online
Deployment Packaged software releases, containers, APIs.
inference, edge deployment.
Evolution Over Dynamic – requires retraining and adaptation as new
Relatively stable once deployed.
Time data arrives.
Performance is probabilistic, evaluated on test sets and
Evaluation Pass/fail tests, deterministic results.
monitored in production.

73
Engineering challenges with large ML models
● Too big to fit on-device
● As of 2022, it’s common for ML models to have hundreds of millions, if not
billions, of parameters, which requires gigabytes of random-access memory
(RAM) to load them into memory.
● Consume too much energy to work on-device
● Getting large models into production, especially on edge devices, is a massive
engineering challenge.
● Too slow to be useful
● Autocompletion is useless if it takes longer to make a prediction than to type
● If unit/CI tests take hours, the development cycles will stagnate

74
Comparison of different LLMs
Approx # Year
Model Notes / Status
Parameters Released
GPT-3 (OpenAI) ~ 175 B 2020 Dense model, widely reported.
PaLM (Google) ~ 540 B 2022 Dense decoder-only model.
LaMDA (Google) ~ 137 B 2021 Dialogue-focused model.
Wu Dao 2.0 (BAAI,
~ 1.75 T 2021 Multimodal, trillion-scale model.
China)
GPT-4 (OpenAI) Rumored 1-1.8 T 2023 Exact size not disclosed; estimates only.
LLaMA-1 (Meta) 7 B – 65 B 2023 Family of models (7B, 13B, 33B, 65B).
LLaMA-3 / 3.1 (Meta) up to ~ 405 B 2024 Largest version; also smaller 8B & 70B.
~ 109 B total
LLaMA-4 “Scout” (Meta) 2025 Mixture of Experts (MoE).
(17 B active)
LLaMA-4 “Maverick” ~ 400 B total
2025 Larger MoE variant.
(Meta) (17 B active)

75
How to …
● Validate data correctness?
● Test features’ usefulness?
● Detect when the underlying data distribution has changed?
● Know if the changes are bad for models without ground truth labels?
● Detect malicious data?
○ Not all data points are equal (e.g. scans of cancerous lungs are more valuable)
○ Bad data might harm your model and/or make it susceptible to attacks

76
4. ML production myths

77
Myth #1: Deploying is hard

78
 Myth #1: Deploying is hard

Deploying is easy. Deploying reliably is hard

79
Myth #2: You only deploy one or two ML
models at a time

80
 Myth #2: You only deploy one or two ML
models at a time

[Link]: 150+ models, Uber: thousands

81
Image from Ville Tuulos (Netflix, Outerbounds)
Myth #3: You won’t need to update your
models as much

82
DevOps: Pace of software delivery is accelerating
● Elite performers deploy 973x more frequently with 6570x faster lead time to
deploy (Google DevOps Report, 2021)
● DevOps standard (2015)
○ Etsy deployed 50 times/day
○ Netflix 1000s times/day
○ AWS every 11.7 seconds

83
 DevOps to MLOps: Slow vs. Fast

84
Left image from Algorithmia | Right image: Machine learning with Flink in Weibo (Qian Yu, QCon 2019)
DevOps Vs MLOps
Aspect DevOps (Traditional Software) MLOps (Machine Learning Systems)
Automating software development &
Core Focus Automating ML lifecycle (data + model + code).
deployment (CI/CD).
Data Collection → Data Prep/Feature Engineering → Model
Pipeline Stages Code → Build → Test → Deploy → Monitor. Training → Model Validation → Deployment → Monitoring &
Retraining.
Relatively slow – updates are triggered when Fast, continuous – retraining cycles triggered by new data, drift,
Iteration Speed
developers push code changes. or model degradation.
Primary Asset Source code, binaries, configuration. Code + Data + Features + Trained Models (weights).
Unit tests, integration tests, regression tests Model evaluation using metrics (accuracy, F1, RMSE, AUC);
Testing
(clear pass/fail). probabilistic outcomes.
Change Drivers Developer edits to the source code. New data arrival, evolving patterns, model drift, concept drift.
Model accuracy, drift detection, bias/fairness, feature distribution
Monitoring System health, uptime, error rates, logs.
shifts.
Release Cycle Controlled releases after testing. Continuous retraining and redeployment to adapt to new data.
Data scientists, ML engineers, MLOps engineers, data engineers,
Team Involvement Developers, testers, DevOps engineers.
domain experts.
Jenkins, GitLab CI/CD, Docker, Kubernetes, MLflow, Kubeflow, TFX, DVC, Feature Stores, Docker, K8s,
Tools
Terraform. cloud ML services.
Failure Modes Bugs, crashes, infrastructure issues. Model performance drop, data drift, bias, poor generalization.

85
Accelerating ML Delivery

86
ML + DevOps =

87
Myth #4: ML can magically transform your
business overnight

88
Myth #4: ML can magically transform your
business overnight

Magically: possible
Overnight: no

89
 Efficiency improves with maturity

90
2020 state of enterprise machine learning (Algorithmia, 2020)
ML engineering is more engineering than ML
MLEs might spend most of their time:

● wrangling data
● understanding data
● setting up infrastructure
● deploying models

instead of training ML models

Myth #5: Most ML engineers don’t need to
worry about scale

92
 Myth #5: Most ML engineers don’t need to
worry about scale

93
StackOverflow Developer Survey 2019