21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
[Link]
PDF 616.03KB
Take the attached file and summarize the main ideias (I need to
understand what the concepts mean) of each sections and
subsections with topics considering the main ideas page by page. If
needed to express the main ideas and not lose good information, the
subsections can be described using a couple of topics. do all page.
Extract also the code from each page in a formatted way with
reserved words colored.
Of course. Here is a detailed, page-by-page summary of the main ideas and
concepts from the provided chapter, along with any code snippets presented in a
formatted style.
Page 1: Chapter 2. Introduction to Machine Learning Systems
Design
Main Ideas:
Holistic Approach: ML Systems Design involves considering the entire system—
business needs, data, infrastructure, deployment, monitoring—and how all
components and stakeholders work together.
Chapter Roadmap: The chapter will cover:
1. Objectives: Understanding why the system is needed, translating business
objectives into ML objectives.
2. Requirements: Defining what makes a good system (reliability, scalability,
maintainability, adaptability).
3. Iterative Process: The non-linear, cyclical nature of developing ML systems.
4. Problem Framing: How to formulate a business problem as a task ML can solve.
5. Data vs. Algorithms: A discussion on the relative importance of data and
intelligent algorithms.
Page 2: Business and ML Objectives
[Link] 1/10
21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
Main Ideas:
The ML-Business Gap: Data scientists focus on ML metrics (accuracy, F1 score),
but companies care about business metrics (profits, sales, costs, customer
satisfaction).
Project Failure: A common reason for ML project failure is focusing solely on
improving ML metrics without demonstrating an impact on business metrics.
Ultimate Business Goal: The primary purpose of a business is to maximize
shareholder profits. Any project, including ML, must contribute to this goal
directly or indirectly.
Linking ML to Business: To succeed, you must tie the ML system's performance
to business performance (e.g., ad revenue, active users).
Page 3: Business and ML Objectives (Continued)
Main Ideas:
Clear Mapping Examples:
Ad Click-Through & Fraud Detection: Easy to map ML performance (click-
through rate, fraud caught) directly to revenue or cost savings.
Netflix's Take-Rate: A custom metric (quality plays / recommendations shown)
that is linked to business metrics like total streaming hours and lower
cancellation rates.
Complex Relationships: The effect of an ML project on business goals can be
complex and sometimes contradictory (e.g., a model could make customers
happier but also let them spend less).
Experimentation: A/B testing is crucial to understand the real impact of an ML
model on business metrics, sometimes favoring a model with worse ML metrics
but better business outcomes.
Attribution Challenges: In complex systems (e.g., cybersecurity), it can be
impossible to trace a business outcome (a security breach) back to the
performance of the ML component.
Page 4: Business and ML Objectives (Continued)
[Link] 2/10
21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
Main Ideas:
The "AI-Powered" Hype: Some companies use ML primarily for marketing, even if
the AI component isn't useful.
Realistic Expectations: ML can transform businesses, but not overnight. It
requires long-term investment (e.g., Google's decades of work).
ROI and Maturity: The return on investment in ML improves with maturity.
Companies with more experience have more efficient pipelines and faster
deployment cycles.
Page 5: Requirements for ML Systems & Reliability
Main Ideas:
Core Requirements: Most ML systems should aim for four key characteristics:
Reliability, Scalability, Maintainability, and Adaptability.
Reliability: The system performs the correct function at the desired level despite
faults (hardware, software, human error).
The "Correctness" Challenge for ML:
A system can call [Link]() correctly but still produce wrong predictions.
Unlike traditional software that crashes or shows errors, ML systems can fail
silently. Users may not realize the system has failed (e.g., a bad translation
from Google Translate).
Page 6: Scalability
Main Ideas:
Ways an ML System Can Grow:
1. Complexity: Moving from simple models (logistic regression) to complex ones
(large neural networks) requiring more resources.
2. Traffic Volume: Serving more prediction requests as the user base grows.
3. Model Count: Managing many models, especially in enterprise contexts (e.g.,
one model per customer).
[Link] 3/10
21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
Scalability Solutions:
Resource Scaling: Up-scaling (adding resources) and down-scaling (removing
resources) as needed.
Autoscaling: Cloud feature to automatically adjust resources, though it can be
tricky to implement (e.g., Amazon Prime Day outage).
Page 7: Scalability & Maintainability
Main Ideas:
Scalability is More Than Resources: It also includes Artifact Management.
One Model: Can be managed manually.
Hundreds of Models: Requires automation for monitoring, retraining, and code
management to ensure reproducibility.
Maintainability: The system should be structured so that people from different
backgrounds (ML engineers, DevOps, SMEs) can collaborate effectively.
Key Practices for Maintainability:
Allow contributors to use tools they are comfortable with.
Document code.
Version code, data, and artifacts.
Ensure models are reproducible.
Enable collaborative problem-solving without blame.
Page 8: Adaptability & Iterative Process
Main Ideas:
Adaptability: The system must be able to adapt to changing data and business
needs. This includes discovering improvements and updating without service
interruption. It is closely linked to maintainability.
Iterative Process: Developing an ML system is iterative and never-ending. It is
not a linear process but a cycle with constant back-and-forth between steps.
[Link] 4/10
21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
Page 9: Iterative Process (Continued)
Main Ideas:
Example Workflow: A detailed example of building an ad prediction model shows
the iterative nature, including:
Relabeling data after error analysis.
Dealing with class imbalance.
Retraining on stale data.
Changing the business objective (from impressions to click-through rate).
Visualization: Figure 2-2 represents the process as a cycle with six key steps.
Page 10 & 11: The Iterative Process Steps
Main Ideas:
Step 1: Project Scoping: Define goals, objectives, constraints, identify
stakeholders, and allocate resources.
Step 2: Data Engineering: The foundation. Involves handling data from various
sources and curating training data (sampling, labeling).
Step 3: ML Model Development: Feature engineering, model selection, training,
and evaluation.
Step 4: Deployment: Making the model accessible to users.
Step 5: Monitoring and Continual Learning: Monitoring for performance decay
and maintaining the model against changing environments.
Step 6: Business Analysis: Evaluating model performance against business goals
to generate insights for scoping new projects or killing unproductive ones.
Page 12: Framing ML Problems
Main Ideas:
Business Problem vs. ML Problem: A business problem (e.g., "slow customer
support") is not an ML problem. An ML problem is defined by inputs, outputs, and
[Link] 5/10
21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
an objective function.
Example - Customer Support: The bottleneck is routing. This can be framed as a
classification problem:
Input: Customer request.
Output: Department (Accounting, Inventory, HR, IT).
Objective: Minimize the difference between predicted and actual department.
Page 13 & 14: Types of ML Tasks
Main Ideas:
Output dictates the task type. The main categories are Regression and
Classification.
Classification vs. Regression:
Classification: Outputs a category (e.g., spam/not spam).
Regression: Outputs a continuous value (e.g., house price).
They can be converted into each other (e.g., quantizing house prices into
buckets for classification, or using a regression output with a threshold for
spam detection).
Binary vs. Multiclass Classification:
Binary: Two classes (e.g., toxic/not toxic). Simpler.
Multiclass: More than two classes.
High Cardinality: A very large number of classes (e.g., thousands of diseases).
Presents data collection challenges, especially for rare classes. Hierarchical
classification can help.
Page 15: Multiclass vs. Multilabel Classification
Main Ideas:
Multiclass: Each example belongs to exactly one class.
Multilabel Classification: Each example can belong to multiple classes (e.g., an
article about tech and finance).
[Link] 6/10
21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
Approaches to Multilabel:
1. Represent the label as a vector with multiple 1 s (e.g., [0, 1, 1, 0] ).
2. Train multiple binary classification models (one per class).
Challenges of Multilabel:
Label Annotation: Harder due to label multiplicity and annotator disagreement.
Prediction Extraction: It's not clear how many classes to pick from the raw
probability output (e.g., pick top 2? top 3?).
Page 16 & 17: Multiple Ways to Frame a Problem
Main Ideas:
Framing Impacts Difficulty: How you frame a problem can make it much easier or
harder.
Example: App Prediction
Bad Framing (Multiclass): Input = user/environment features. Output =
probability distribution over all N apps.
Problem: Adding a new app requires retraining the entire model.
Good Framing (Regression): Input = user features, environment features, and a
specific app's features. Output = a single score for that app.
Advantage: To add a new app, you just need its features and run the existing
model; no retraining needed.
Page 17 & 18: Objective Functions
Main Ideas:
Objective/Loss Function: Guides the learning process by measuring the error of
predictions. The goal is to minimize this loss.
Common Loss Functions:
Regression: RMSE (Root Mean Squared Error), MAE (Mean Absolute Error).
Binary Classification: Logistic Loss (Log Loss).
[Link] 7/10
21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
Multiclass Classification: Cross Entropy.
Code Example: Calculating Cross Entropy loss.
Code from Page 18:
python
import numpy as np
def cross_entropy(p, q):
return -sum([p[i] * [Link](q[i]) for i in range(len(p))])
p = [0, 0, 0, 1] # Ground truth label
q = [0.45, 0.2, 0.02, 0.33] # Model's predicted probability
cross_entropy(p, q)
import : A keyword to include the numpy library.
def : A keyword to define a function named cross_entropy .
return : A keyword to specify the output of the function.
for & in : Keywords to create a loop that iterates over a range.
range : A function to generate a sequence of numbers.
len : A function to get the length of a list.
Page 18, 19 & 20: Decoupling Objectives
Main Ideas:
Multiple Objectives: Real-world systems often have multiple, sometimes
conflicting, goals (e.g., maximize engagement vs. minimize misinformation).
Combined Loss Approach: Merge losses into one:
loss = α * quality_loss + β * engagement_loss .
Problem: Tuning α and β requires retraining the model.
Decoupled Approach (Preferred): Train separate models for each objective (e.g.,
a quality_model and an engagement_model ). Then combine their scores:
α * quality_score + β * engagement_score .
Advantages:
1. Tune α and β without retraining the models.
[Link] 8/10
21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
2. Easier maintenance as different models can be updated on different
schedules.
Page 20 & 21: Mind Versus Data
Main Ideas:
The Central Debate: What is more important for ML success: intelligent algorithm
design ("Mind") or large amounts of data ("Data")?
The "Mind-over-Data" Camp:
Judea Pearl: Emphasizes "Mind over Data," arguing that current data-centric
approaches are limited and will become outdated.
Christopher Manning: argues that structure and intelligent design allow
systems to learn more from less data.
The "Data-over-Mind" Camp:
Richard Sutton's "Bitter Lesson": General methods that leverage massive
computation (and thus, data) are ultimately the most effective in the long run.
Peter Norvig: "We don't have better algorithms. We just have more data."
Monica Rogati's Data Science Hierarchy: Data is the foundation; without it,
there is no data science.
Page 22 & 23: Mind Versus Data (Continued)
Main Ideas:
The Nuance: The debate is not about whether data is necessary, but whether a
finite amount of it is sufficient. Infinite data might solve everything, but we only
have finite data.
Industry Trend: Recent progress (e.g., AlexNet, BERT, GPT) heavily relies on ever-
larger datasets.
Data Quality Matters: More low-quality data (outdated, mislabeled) can hurt
performance, not help it.
[Link] 9/10
21/11/2025, 17:07 ML Systems Design and Business Objectives Overview - DeepSeek
Page 23 & 24: Summary
Main Ideas:
Start with "Why": ML projects must be driven by business objectives, which are
translated into ML objectives.
Define Requirements: A good system meets requirements like reliability,
scalability, maintainability, and adaptability.
Iterative, Not Linear: ML development is a cyclical, ongoing process.
Data is Crucial: Regardless of the "Mind vs. Data" debate, data's importance in
modern ML is undeniable. A significant part of the book will focus on data.
Looking Ahead: The following chapters will zoom into the building blocks of ML
systems, starting with data engineering.
[Link] 10/10