System Design Case Study: Building a
Scalable MLOps Platform
Design an MLOps Platform for Large-Scale Model Training and Inference
Business Context
A multinational enterprise is building a next-generation ML-driven decisioning system. This
system must train and deploy 10,000 machine learning models, each customized for different
customer segments, geographies, or product lines. Once trained, each model needs to run
batch inference on 750 million records on every model.
The platform must be scalable, cost-efficient, auditable, and automated, supporting
continuous retraining, versioning, monitoring, and governance of the full ML lifecycle.
Design Goals
The goal is to design an end-to-end MLOps Platform that supports:
● Training at scale: Parallel training of 10,000 models across a variety of tasks
(classification, regression, clustering/segmentation), each with <10M training records
● Hyperscale inference: Inference on ~7.5 trillion records in total (750M universe ×
10,000 models) inside a day
● Operational excellence: Continuous integration, model monitoring, alerting.
● Governance and reproducibility: Model lineage, versioning, auditability.
● Cost and compute efficiency. Most optimal choice of tech stack and design that
achieves optimal cost and compute efficiency
Your Task
As a candidate, you're asked to design the architecture and an ongoing operational model
for this platform. Please walk through:
1. Architecture and Components
Describe how you would design the end-to-end MLOps pipeline, including:
a. Data Ingestion & Feature Engineering
● How will raw data be processed?
● Where and how will features be stored and reused?
● How will you optimize feature engineering. Training data has <10M records, 3000+ raw
feature elements which when transformed can lead to 25K-30K+ features. How will you
handle feature engineering and transformation at training and at inference time?
b. Model Training Framework
● How do you orchestrate the training of 10,000 models?
● What compute framework will you use?
● How will models be parameterized and managed?
c. Batch Inference
● How will you scale inference to score 750 million records for 10,000 models inside a
day?
● How would you architect this to be cost-efficient and fault-tolerant?
● What technology and design choices will you make to support this platform?aht show u
2. Technology Stack
Select and justify your choices for:
● ML Frameworks (e.g., TensorFlow, PyTorch, XGBoost)
● Implementation programming language (e.g., Python, Pyspark, Scala, C++, Java)
● Feature Store (e.g., Feast, custom data lake architecture)
● Pipeline Orchestration (e.g., Kubeflow, Airflow, Metaflow)
● Storage (e.g., S3, Parquet, Redis)
● Model Tracking and Versioning (e.g., MLflow, DVC)
● Serving/Deployment (e.g., KServe, Seldon Core)
● Monitoring & Alerting (e.g., Prometheus, Grafana, Evidently)
● Infrastructure as Code (e.g., Terraform, Ansible)
3. Performance & Bottlenecks
Discuss potential challenges and bottlenecks you will encounter while implementing such a
solution and Describe how you'd address these issues in your design.
List the pros/cons, advantages/disadvantages/limitations of their proposed design and what next
steps would you propose to prove your approach will scale and perform to meet the
requirements
4. Scalability & Optimization
● How would you scale model training efficiently to 10,000 models
● How would you parallelize inference over 750 million records/model?
What techniques would you use for compute cost optimization (e.g., Spot Instances,
batch queues)?
● Which processing frameworks will you use for scaling?
5. Governance & Compliance
● How would you ensure reproducibility of each model and inference run?
● How would you track model lineage and data versioning?
● How would you enforce access controls, audit logs, and metadata tracking?
● How would you handle changes to the feature space, such as the retirement or
modification of existing features?
● How would you design for compliance standards such as GDPR, CCPA, HIPAA, or
others?
6. Automation & CI/CD
● How would you set up continuous integration for retraining?
● How would you test model performance before deployment?
● What does your CI/CD pipeline look like for both training and inference?
● How would you handle model ensembles and model-to-model dependencies (models
which require the inference output from other models)?
● How would a data scientist deploy a custom model in this pipeline?
7. Monitoring & Observability
● How would you monitor model drift and data drift at scale?
● What alerting system would you use? How would you integrate this into the automation?
● How would you integrate performance logging and telemetry?
● How would you provide business metrics and observability for non-technical business
stakeholders?
8. Optional Discussion: Real-Time Capabilities
If required in the future, how would your platform adapt to:
● Online learning or incremental updates?
● Low-latency real-time inference for key models?
● Streaming data during training or inference?
Expected Deliverables from Candidate
● High-level architecture diagram
● Detailed explanation of component choices
● Consideration of scale, availability, fault tolerance, cost
● Discussion of bottlenecks and mitigation strategies
○ Pros/Cons, advantages/disadvantages/limitations of their proposed design and
what next steps would you propose to prove your approach
● Approach to automation, reproducibility, and governance
● Smaller scale implementation of such an automated pipeline
○ Pipeline does not need to have all the scalability requirements, but needs
to demonstrate usage of the recommended tools and orchestration
○ You can use tools/platform/dataset of your choice for this small scale
implementation