Module-3

CI/CD and Deployment Strategies for Scalable ML Systems

Model Deployment and Serving, Continuous Integration and Delivery (CI/CD) for ML
Model packaging and containerization (e.g., Docker), Infrastructure provisioning and
orchestration (e.g., Kubernetes), Deploying models as scalable services, managing model
endpoints and versioning, Version control and collaboration (e.g., Git), Building
reproducible ML pipelines, Automated testing and code quality checks, Continuous
integration and deployment strategies.

Model Deployment and Serving:

What is Model Deployment?

Model Deployment is the process of taking a trained machine learning model and
making it available to be used in a real-world environment, such as a website, mobile app,
or production system.

What is Model Serving?

Model Serving is the process of providing predictions (inference) using the deployed
model. It usually involves:
 Sending input data to the model
 Getting predicted results from the model
 Often done through APIs (Application Programming Interfaces)

Key Objectives of Deployment and Serving:

 Make the model accessible to users or systems

 Ensure the model gives real-time or batch predictions
 Maintain speed, reliability, and scalability

Steps in Model Deployment:

1. Train and Save the Model
o Example: Save as .pkl, .h5, or .joblib file
2. Choose Deployment Type
o Local system
o Web server (e.g., Flask, FastAPI)
o Cloud service (e.g., AWS, GCP, Azure)
3. Create an API to Serve the Model
o Use Flask or FastAPI to build an API endpoint (e.g., /predict)
4. Containerize with Docker (optional)
o Makes the deployment portable and consistent
5. Deploy on a Server or Cloud
o Host the model using platforms like:
 Heroku
 Google Cloud Platform (GCP)
 Amazon Web Services (AWS)
 
Microsoft Azure
6. Test and Monitor
o Make sure the model is working correctly
o Monitor performance and errors

Types of Serving:

Type Description Example

Batch Serving Predictions on large data at once Monthly customer reports
Real-Time Serving Instant predictions per request Predicting if email is spam

Tools Used in Deployment and Serving:

 Flask / FastAPI – Create APIs

 Docker – Package and ship applications
 Kubernetes – Manage deployments at scale
 TensorFlow Serving – For serving TensorFlow models
 TorchServe – For PyTorch models

Real-Life Example:
 A model trained to predict heart disease is deployed as a web service.
 A user enters health data into a web form.
 The web app sends this data to the deployed model.
 The model returns a prediction (Yes/No for disease risk).

Key Points to Remember:

 Deployment = Putting model into real use
 Serving = Providing predictions from the model
 Can be local or cloud-based
 Tools like Flask, Docker, and cloud platforms are used
 Continuous monitoring is important

CI/CD for ML Model Packaging and Containerization

What is CI/CD?

 CI/CD stands for:

o CI – Continuous Integration
o CD – Continuous Delivery or Continuous Deployment
 It's a DevOps practice used to automate the steps from coding to deployment.

Why CI/CD in Machine Learning?

In Machine Learning (ML), CI/CD helps to:

 Automatically test, package, and deploy models

 Keep the model up-to-date and reliable
  Reduce manual errors
 Speed up model delivery to production

CI/CD Pipeline in ML (Simplified Steps):

1. Code Commit
o ML code, model training scripts, and configuration files are pushed to Git.
2. Continuous Integration
o Automated tests are run (e.g., unit tests, model accuracy checks)
o Code is merged safely
3. Model Packaging
o Trained model is saved and wrapped with the application logic (API code)
4. Containerization
o The model and code are put into a Docker container
o Ensures it works the same on any system
5. Continuous Delivery/Deployment
o Container is automatically deployed to staging or production
o New model versions are served to users

What is Packaging in ML?

 Putting together:
o Trained model file (e.g., .pkl, .h5)
o API code (Flask/FastAPI)
o Dependencies ([Link])

What is Docker (Containerization)?

 Docker is a tool used to:

o Package the model, code, and environment into one container
o Run it on any machine without configuration issues

CI/CD with Docker – How It Works:

1. Create a Dockerfile
o Defines how to build the environment for your ML model
2. Build Docker Image
o Runs the code, installs dependencies, and prepares the model
3. Push to Container Registry
o e.g., Docker Hub or AWS ECR
4. Deploy
o Run the container on a server or cloud (using Kubernetes or a simple VM)

Common Tools Used:

Task Tools
Version Control Git, GitHub, GitLab
CI/CD Pipelines GitHub Actions, GitLab CI, Jenkins
Containerization Docker
 Deployment Heroku, AWS, Azure, GCP

Benefits of CI/CD in ML:

 Faster delivery of models

 Reduced bugs and errors
 Easier rollback to previous versions
 Better team collaboration
 Works consistently on all machines

Simple Example Flow:

1. Data scientist pushes updated model code to GitHub

2. GitHub Actions automatically:
o Runs tests
o Builds Docker container
o Deploys it to the cloud
3. New model is now live for predictions

Key Points to Remember:

 CI/CD automates testing and deployment

 Docker helps run ML models anywhere
 Pipelines improve speed, quality, and consistency
 Useful in real-world ML systems where updates happen often

Infrastructure Provisioning and Orchestration (e.g., Kubernetes)

What is Infrastructure Provisioning?

 Provisioning means setting up the resources needed to run applications or ML

models.
 These resources include:
o Servers (VMs or physical)
o Storage
o Databases
o Networking tools

Example: Allocating a virtual machine on AWS to deploy a machine learning model.

What is Orchestration?

 Orchestration means automating the management of infrastructure.

 This includes:
o Starting/stopping containers
o Scaling applications
o Load balancing
 o Monitoring and healing failed services

Example: Kubernetes automatically restarts your ML model if it crashes.

Kubernetes: Container Orchestration Platform

 Kubernetes (K8s) is an open-source platform used to automate deployment,

scaling, and management of containerized applications.
 It works well with Docker containers.

Key Features of Kubernetes:

Feature Description
Pods Smallest unit in Kubernetes that runs containers
Nodes Machines where containers run
Clusters Group of nodes
Services Expose your ML model to the internet or other apps
Deployments Manage updates and rollbacks of containers

How ML Uses Provisioning and Orchestration:

Step Action
Train Model Provision GPU-enabled VMs
Package Model Create Docker container
Deploy Model Use Kubernetes for orchestration
Scale Up Kubernetes adds more instances if traffic increases
Monitor Kubernetes restarts container if it fails

Popular Tools Used:

Purpose Tools
Provisioning Terraform, AWS CloudFormation
Orchestration Kubernetes, Docker Swarm
Containerization Docker
Monitoring Prometheus, Grafana

Real-Life Example:

1. An ML model predicting electricity consumption is deployed in a Docker container.

2. Kubernetes is used to run 3 copies of the model to handle more users.
3. If one instance crashes, Kubernetes restarts it automatically.
4. If more users come in, Kubernetes adds new containers as needed.

Benefits of Provisioning + Orchestration:

 Easy to scale ML models automatically

 Reduces manual setup and errors
 High availability and reliability
 Saves time and resources
  Supports continuous deployment

Key Points to Remember:

 Provisioning sets up the needed infrastructure.

 Orchestration manages and automates that infrastructure.
 Kubernetes is widely used to manage containerized ML applications.
 Helps build scalable, reliable, and automated ML systems.

Deploying Models as Scalable Services

What Does It Mean?

 It means turning a trained machine learning model into a web service or API that
can:
o Take input from users
o Give predictions in real-time
o Handle multiple users or requests at once

Why Scalability Matters?

 If only one user uses the model, simple deployment is enough.

 But if 1000s of users or apps access the model, it must:
o Handle high traffic
o Give fast responses
o Stay available and reliable

That’s why scalable deployment is important.

How to Deploy a Scalable ML Model

1. Convert Model to API

 Use Flask or FastAPI to create an endpoint like /predict

 The model is loaded in the backend, and predictions are sent as response

2. Use Containers

 Use Docker to package the model, code, and dependencies into one unit
 Easy to run on any machine

3. Use Orchestration Tools

 Tools like Kubernetes can:

o Run multiple copies of your model (replicas)
o Restart model if it fails
o Balance the load among servers
4. Use Cloud Platforms

 Host the model on:

o AWS (SageMaker, EC2)
o Google Cloud (Vertex AI, GKE)
o Azure ML

5. Add Auto-scaling and Monitoring

 Automatically increase or decrease resources based on:

o Number of users
o CPU or memory usage
 Monitor using Prometheus, Grafana, or cloud dashboards

Basic Flow of Scalable Model Deployment

Client Request → API (Flask/FastAPI) → ML Model → Prediction → Response
|
Docker Container
|
Kubernetes or Cloud Service

Example Use Case

 An online medical app uses a diabetes prediction model.

 The model is deployed as an API.
 As more users log in, Kubernetes adds more containers.
 The model keeps running smoothly with low response time.

Key Tools Used:

Task Tool
API creation Flask, FastAPI
Containerization Docker
Orchestration Kubernetes
Hosting AWS, GCP, Azure
Monitoring Prometheus, Grafana

Key Points to Remember:

 Scalable service = Model that works well even with many users
 Use API + Docker + Kubernetes for best results
 Cloud platforms provide tools for scaling, deployment, and monitoring
 Ensures the model is always available, fast, and reliable

Managing Model Endpoints and Versioning

What is a Model Endpoint?

  A model endpoint is a URL or API address where your deployed model listens for
requests.
 It allows other apps or users to send data and get predictions from your model.

Example: [Link] – this URL sends data to your model and returns
output.

Why Endpoints Are Important:

 They make ML models usable in real-world applications

 Allow easy access to the model from web or mobile apps
 Provide real-time prediction services

What is Model Versioning?

 Model Versioning is the process of keeping track of different versions of your

trained models.
 Every time you improve or retrain the model, you save it as a new version.

Example:

 v1 → Model trained on 2023 data

 v2 → Improved model trained on 2024 data with better accuracy

Why Model Versioning is Important:

 Helps compare model performance

 Allows rollback to previous versions if the new one fails
 Ensures traceability and accountability
 Useful for A/B testing and model improvement tracking

How to Manage Endpoints and Versions:

1. Model Hosting Platforms

 Use cloud services like:

o AWS SageMaker
o Google Vertex AI
o Azure ML
 These platforms help in deploying and managing multiple model versions with
endpoints.

2. Use Tools like MLflow

 MLflow allows you to:

o Track model versions
o Register models
o Deploy models with unique endpoints
3. Endpoint Routing

 You can manage multiple endpoints:

o /predict-v1 → Uses old model
o /predict-v2 → Uses updated model
 Or route traffic automatically (e.g., 70% to v1, 30% to v2 for testing)

4. Monitoring and Logging

 Monitor each version’s:

o Response time
o Accuracy
o User feedback
 Helps in deciding when to upgrade or rollback

Example Use Case:

A bank uses a fraud detection model:

 v1: Works okay, but misses some fraud cases

 v2: Improved version, deployed to a new endpoint
 The bank uses both endpoints for a week to compare results
 After testing, they switch all traffic to v2

Key Tools Used:

Task Tool
Endpoint creation Flask, FastAPI
Version tracking MLflow, DVC
Deployment AWS SageMaker, Azure ML, GCP
Monitoring Prometheus, Grafana, Cloud dashboards

Key Points to Remember:

 Endpoints are used to access the model for predictions

 Versioning keeps track of changes and improvements
 Use cloud or MLOps tools for managing endpoints and versions
 Always monitor and test before switching to a new version

Managing Model Endpoints and Versioning

What is a Model Endpoint?

 A model endpoint is a URL or API address where a deployed machine learning

model can be accessed.
 Other applications (like websites or mobile apps) can send data to this endpoint and
receive predictions in return.
Example: [Link]

What is Model Versioning?

 Model versioning means saving and managing different versions of a machine

learning model.
 Each version may have different:
o Training data
o Parameters
o Accuracy levels

📌 Example:

 Model v1: Trained in January

 Model v2: Trained in June with updated data

Why This is Important

Feature Purpose
Endpoint Allows model to serve predictions via the internet
Versioning Tracks changes and improvements to the model

Benefits of Managing Endpoints & Versions:

 Easy to update or rollback models

 Supports A/B testing with multiple versions
 Ensures reliability and traceability
 Simplifies debugging and monitoring

How to Manage Endpoints and Versioning

1. Using Cloud Platforms

Platforms like:

 AWS SageMaker
 Google Vertex AI
 Azure ML

They allow:

 Hosting multiple model versions

 Creating separate or shared endpoints
 Routing traffic to specific versions

2. Using MLflow

 MLflow is a tool for tracking and managing ML models.

 Features:
 o Model registry
o Auto-versioning
o Serving models with versioned endpoints

3. Manually Managing Endpoints

If using Flask or FastAPI:

 Create separate routes for each version:

o /predict-v1
o /predict-v2
 Switch between versions by updating code or routing logic

Switching Between Model Versions

 Keep older versions live for backup

 Use monitoring to test new versions before full release
 Support gradual rollout (e.g., 10% traffic to new version)

Example Use Case

A retail company uses an ML model to predict sales:

 v1 is live, but v2 (newer model) is more accurate.

 Both versions are deployed with different endpoints.
 After testing v2 for a week, it replaces v1 as the main model.

Key Tools Used

Purpose Tools
Endpoint creation Flask, FastAPI
Model versioning MLflow, DVC
Cloud deployment AWS, GCP, Azure
Monitoring Prometheus, Grafana

Key Points to Remember

 A model endpoint lets apps send data and receive predictions.

 Versioning helps keep track of model improvements.
 Use cloud tools or ML platforms to manage them easily.
 Good management leads to more stable, scalable, and safe ML systems.

Version Control and Collaboration (e.g., Git)

What is Version Control?

  Version Control is a system that tracks changes in files (like code, documents, or
ML models) over time.
 It helps you:
o Save different versions
o Go back to a previous version
o Work on new updates without losing old work

Example: Updating your ML model code and still being able to access the older version if
needed.

What is Git?

 Git is a popular version control tool.

 It helps developers and data scientists:
o Track code changes
o Collaborate with others
o Share updates without overwriting others' work

Why Use Git in Machine Learning Projects?

 Manage changes in:

o Data preprocessing scripts
o Model training code
o Notebooks and results
 Work in teams without conflict
 Keep history of experiments and improvements

Basic Git Commands You Should Know

Command Purpose
git init Start a new Git project
git clone Copy an existing Git repo
git add Stage files for commit
git commit Save changes with a message
git push Upload changes to remote (e.g., GitHub)
git pull Download latest changes from team
git status Show changes made
git log Show commit history

Collaboration with Git (Using GitHub/GitLab)

 Team members can:

o Work on different branches
o Merge their changes together
o Review each other’s code using pull requests
 Helps prevent code conflicts and mistakes

Platforms for Git Collaboration:

 Platform Use
GitHub Most popular platform for open-source and team projects
GitLab Git + CI/CD tools built-in
Bitbucket Git-based tool by Atlassian, integrates with Jira

Using Git in ML Projects

 Track scripts (.py, .ipynb)

 Store model versions (.pkl, .h5)
 Save configuration files ([Link], [Link])
 Keep documentation ([Link])

Real-Life Example:

Two students are working on a heart disease prediction model:

 One updates the data cleaning script

 Another trains the model
 Both push changes to GitHub
 Git merges the code safely and keeps a log of who did what

Benefits of Using Git

 Track all project changes

 Recover old versions
 Share and merge team work safely
 Manage multiple features with branches
 Improve team productivity and project reliability

Key Points to Remember

 Git is a tool for version control and team collaboration.

 Helps track code changes, avoid errors, and work with others.
 Platforms like GitHub and GitLab make collaboration easier.
 Important for managing machine learning code, experiments, and results.

Building Reproducible ML Pipelines

What is a Reproducible ML Pipeline?

A reproducible ML pipeline is a structured and automated process that ensures the
same ML results can be repeated every time using the same code, data, and environment.

Example: If you train a model today and retrain it a week later with the same data, you
should get the same result.

Why Reproducibility is Important?

  Ensures consistency in results
 Makes it easy to debug and fix issues
 Helps with team collaboration and sharing work
 Required for auditing, compliance, and research

Key Components of an ML Pipeline

1. Data Collection
o Load and store raw data
2. Data Preprocessing
o Clean, transform, and split data
3. Feature Engineering
o Create new features to improve performance
4. Model Training
o Train the model with consistent settings
5. Model Evaluation
o Measure performance using accuracy, F1-score, etc.
6. Model Deployment
o Make the model available for prediction via API

How to Make ML Pipelines Reproducible

1. Use Version Control

 Use Git to track changes in code and collaborate safely

2. Fix Random Seeds

 Set a random seed in your ML code (e.g., random_state=42)

 Ensures same data splits and model behavior

3. Use Environment Files

 Use [Link] or [Link] to lock libraries and versions

 Use tools like conda or virtualenv to manage environments

4. Use Pipelines or Workflow Tools

 Tools like Scikit-learn Pipelines, MLflow, Kubeflow, or Apache Airflow

 Automate and organize steps

5. Log Everything

 Use tools like MLflow, Weights & Biases, or even Excel to log:
o Data version
o Parameters used
o Model accuracy
 o Date and time of experiments

6. Use Data Versioning Tools

 Tools like DVC (Data Version Control) help track dataset changes just like Git does
for code

Tools That Help in Reproducible ML

Purpose Tool
Code versioning Git, GitHub
Environment management Conda, Docker
Experiment tracking MLflow, W&B
Data versioning DVC
Workflow automation Airflow, Kubeflow

Example Use Case:

A team is working on a model to predict student performance.

 One person writes preprocessing code

 Another works on model training
 They use Git to share code, MLflow to track experiments, and DVC to manage data
 A new team member can replicate the same results using the pipeline setup

Key Points to Remember

 Reproducibility means getting the same result every time

 Use version control, fixed seeds, and environment files
 Tools like MLflow, DVC, and Docker help automate and track everything
 A reproducible pipeline is essential for professional ML development

Automated Testing and Code Quality Checks

What is Automated Testing?

 Automated testing means writing scripts that automatically check if your code
works correctly.
 It saves time and avoids human error by running tests every time the code changes.

Example: A test script checks if your ML model loads correctly and returns a valid
prediction.

What are Code Quality Checks?

 Code quality checks are tools and rules used to ensure that code is:
o Clean
 o Readable
o Well-structured
o Error-free

Example: Checking if variables have meaningful names or if there are unused imports.

Why These Are Important in ML Projects:

Benefit Explanation
Reliability Ensures your model code works as expected
Speed Detects bugs early in development
Collaboration Makes it easier for teams to understand and work on the same code
Maintainability Cleaner code is easier to update or debug later

Types of Automated Tests in ML Projects

Test Type What It Checks

Unit Test Checks small pieces of code (e.g., a function that scales data)
Integration Test Checks if different parts of your code work well together
Regression Test Ensures new changes don’t break existing features
Model Test Checks if the model runs, predicts, and performs correctly

Tools for Automated Testing

Tool Use
PyTest Simple Python testing framework
unittest Built-in Python testing module
nose2 Extension of unittest
doctest Tests from examples in docstrings

How to Write a Simple Test (Example with PyTest)

def add(a, b):

return a + b

def test_add():
assert add(2, 3) == 5

Run it with the command:

pytest test_script.py

Tools for Code Quality Checks

Tool What It Does

Pylint Checks for coding errors, style issues
Flake8 Checks for Python syntax and style problems
Black Automatically formats Python code
 Prettier Formats code for many languages (JavaScript, HTML, etc.)

Use in CI/CD Pipelines

 Integrate testing and quality checks in CI/CD pipelines (e.g., GitHub Actions, GitLab
CI)
 Automatically run tests and checks on every code push or pull request

Example Use Case:

You write a function to normalize input data.

A unit test ensures it returns values between 0 and 1.
Before pushing your code, Pylint checks that your code follows best practices.
If tests fail, the CI/CD pipeline blocks the deployment.

Key Points to Remember

 Automated testing ensures your code works correctly
 Code quality checks ensure your code is clean and professional
 Use tools like PyTest, Pylint, and Black
 Run tests regularly and in CI/CD pipelines
 Essential for trustworthy, scalable, and team-friendly ML development

Continuous Integration and Deployment Strategies (CI/CD)

What is Continuous Integration (CI)?

 Continuous Integration (CI) is a process where developers frequently merge their

code changes into a shared repository.
 Each change is automatically tested to ensure it doesn't break anything.

Example: A student pushes model code to GitHub, and a CI tool runs tests instantly.

What is Continuous Deployment (CD)?

 Continuous Deployment (CD) is the practice of automatically deploying every

change that passes the CI tests to production.
 Sometimes, Continuous Delivery is used instead, where changes are deployed
manually after testing.

Example: Once your model passes all tests, it gets deployed as an API automatically.

Why CI/CD is Important in ML Projects

Benefit Description
Speed Fast delivery of updates and features
Reliability Automatic testing reduces bugs
Consistency Same steps every time ensures fewer mistakes
 Teamwork Makes collaboration smooth and organized

CI/CD Tools You Can Use

Tool Use
GitHub Actions Create custom workflows for CI/CD
GitLab CI/CD Built-in CI/CD pipelines
Jenkins Open-source automation server
CircleCI / Travis CI Cloud-based CI/CD services

CI/CD Workflow (Simplified)

1. Code Push → Developer commits code to Git

2. CI Triggered → Automated tests run
3. ✅ If Tests Pass → Model is packaged (e.g., Docker)
4. CD Starts → Code or model is deployed to:
o Test server (for staging)
o Production server (live)

CI Strategies
Strategy Description
Build on Every Commit Test every code change instantly
Scheduled Builds Run tests at regular times (e.g., nightly)
Branch-Based CI Only run CI on certain branches like main or dev

CD Strategies
Strategy Description
Manual Deployment Developer manually pushes code to production
Auto Deployment Code is automatically deployed after tests pass
Blue-Green Two identical environments (Blue & Green); switch traffic to the
Deployment new one after deployment
Canary Deployment Gradually send traffic to new version (e.g., 10% → 50% → 100%)
to detect issues early

Example Use Case:

An ML model predicting crop yield is developed by a team.

 Developer pushes code to GitHub

 GitHub Actions run unit tests
 If all tests pass, a Docker container is built
 The container is deployed to AWS EC2 using CI/CD
 Monitoring tools check performance automatically

Key Points to Remember

 CI/CD automates testing and deployment of code and models

 CI ensures code works before it's merged
 CD ensures working code is delivered to users quickly and safely
 Use strategies like blue-green or canary for safer deployments
 Helps teams work faster, safer, and more efficiently