Model Deployment and

Prediction Service
Moving from Development to Production
Batch vs. Online
Prediction
Fundamentally affects users and developers
Overview of Prediction Modes
The process of generating predictions is called inference

Batch Prediction Online Prediction Online (Streaming) Prediction

Uses only batch features Uses batch features (e.g., Uses both batch and streaming
precomputed embeddings) features
Online Prediction
(Synchronous)
01 02

Request Arrives Generate Prediction

User sends request via RESTful Model processes immediately
API

03

Return Result
Prediction sent back instantly

Example: Google Translate—enter English, get French translation

instantly

Alternative name: On-demand prediction

Batch Prediction
(Asynchronous)
Generate Periodically
Predictions created on schedule

Store Results
Save in SQL tables or in-memory DB

Retrieve When Needed

Fetch precomputed predictions

Example: Netflix generates movie recommendations every 4 hours;

fetched when users log on
Terminology Confusion

"Online" and "batch" can be confusing—both can handle multiple samples or single samples

Alternative Terms Still Imperfect

• Synchronous prediction Online prediction using real-time transport technically

• Asynchronous prediction involves asynchronous requests
Feature Types: Batch vs. Streaming
Batch Features Streaming Features

Computed from historical data in databases or data Computed from real-time data in transports
warehouses
Example: Orders in last 10 minutes, available delivery
Example: Mean restaurant prep time in the past people

Batch Prediction: Uses only batch features Online Prediction: Can use batch features alone OR
both batch and streaming
Online vs. Streaming Features
These terms are sometimes used interchangeably but are different

Online Features
General term for any feature used for online prediction

Includes batch features stored in memory (e.g.,

precomputed item embeddings)

Streaming Features
Refers exclusively to features computed from streaming
data
Hybrid Prediction Solutions
Online and batch prediction don't have to be mutually exclusive

Popular Queries
Precompute predictions in batch

Less Popular
Generate predictions online

Real-world example: DoorDash and UberEats

• Batch: Restaurant recommendations (too slow online)

• Online: Food item recommendations after restaurant selected
Key Differences
Feature Batch Prediction Online Prediction

Frequency Periodical (e.g., every 4 hours) As soon as requests come

Useful for Processing accumulated data when When predictions needed as soon as
immediate results aren't needed data sample generated

Optimized for High throughput Low latency

Examples Recommender systems Fraud detection, autonomous

vehicles
Drawbacks of Batch Prediction
1 2 3

Less Responsive Requires Advance Knowledge Catastrophic Failures

Model doesn't adapt to changing Critical scenarios need real-time:
user preferences until next batch Must know what requests to high-frequency trading,
generate predictions for autonomous vehicles, voice
Example: Netflix user starts
assistants, fraud detection
watching comedy— Unpredictable queries (e.g., text
recommendations won't update translation) require online
for hours prediction
Moving to Online Prediction
Batch prediction is often a workaround when online prediction isn't cheap or fast enough

Pipeline Unification
Fast Model
Unify batch (training) and
Real-Time Pipeline
Generate predictions at acceptable streaming (inference) pipelines
Extract streaming features, input speed (milliseconds for consumer using Apache Flink
to model, return prediction in near apps)
real time
Model
Compression
Making models smaller and faster
Model Compression and Inference Optimization
If a model takes too long to generate online predictions, reduce inference latency

Inference Optimization Model Compression Hardware

Make model do inference faster Make the model smaller Make hardware run faster

Model compression originally aimed at edge devices, but smaller models often run faster too
Four Common Compression Techniques
Low-Rank Factorization
1
Replace high-dimensional tensors with lower-dimensional ones

Knowledge Distillation
2
Small student model mimics larger teacher model

Pruning
3
Remove uncritical parameters or set them to zero

Quantization
4
Use fewer bits to represent parameters

Research papers and open-source utilities growing rapidly

Low-Rank Factorization
Key Idea: Replace high-dimensional tensors with lower-dimensional tensors

SqueezeNets MobileNets

AlexNet-level accuracy with 50× fewer parameters Decomposed standard convolution into depthwise and
pointwise
Replaced 3×3 convolution with 1×1
8-9× parameter reduction

Limitation: Often specific to certain model types (e.g., CNNs), requires significant architectural knowledge
Knowledge Distillation

Teacher Model Student Model Deploy

Large model or ensemble Small model trained to mimic teacher Small model goes to production

Example: DistilBERT

40% 97% 60%

Size reduction Language understanding retained Faster inference

✓ Advantage: Works regardless of architectural differences

✗ Disadvantage: Requires teacher network—adds time and data requirements

Pruning
Originally used for decision trees—now applied to neural networks

Two Meanings Results

1. Remove entire nodes Creates sparse networks

(changes architecture) requiring less storage
2. Set least useful parameters
90%+ reduction in nonzero
to 0 (more common)
parameters without
compromising accuracy
Quantization: The Most Common Method
Most general and commonly used compression method

1-bit
Binary weight neural networks (extreme case)

8-bit
Fixed-point integers (common)

16-bit
Half precision (50% memory reduction)

32-bit
Single precision floating point (default)

Generalizes over tasks and architectures, straightforward to implement

Fixed-Point Quantization
Models represented entirely in integers—each taking only 8 bits

Memory Reduction Speed Improvement Larger Batches

Significantly smaller memory Less precision speeds up Allows increasing batch size
footprint computation, reduces latency

Extreme case: 1-bit representation (BinaryConnect, XNOR-Net)

Quantization: Implementation
Downsides When to Quantize

• Reduced value range During Training: Quantization-aware training in lower precision

• Rounding errors
Post-Training: Train in 32-bit, quantize for inference
• Risk of under/overflow
• Performance changes

Fixed-point inference is standard in industry—TensorFlow Lite, PyTorch Mobile, TensorRT all offer post-training
quantization
Case Study: Scaling BERT at Roblox
Goal: Handle 25,000+ inferences/second at < 20ms latency

Started with large BERT Replaced with DistilBERT

Dynamic shape input Quantized the model

7x 8x
Latency Reduction Throughput Increase
Converting 32-bit to 8-bit integers Biggest performance boost from quantization
Inference Optimization Overview
After code is "lowered" to hardware, performance can still be an issue

The Problem Hand-Optimization The Solution

Naive data movement across Can run 23× faster than standard Optimizing compilers automate
frameworks (pandas, NumPy, libraries the optimization process
TensorFlow) causes slowdown

Optimization engineers are hard to hire and expensive

ML on Cloud and Edge
Where computation happens matters
Cloud vs. Edge Deployment
On the Cloud On the Edge
Computation on public or private clouds Computation on consumer devices
• Easiest starting point • Browsers, phones, laptops
• Managed services (AWS, GCP) • Cars, smartwatches
• ASICs, FPGAs
Downsides of Cloud
Deployment
Cost
ML models are compute-intensive, compute is expensive

Large companies spend hundreds of millions annually on cloud

bills

Mistakes can lead startups to bankruptcy

Internet Reliance
Requires stable internet connection for data transfer

Edge computing works where connections are unreliable or

non-existent
Advantages of Edge Deployment

Cost Control Reduced Latency Privacy & Compliance

Moving computation to edge reduces Network latency can be bigger Easier to comply with GDPR. Limits
required cloud computation, lowering bottleneck than inference latency (up security risk of storing many users'
server costs to seconds). On-device eliminates data centrally
transfer time
Edge Device Requirements
To move computation to edge, devices must have:

Sufficient compute power Enough memory Sufficient battery

Handle the calculation Store and load models Running full BERT can quickly
drain phone battery

30B+
Active Edge Devices
Projected worldwide by 2025

Companies (Google, Apple, Tesla) developing specialized ML chips

Compiling for Arbitrary Hardware
A model built in one framework must be supported by specific hardware

Intermediate Representation (IR)

Bridge between frameworks and platforms

ML Framework
PyTorch, TensorFlow

Hardware Backend
CPUs, GPUs, TPUs

Compilers generate high- and low-level IRs before generating machine code native to hardware. This process is
called "lowering"
Local and Global
Optimization
Local Optimization Techniques
Optimize single operators to speed up models

Vectorization
Execute multiple contiguous elements simultaneously to reduce data I/O latency

Parallelization
Divide input array into independent chunks, perform operation on each individually

Loop Tiling
Change data accessing order to leverage hardware's memory layout and cache (hardware dependent)

Operator Fusion
Fuse multiple operators into one to avoid redundant memory access (e.g., combining loops)
Global Optimization and the
Future
Global Optimization The Future Trend
Leverages higher-level As hardware becomes more
structures of computation powerful and optimized for ML:
graph.
• Online prediction on-device
Achieves bigger speedups than
local optimization. • More responsive to users

ML-Powered Compilers: Use • Private, no network latency

ML techniques (like autoTVM) to • Requires powerful edge
narrow search space and hardware
predict execution time