Detailed Exam Notes on Distributed Machine Learning and Ensembling Techniques
1. Data Slicing
Explanation:
Data slicing in distributed machine learning involves partitioning a large dataset into
smaller subsets, or "slices," to enable parallel processing across multiple machines or
nodes. This technique is essential for handling massive datasets that exceed the memory
or computational capacity of a single device, significantly reducing training time and
improving efficiency. It is a cornerstone of data parallelism, ensuring balanced workload
distribution and fault tolerance.
Key aspects include:
• Partitioning Strategies:
o Horizontal Slicing: Divides data by rows, assigning subsets of records (e.g.,
user transactions) to different nodes.
o Vertical Slicing: Splits data by columns, focusing on specific features (e.g.,
age, income), useful for feature-specific processing.
o Randomized Slicing: Randomly distributes data to ensure diversity and
prevent bias, critical for tasks like natural language processing.
o Stratified Slicing: Maintains class proportions in each subset, vital for
imbalanced datasets like fraud detection.
• Preprocessing: Requires cleaning (removing missing values, outliers),
normalization (scaling features), shuffling (to avoid bias), and augmentation (to
increase dataset variety).
• Applications: Used in image processing (e.g., splitting large image datasets), text
analysis (e.g., sentiment analysis), and IoT (e.g., sensor data for predictive
maintenance).
• Tools: Lightweight tools like Pandas and NumPy for smaller datasets, and scalable
frameworks like Apache Spark, TensorFlow, and PyTorch for larger datasets.
• Challenges: Include load imbalance (uneven slices causing idle nodes), data skew
(non-representative slices introducing bias), and communication overhead
(synchronizing results across nodes).
Example:
A dataset of 10 million customer transactions is horizontally sliced into 10 subsets of 1
million records each. Each subset is processed by a different node in a cluster. For
instance, Node 1 handles transactions 1-1,000,000, Node 2 handles 1,000,001-2,000,000,
and so on. Each node trains a recommendation model on its subset, and the results are
aggregated to update the global model, reducing training time from days to hours and
ensuring fault tolerance if a node fails.
Diagram:
A diagram of data slicing illustrates the process as follows:
• A large dataset is represented as a single block.
• Horizontal Slicing: The block is divided into rows, creating subsets (Slice 1, Slice 2,
etc.).
• Vertical Slicing: The block is divided into columns, creating feature-specific
subsets (Slice 3, Slice 4, etc.).
• Analysis: Each slice is processed in parallel by different nodes, and results are
combined for final analysis.
This can be visualized as:
graph LR;
A["Data"] -->|"Horizontal Slicing"| B(Slice 1);
A -->|"Horizontal Slicing"| C(Slice 2);
A -->|"Vertical Slicing"| D(Slice 3);
A -->|"Vertical Slicing"| E(Slice 4);
B --> F["Analysis"];
C --> F;
D --> F;
E --> F;
Source: NumberAnalytics: Data Slicing, Snorkel: Slice-based Learning
2. Model Slicing
Explanation:
Model slicing, synonymous with model parallelism, involves dividing a large machine
learning model into smaller components, such as layers or operations, and distributing
them across multiple devices or nodes. This approach is critical for training models that
exceed the memory or computational capacity of a single device, such as large
transformers like BERT.
Key aspects include:
• Partitioning Strategies:
o Layerwise Partitioning: Assigns different layers (e.g., convolutional, dense)
to separate devices.
o Operator-Level Partitioning: Splits computational operations (e.g., matrix
multiplications) across devices.
o Pipeline Parallelism: Divides the model into sequential stages, processed
on different devices.
o Tensor Slicing: Breaks large tensors (e.g., weights, activations) into smaller
parts for parallel processing.
• Preprocessing: Involves profiling to identify computational bottlenecks, analyzing
layer dependencies, quantizing to reduce precision, and pruning redundant
connections.
• Applications: Used for large language models (e.g., transformers), autonomous
systems (e.g., self-driving cars), and healthcare (e.g., medical imaging).
• Tools: TensorFlow, PyTorch, and NVIDIA Triton support model slicing.
• Challenges: Include managing dependencies between model parts, handling
communication overhead, and addressing synchronization delays.
Example:
A deep neural network for image recognition with convolutional and dense layers is split
using pipeline parallelism. Stage 1 (convolutional layers) is assigned to GPU 1, Stage 2
(pooling and flattening) to GPU 2, and Stage 3 (dense layers) to GPU 3. Input data flows
through each stage, with intermediate outputs passed between GPUs, enabling concurrent
processing and reducing training time.
Diagram:
A diagram of model parallelism shows:
• Model Weights Partitioning: Model weights are divided across cores, with larger
matrices (e.g., Feed Forward Network layers) represented by different sizes and
colors.
• Data Batch Splitting: The data batch is split across cores, each holding an equal
number of tokens, ensuring consistent memory usage.
This can be visualized as a 4x4 grid of 16 cores, with the first row showing model
weights distributed across cores and the second row showing data batches split
across cores.
Source: Towards Data Science: Distributed Training, arXiv: Google Switch Transformers
3. Model Serving
Explanation:
Model serving is the deployment of a trained machine learning model to generate
predictions or inferences on new data, often in real-time or batch scenarios. In distributed
systems, it requires scalability to handle high request volumes and reliability to ensure
fault tolerance.
Key aspects include:
• Architecture: Involves model storage (e.g., .h5 files), a model server (e.g., FastAPI),
and a load balancer to distribute requests.
• Scaling Techniques:
o Horizontal Scaling: Replicates model servers across multiple machines to
balance load.
o Sharded Services: Splits large requests into smaller parts for parallel
processing.
o Load Balancing: Uses algorithms like round-robin to distribute requests
evenly.
• Applications: Used in video tagging (e.g., YouTube), digit recognition (e.g., MNIST),
and recommendation systems.
• Tools: FastAPI, Ray Serve, and TensorFlow Serving.
• Challenges: Include latency, fault tolerance, and resource management, addressed
by replication and load balancing.
Example:
A digit recognition model trained on the MNIST dataset is deployed using FastAPI. Users
upload images, and the server preprocesses them to predict digits (e.g., "7"). To handle
thousands of requests, the server is replicated across four machines, with a load balancer
distributing requests using a round-robin algorithm, ensuring low latency and high
reliability.
Diagram:
A model serving architecture includes:
• Model Server: Hosts the trained model and processes requests.
• Replicated Servers: Multiple server copies for horizontal scaling.
• Load Balancer: Distributes requests to servers.
• Sharded Services: Large requests are split into shards, processed in parallel, and
results aggregated.
Source: Xebia: Model Serving Architectures, Manning: Model Serving Patterns
4. Data Parallelism
Explanation:
Data parallelism involves dividing a large dataset into smaller subsets, with each subset
processed by a different worker node running the same model. Gradients computed by
each node are aggregated (e.g., averaged) by a parameter server to update the global
model. This approach is ideal for large datasets, enabling faster training through parallel
processing.
Key aspects include:
• Mechanism: Each node trains a replicated model on its data subset, sending
gradients to a parameter server for aggregation.
• Techniques:
o Batching: Splits data into mini-batches for efficient processing.
o All-Reduce: A communication-efficient method for gradient aggregation.
o Caching: Stores data in memory or SSD to reduce latency in multi-epoch
training.
• Applications: Used in recommendation systems (e.g., Netflix), image classification,
and NLP tasks.
• Tools: TensorFlow, PyTorch, and Apache Spark.
• Challenges: Include synchronization overhead (e.g., asynchronous updates) and
load balancing, addressed by queuing or locking mechanisms.
Example:
A convolutional neural network is trained on 10,000 images, split into three subsets (3,000,
3,000, 4,000 images) across Workers 1, 2, and 3. Each worker trains the same model,
computes gradients, and sends them to a parameter server, which averages them to
update the model, reducing training time significantly.
Diagram:
A diagram of data parallelism shows:
• Data Splitting: The dataset is divided into subsets, each assigned to a worker node.
• Model Replication: Each node has a copy of the same model.
• Gradient Aggregation: Gradients from all nodes are aggregated to update the global
model.
Source: Towards Data Science: Distributed Training, Medium: Distributed ML Training
5. Model Parallelism
Explanation:
Model parallelism divides a large model across multiple devices or nodes, with each
handling a specific part (e.g., layers). The data flows through the distributed model, with
intermediate outputs passed between nodes. This is suitable for large models like
transformers that exceed single-device memory capacity.
Key aspects include:
• Mechanism: Each node processes a portion of the model, computing gradients for
its part, which are aggregated to update the model.
• Techniques:
o Layerwise Partitioning: Assigns different layers to different devices.
o Pipeline Parallelism: Divides the model into sequential stages.
o Tensor Parallelism: Splits tensors across devices.
• Applications: Used for large language models (e.g., BERT), autonomous systems,
and medical imaging.
• Tools: TensorFlow, PyTorch, and NVIDIA Triton.
• Challenges: Include dependency management, communication overhead, and
synchronization delays, addressed by fixed responsibility assignments.
Example:
A neural network with convolutional (Conv1, Conv2) and dense (Dense1) layers is split
across three parameter servers (PS1, PS2, PS3). Workers process data through their
assigned layer, compute gradients, and send them to the respective servers, which
aggregate and update the model, Ascending(1)Descending(0)model, enabling training on
limited-memory devices.
Diagram:
A diagram of model parallelism shows:
• Model Splitting: The model is divided into parts (e.g., layers), each assigned to a
different core.
• Data Flow: Data flows through the distributed model, with intermediate outputs
passed between cores.
Source: Towards Data Science: Distributed Training, Colossal-AI: Parallelism Paradigms
6. Data Parallelism vs. Model Parallelism
Comparison:
Data parallelism and model parallelism are two primary strategies for distributed machine
learning, each suited to different scenarios:
Aspect Data Parallelism Model Parallelism
Splits dataset across nodes, Splits model across nodes, each
Definition
each running the same model. handling a different part.
Large datasets, manageable Large, complex models exceeding
Use Case
models. single-device memory.
Dataset divided into subsets, Entire dataset or subset processed
Data Distribution
each node processes a subset. through distributed model parts.
- Model Same model replicated across Model partitioned, different parts on
Distribution nodes. different nodes.
Requires coordination of
Requires gradient aggregation
Synchronization intermediate outputs between model
across nodes.
parts.
Aspect Data Parallelism Model Parallelism
Scalable, efficient for data-
Handles large models, balances
Advantages heavy tasks, simpler
computational load.
implementation.
Dependency management,
Synchronization overhead, load
Challenges communication overhead, complex
balancing issues.
implementation.
TensorFlow mirror strategy, TensorFlow parameter server
Tools
PyTorch DDP. strategy, PyTorch pipeline parallelism.
Example:
• Data Parallelism: Training a model on 10,000 images split across three nodes, each
with the same model, aggregating gradients to update the model.
• Model Parallelism: Splitting a 10-layer neural network across three nodes, each
processing a different layer, with data flowing sequentially through the layers.
Diagram:
• Data Parallelism: Dataset split into subsets, each processed by a node with the
same model, gradients aggregated centrally.
• Model Parallelism: Model split into parts, each processed by a different node, with
data flowing through the distributed model.
Source: Analytics India Mag: Parallelism Comparison, Medium: Model Parallelism
7. Federated Learning
Explanation:
Federated learning is a privacy-preserving distributed machine learning approach where
multiple clients (e.g., smartphones, hospitals) collaboratively train a shared model without
sharing raw data. Each client trains locally on its data, sending only model updates (e.g.,
gradients) to a central server, which aggregates them (e.g., using FedAvg) to refine the
global model.
Key aspects include:
• Decentralized Data: Data remains local, ensuring privacy and security.
• Model Updates: Only weights or gradients are shared, not raw data.
• Central Server: Coordinates training and aggregates updates.
• Applications: Used in mobile devices (e.g., Google Gboard for next-word
prediction), healthcare (e.g., mammogram analysis), and IoT.
• Challenges: Include communication costs, data heterogeneity, and security risks
(e.g., potential data leakage from gradients).
Example:
Multiple hospitals collaborate on a mammogram classification model. Each hospital trains
the model locally on its patient data, sends updates to a central server, which aggregates
them using FedAvg to create a global model, improving predictions while keeping data
private.
Diagram:
A federated learning diagram includes:
• Central Model: A global model initialized on a central server.
• Local Training: The model is sent to devices, trained locally, and updates
(encrypted deltas) are sent back.
• Secure Aggregation: Updates from devices are combined anonymously using zero-
sum masking.
• Differential Privacy: Noise is added to obscure rare data, limiting single-device
contributions.
• Final Update: The improved global model is redistributed to devices after iterations.
Source: Google: Federated Learning, IBM: Federated Learning
8. Ensembling Techniques in Machine Learning
Explanation:
Ensembling techniques combine predictions from multiple models to improve accuracy,
robustness, and generalization. These methods reduce bias and variance, making them
effective for complex datasets.
Key methods include:
• Bagging: Trains multiple models on different data subsets (e.g., Random Forests),
combining predictions via averaging or voting.
• Boosting: Trains models sequentially, each correcting previous errors (e.g.,
XGBoost).
• Stacking: Uses a meta-learner to combine predictions from base models.
• Voting: Combines predictions via majority vote (classification) or averaging
(regression).
Example:
Three models predict a digit from an image: Model 1 (0.8 confidence for "7"), Model 2 (0.9
for "7"), Model 3 (0.7 for "9"). Using majority voting, the final prediction is "7" (two models
agree), improving accuracy by leveraging diverse model strengths.
Diagram:
An ensembling diagram includes:
• Bagging: Multiple models trained on bootstrapped datasets, predictions aggregated
via voting or averaging.
• Boosting: Models trained sequentially, each focusing on previous errors, with
weighted predictions combined.
• Stacking: Base models’ predictions fed into a meta-learner for final prediction.
Source: Wikipedia: Ensemble Learning, GeeksforGeeks: Ensemble Learning
Key Citations
• Ultimate Guide to Data Slicing in Big Data Analytics
• Slice-based Learning by Snorkel
• Distributed Parallel Training: Data and Model Parallelism
• Google Switch Transformers Paper
• Machine Learning Model Serving Architectures
• Distributed Machine Learning Patterns
• Distributed Machine Learning Training Part 1
• Paradigms of Parallelism in Colossal-AI
• Data Parallelism vs. Model Parallelism Comparison
• Model Parallelism in Deep Learning
• Federated Learning by Google
• What is Federated Learning by IBM