Tutorial 8

The way to adapt a large language model (LLM) to a new task is to fine-tune all of
its parameters. This approach, known as full fine-tuning, updates every weight in
the model using task-specific data. While this method is effective and has
historically achieved strong performance, it becomes increasingly impractical as
model sizes grow.

The Core Problem: Scale

Modern LLMs contain billions of parameters. Updating all of these parameters
during training introduces significant computational and memory challenges.

Consider what must be stored during training:

• Model parameters (weights)

• Forward activations (intermediate outputs)
• Gradients (for backpropagation)
• Optimizer states (e.g., momentum, variance for Adam)

These components collectively require far more memory than the model weights
alone.

Limitations of Full Fine-Tuning:

Beyond memory usage, full fine-tuning introduces several additional challenges:

1. High Computational Cost

Training all parameters requires:

• More time
• More energy
• Expensive hardware

2. Poor Scalability Across Tasks

For each new task:

• A separate copy of the model must be fine-tuned and stored

• This quickly becomes inefficient in multi-task settings
3. Redundancy in Learning

Large models are often over-parameterized, meaning:

• Not all parameters need to change to learn a new task

• Many updates are unnecessary

These limitations motivate a more efficient strategy which is Instead of updating

the entire model, we update only a small number of additional or selected
parameters

This leads to the concept of:

Parameter-Efficient Fine-Tuning (PEFT)

Designed to achieve a balance between efficiency and performance, making it a

practical solution for working with modern large-scale models.

1. Reduced Memory and Compute Requirements

By training only a small fraction of the model parameters, PEFT significantly

lowers:

• GPU memory usage

• Computational cost
• Hardware requirements

This makes it possible to fine-tune large models even on limited resources.

2. Faster Training

Since fewer parameters are updated:

• Training becomes much faster

• Experiments can be run more frequently
• Iteration cycles are significantly shorter
3. Scalability and Multi-Task Flexibility

PEFT enables using a single base model for multiple tasks:

• The main model remains frozen

• Only small task-specific components are added

This allows efficient switching between tasks without duplicating the entire model.

4. Strong Performance

Despite updating only a small subset of parameters:

• PEFT methods often achieve performance comparable to full fine-tuning

• In some cases, they even match state-of-the-art results

5. Flexible and Modular Adaptation

PEFT methods are:

• Easy to integrate into existing models

• Modular (e.g., adapters, LoRA, prompt tuning)
• Suitable for a wide range of tasks and domains

PEFT Methods:

1. Adapters

Adapters introduce small neural module inside each Transformer layer.

Instead of modifying the original model weights, we insert additional layers that
can learn task-specific information.

Architecture and Structure:

Each transformer layer is augmented with a small bottleneck network consisting

of:

• Down-projection → reduces dimensionality

• Non-linearity → introduces learning capacity
• Up-projection → restores original dimension
This forms a compact bottleneck module that is easy to train.

Training Strategy

• The original model is frozen completely

• Only the adapter layers are trained

Advantages

• Very few parameters need to be updated

• Easy to switch between tasks (just swap adapters)
• Strong performance despite small size

The main model already contains general knowledge, and adapters simply
specialize it for a specific task.

2. Prompt Tuning

Instead of modifying model weights, we:

Learn a set of continuous prompt embeddings that guide the model

How It Works
• Add a set of learnable vectors (tokens) at the input
• These tokens are trained while the model remains frozen

Advantages
• Extremely lightweight (very few parameters)
• Efficient for multi-task batching
• No need to modify internal model structure

Limitations
• Works well mainly for very large models
• Performance drops for:
o Smaller models
o Complex or reasoning-heavy tasks

Extensions:
Multi-Layer Prompt Tuning

• Instead of adding prompts only at input

➡️ Prompts are injected into multiple transformer layers

This improves performance by influencing deeper representations.

Prefix Tuning

Prefix tuning is a parameter-efficient technique that adapts a pretrained model by

introducing a set of learnable prefix vectors, while keeping all original model
parameters frozen.

How It Works

• A sequence of trainable vectors (prefixes) is added to the model

• These prefixes are treated as if they were part of the input sequence
• The model processes them just like normal tokens
• Only the prefix parameters are updated during training
Key Advantages

• Requires very few trainable parameters

• Keeps the pretrained model completely frozen
• Allows efficient adaptation across multiple tasks
• Enables different tasks to be handled within the same model

3. LoRA (Low-Rank Adaptation)

LoRA is one of the most important and widely used PEFT methods .

Instead of updating the full weight matrix W, we approximate the update using two
smaller matrices:

W+ΔW=W+BA

Where:

• A: low-rank matrix
• B: low-rank matrix
• r≪d (rank is much smaller than original dimension)

Key Benefits

• Massive reduction in trainable parameters

• Original model weights remain unchanged (frozen)
• Highly efficient and scalable

Small rank (r) → more efficient, fewer parameters

Larger rank (r) → better performance

There is a trade-off between efficiency and accuracy

Where LoRA is Applied

LoRA is applied to specific weight matrices inside the Transformer, most

commonly:

• The self-attention layers (e.g., query, key, value projections)

This allows LoRA to efficiently influence the most important parts of the model’s
computation.

Important Properties

1. No Additional Inference Latency

Once training is complete:

• The low-rank updates can be merged with the original weights

• This means there is no extra computational cost during inference

2. Easy Task Switching

• Different tasks can use different LoRA parameters

• The base model remains unchanged
• Switching tasks only requires swapping the small LoRA components

Extensions of LoRA:
Several improvements have been proposed:

• QLoRA → combines LoRA with quantization for memory efficiency

• LoRA+ → uses different learning rates for faster convergence
• DyLoRA → dynamically adjusts rank during training
Intrinsic dimensionality:
The minimum number of dimensions required for a model to reach close to its
optimal performance which helps us understand how much of the model’s capacity
is truly necessary for a given task.

Larger models tend to have lower intrinsic dimensionality

→This means they require fewer effective parameters to adapt to new tasks

As models are pretrained on massive datasets, they become:

o More expressive
o More generalizable
o Easier to adapt with minimal changes

Sparse Subnetworks and Pruning:

It is based on the idea that:

Not all parameters in a large model are equally important.

Instead of adding new parameters (as in PEFT methods), we can improve

efficiency by removing unnecessary parameters and keeping only the most
useful ones.

This approach is known as pruning.

What is Pruning?

Pruning is a technique that:

• Removes less important weights from a neural network

• Creates a smaller and more efficient subnetwork

This is typically done by applying a binary mask over the model parameters:

• Important weights → kept

• Unimportant weights → set to zero
How Pruning Works?

1. Train the full model

2. Remove weights with low importance (commonly based on small
magnitude)
3. Retrain the remaining weights to recover performance

This process can be:

• One-shot pruning → done once

• Iterative pruning → repeated multiple times for better results

Pruning followed by retraining helps maintain model performance while reducing

complexity

Why Pruning Works

Large language models are typically over-parameterized, meaning:

• They contain more parameters than necessary

• Many weights contribute very little to the final performance

By removing these redundant parameters, we can:

• Reduce model size

• Improve efficiency
• Maintain comparable performance

Limitations of Pruning

• Requires careful selection of which weights to remove

• May require multiple retraining steps
• Can introduce additional complexity in implementation

Training LLM at Scale:

As model sizes continue to grow into the billions of parameters, training large
language models (LLMs) becomes increasingly challenging. To address these
challenges, To address these challenges, several techniques have been developed
such as quantization and mixed precision training, both of which aim to reduce
memory usage and improve computational efficiency while maintaining strong
model performance.

1) Quantization:
Quantization aims to reduce the memory of a model by representing its parameters
using fewer bits.

Instead of storing each parameter in full 32-bit precision (FP32), we can use lower-
precision formats such as FP16, BF16, or even INT8.

Format Memory per Parameter

FP32 4 bytes
FP16 2 bytes
BF16 2 bytes
INT8 1 byte

This shows that quantization can reduce memory usage by up to 4×, which is
critical for training and deploying large models.

What is BF16?
BF16 (Brain Floating Point 16) is a truncated version of FP32 that keeps the same range but with
lower precision

How BF16 Works

A floating-point number is represented using:

• Sign bit
• Exponent
• Mantissa (precision)
BF16 keeps:

• Same exponent as FP32 → same numerical range

• Fewer mantissa bits → lower precision

Because BF16 has fewer mantissa bits:

• It cannot distinguish very close values precisely

• Slight loss in numerical precision
• But usually acceptable in deep learning

Quantization Advantages:
Reducing memory usage allows:

• Training larger models on limited hardware

• Faster data movement between memory and compute units
• Lower overall computational cost

2) Mixed Precision Training:

It is a technique that improves efficiency by combining different numerical
precisions during training.
Instead of using full 32-bit precision (FP32) for all computations, it uses:

• Lower precision (BF16 or FP16) for most operations

• Full precision (FP32) for a master copy of the model weights

How It Works:
During training:

• Forward and backward computations are performed using BF16 or FP16

• A separate FP32 master copy of the parameters is maintained
• Parameter updates are applied to the FP32 copy to ensure numerical stability

Mixed Precision Advantages:

This combination allows us to:

• Reduce memory usage significantly (since most tensors use 16-bit

precision)
• Speed up computation on modern hardware
• Maintain training stability by preserving high-precision updates

Multi-GPU Training:
As model sizes continue to grow, a single GPU is often no longer sufficient to
handle the memory and computational requirements of training large language
models. To overcome this limitation, training is distributed across multiple GPUs
using different parallelization strategies.

1. Data Parallelism
The dataset is divided across multiple GPUs, while each GPU holds a complete
copy of the model.
How It Works

• Each GPU processes a different subset of the data

• Forward and backward passes are computed independently
• Gradients from all GPUs are then aggregated and synchronized

Advantages:

• Simple and widely used

• Scales well with larger datasets

[Link] Data Parallel (DDP)

It is an optimized version of data parallelism designed to improve efficiency and scalability.

How It Works

• Each GPU still maintains a full copy of the model

• Gradients are synchronized across GPUs during training
• Communication is optimized to reduce overhead

Advantages

• Faster and more efficient than basic data parallelism

• Minimizes communication bottlenecks
• Widely used in modern deep learning frameworks
3. FSDP / ZeRO (Memory-Efficient Training)
Instead of storing the entire model on every GPU:

Each GPU stores only a portion of the model parameters

How It Works

• Model parameters are sharded (split) across GPUs

• Each GPU holds only part of:
o Parameters
o Gradients
o Optimizer states

Advantages:

• Significantly reduces memory usage per GPU

• Enables training of much larger models

Disadvantages:

• Increased communication overhead between GPUs

• More complex implementation