SMALL LANGUAGE

MODELS
A Complete Course from Idea to Production

Training • Fine-Tuning • Distillation • Domain Adaptation • Agentic Systems

Python-First Practical Guide | 2025 Edition

Covers Topics
Architecture Transformer internals, RoPE, GQA, SwiGLU, RMSNorm
Training Pre-training loop, mixed precision, distributed training
Fine-Tuning LoRA, QLoRA, instruction tuning, DPO
Domain Models Continued pre-training, domain tokenisers, custom vocab
Distillation Offline, online KL, structural pruning, DistilBERT-style
Optimisation GGUF, GPTQ, vLLM, speculative decoding, structured output
Agents ReAct, tools, multi-agent, long-term memory
Production Docker, monitoring, evaluation, benchmarking
Module 1 — What Are Small Language Models?

1.1 Defining 'Small'

There is no single universal threshold separating a 'small' language model from a 'large' one, but
the community has converged on a practical definition: an SLM typically has fewer than 10 billion
parameters, runs comfortably on consumer hardware (a single GPU or even a modern CPU), and
fits entirely in RAM during inference.

Size Class Parameter Range Typical Hardware

Nano < 100 M CPU, Raspberry Pi 5
Micro 100 M – 500 M CPU or entry GPU
Small 500 M – 3 B Single consumer GPU (8 GB VRAM)
Medium-Small 3B–7B Single high-end GPU (16–24 GB)
Medium 7 B – 13 B One or two GPUs
Large > 13 B Multi-GPU cluster

1.2 Why SLMs? The Autonomy Argument

Large cloud-based models are powerful but create dependencies: you pay per token, send data to
a third party, face rate limits, and cannot customise weights. SLMs running locally or on private
infrastructure give you full data privacy, zero marginal inference cost, custom fine-tuning,
deterministic versioned behaviour, and the ability to build agents that run autonomously without API
budget constraints.

1.3 The SLM Landscape in 2025

Model Params Notable Strengths

Llama 3.2 (Meta) 1B / 3B Excellent base for fine-tuning, permissive license
Phi-3 / Phi-4 (Microsoft) 3.8B / 14B State-of-the-art reasoning per parameter
Gemma 2 (Google) 2B / 9B Strong code and instruction following
Qwen 2.5 (Alibaba) 0.5B – 7B Multilingual, strong math and reasoning
Mistral 7B / Mistral Nemo 7B / 12B Fast inference, Apache 2.0 license
SmolLM2 (HuggingFace) 135M – 1.7B Designed for on-device deployment
TinyLlama 1.1B Trained on 3T tokens, great for experimentation

1.4 Three Approaches to Building Your SLM

• Train from scratch — build every weight yourself. Best for full control; requires significant data
and compute.
• Fine-tune a foundation model — start from a pre-trained checkpoint, adapt it to your domain. The
practical sweet spot for most use-cases.
• Distil from a larger model — compress a big teacher model into a small student. Best ratio of
performance to size.
Module 2 — Mathematical & Architectural Foundations

2.1 The Transformer Architecture

Every modern language model is built on the Transformer architecture (Vaswani et al., 2017). The
key insight is the attention mechanism: instead of processing tokens sequentially, attention allows
every token to 'look at' every other token simultaneously. This enables massive parallelism during
training and rich contextual representations.

Core Transformer Components

Embedding Layer → Positional Encoding → N × (Multi-Head Self-Attention + Feed-Forward
Network + LayerNorm + Residual) → Output Projection → Softmax over Vocabulary

2.2 Scaled Dot-Product Attention

Given query Q, key K, and value V matrices (all linear projections of the input), attention is
computed as:

Attention(Q, K, V) = softmax( Q · Kᵀ / √dₖ ) · V

Where:
dₖ = dimension of keys (√dₖ scaling prevents gradient vanishing)

MultiHead(Q,K,V) = Concat(head₁,...,headₕ) · Wₒ
headᵢ = Attention(Q·Wᵢq, K·Wᵢk, V·Wᵢv)

2.3 Key Architectural Choices in Modern SLMs

Choice Modern Best Practice & Why

Positional Encoding Rotary Position Embedding (RoPE) — encodes relative positions,
extrapolates to longer sequences
Normalisation RMSNorm (pre-norm) — faster and more stable than LayerNorm
Activation SwiGLU in FFN — better empirical performance than ReLU/GELU
Attention Mechanism Grouped Query Attention (GQA) — fewer KV heads, faster
inference, lower memory
Tokenisation BPE (Byte-Pair Encoding) or SentencePiece — sub-word
tokenisation
Tied Embeddings Input embedding weights shared with output projection — reduces
 parameter count

2.4 SwiGLU Feed-Forward Network

FFN(x) = W₂ · (SiLU(W₁ · x) ⊗ W₃ · x)

# Typical dimension ratios:

# Standard: d_ffn = 4 × d_model
# Llama-style: d_ffn = 8/3 × d_model (saves ~25% params vs standard)

2.5 Chinchilla Scaling Laws

Optimal compute requires balancing model size (N) and training tokens (D). The key result: for a
fixed compute budget, you should train a smaller model on more tokens rather than a larger model
on fewer tokens.

Optimal D ≈ 20 × N

Examples:
1B param model → ~20B tokens (minimum)
3B param model → ~60B tokens
7B param model → ~140B tokens

Note: Llama 3.2 (1B) was trained on ~9T tokens ('over-trained')

which makes it much more efficient at inference time.
Module 3 — Building an SLM from Scratch in Python

3.1 Environment Setup

python -m venv slm_env && source slm_env/bin/activate

# Core ML stack
pip install torch --index-url [Link]
pip install transformers datasets tokenizers accelerate
pip install einops sentencepiece tqdm wandb bitsandbytes peft trl

3.2 Training a BPE Tokeniser

from tokenizers import Tokenizer
from [Link] import BPE
from [Link] import BpeTrainer
from tokenizers.pre_tokenizers import ByteLevel

def train_tokenizer(files, vocab_size=32000):

tok = Tokenizer(BPE(unk_token='<|unk|>'))
tok.pre_tokenizer = ByteLevel(add_prefix_space=False)
trainer = BpeTrainer(
vocab_size=vocab_size,
special_tokens=['<|pad|>','<|bos|>','<|eos|>','<|unk|>'],
min_frequency=2,
)
[Link](files, trainer)
[Link]('[Link]')
return tok

3.3 Model Configuration Dataclass

from dataclasses import dataclass

@dataclass
class SLMConfig:
vocab_size: int = 32000
d_model: int = 512 # embedding dimension
n_layers: int = 8 # transformer blocks
n_heads: int = 8 # query heads
n_kv_heads: int = 4 # key/value heads (GQA)
d_ffn: int = 1365 # 8/3 * d_model (SwiGLU)
max_seq_len: int = 2048
norm_eps: float = 1e-5
rope_base: float = 10000.0
 # 120M model: SLMConfig(d_model=512, n_layers=8)
# 400M model: SLMConfig(d_model=1024, n_layers=16)
# 1.1B model: SLMConfig(d_model=2048, n_layers=22)

3.4 RMSNorm, RoPE, and Grouped Query Attention

import torch, [Link] as nn, [Link] as F

class RMSNorm([Link]):
def __init__(self, dim, eps=1e-5):
super().__init__()
[Link] = [Link]([Link](dim))
[Link] = eps
def forward(self, x):
return x / [Link](2).mean(-1,keepdim=True).add([Link]).sqrt() *
[Link]

def precompute_rope(dim, max_len, base=10000.0):

theta = 1.0 / (base ** ([Link](0, dim, 2).float() / dim))
t = [Link](max_len)
freqs = [Link](torch.ones_like([Link](t, theta)), [Link](t,
theta))
return freqs # complex64, shape [max_len, dim//2]

def apply_rope(x, freqs):

x_c = torch.view_as_complex([Link]().reshape(*[Link][:-1], -1, 2))
return torch.view_as_real(x_c * freqs[None, :[Link][1], None]).flatten(-
2).type_as(x)

class GQAttention([Link]):
def __init__(self, cfg):
super().__init__()
self.h, self.kv_h = cfg.n_heads, cfg.n_kv_heads
self.d = cfg.d_model // cfg.n_heads
self.n_rep = self.h // self.kv_h
[Link] = [Link](cfg.d_model, self.h * self.d, bias=False)
[Link] = [Link](cfg.d_model, self.kv_h * self.d, bias=False)
[Link] = [Link](cfg.d_model, self.kv_h * self.d, bias=False)
[Link] = [Link](self.h * self.d, cfg.d_model, bias=False)

def forward(self, x, freqs):

B,T,_ = [Link]
q = apply_rope([Link](x).view(B,T,self.h, self.d), freqs)
k = apply_rope([Link](x).view(B,T,self.kv_h,self.d), freqs)
v = [Link](x).view(B,T,self.kv_h,self.d)
k = k.repeat_interleave(self.n_rep,dim=2)
v = v.repeat_interleave(self.n_rep,dim=2)
q,k,v = [[Link](1,2) for t in (q,k,v)]
out = F.scaled_dot_product_attention(q,k,v,is_causal=True)
return [Link]([Link](1,2).reshape(B,T,-1))
3.5 Full SLM Model with Generation
class SwiGLU([Link]):
def __init__(self, cfg):
super().__init__()
self.w1 = [Link](cfg.d_model, cfg.d_ffn, bias=False)
self.w2 = [Link](cfg.d_ffn, cfg.d_model, bias=False)
self.w3 = [Link](cfg.d_model, cfg.d_ffn, bias=False)
def forward(self, x):
return self.w2([Link](self.w1(x)) * self.w3(x))

class Block([Link]):
def __init__(self, cfg):
super().__init__()
[Link] = GQAttention(cfg)
[Link] = SwiGLU(cfg)
self.norm1 = RMSNorm(cfg.d_model, cfg.norm_eps)
self.norm2 = RMSNorm(cfg.d_model, cfg.norm_eps)
def forward(self, x, freqs):
x = x + [Link](self.norm1(x), freqs)
x = x + [Link](self.norm2(x))
return x

class TinySLM([Link]):
def __init__(self, cfg):
super().__init__()
[Link] = [Link](cfg.vocab_size, cfg.d_model)
[Link] = [Link]([Block(cfg) for _ in range(cfg.n_layers)])
[Link] = RMSNorm(cfg.d_model)
self.lm_head = [Link](cfg.d_model, cfg.vocab_size, bias=False)
self.lm_head.weight = [Link] # weight tying
self.register_buffer('freqs', precompute_rope(
cfg.d_model // cfg.n_heads, cfg.max_seq_len, cfg.rope_base))

def forward(self, ids, targets=None):

x = [Link](ids)
for layer in [Link]:
x = layer(x, [Link][:[Link](1)])
logits = self.lm_head([Link](x))
loss = F.cross_entropy([Link](-1,[Link](-1)), [Link](-1),
ignore_index=-100) if targets is not None else None
return logits, loss

@torch.inference_mode()
def generate(self, ids, max_new=200, temp=0.8, top_p=0.9):
for _ in range(max_new):
logits,_ = self(ids[:,-2048:])
probs = [Link](logits[:,-1]/temp, -1)
sp,si = [Link](descending=True)
mask = [Link](-1) - sp > top_p
sp[mask] = 0; sp /= [Link]()
ids = [Link]([ids, [Link](-1, [Link](sp,1))], -1)
return ids
Module 4 — Pre-Training on Custom Corpora

4.1 Data Collection & Quality Filtering

The quality of pre-training data determines the ceiling of your model's capabilities. For a domain-
specific model, collect text representative of the tasks you want the model to perform.

Data Quality Mantra

10 billion tokens of clean, relevant data beats 100 billion tokens of noisy web text for a domain-
specific SLM. Invest 80% of your pre-training effort in data curation.

Domain Data Sources

• Technical/Code: GitHub (permissive licences), StackOverflow, docs, arXiv
• Medical: PubMed, [Link], drug label databases
• Legal: CourtListener, EUR-Lex, public legislative databases
• Finance: SEC EDGAR, financial news, annual reports
• General: CommonCrawl (filtered), Wikipedia, Project Gutenberg

Cleaning & Deduplication Pipeline

import re, unicodedata
from datasketch import MinHash, MinHashLSH # pip install datasketch

def clean_text(text):
text = [Link]('NFKC', text)
text = [Link](r'<[^>]+>', ' ', text) # strip HTML
text = [Link](r'\s+', ' ', text).strip()
return text

def quality_filter(text, min_chars=200, max_chars=100_000):

if not (min_chars <= len(text) <= max_chars): return False
num_ratio = sum([Link]() for c in text) / len(text)
if num_ratio > 0.30: return False # skip tables/logs
return True

# MinHash deduplication — removes near-duplicate documents

def dedup_with_minhash(documents, threshold=0.8, num_perm=128):
lsh = MinHashLSH(threshold=threshold, num_perm=num_perm)
seen, unique = set(), []
for i, doc in enumerate(documents):
m = MinHash(num_perm=num_perm)
for word in [Link]().split(): [Link]([Link]())
result = [Link](m)
if not result: # no near-duplicate found
 [Link](str(i), m)
[Link](doc)
return unique

4.2 The Training Loop with Mixed Precision

import torch, math
from [Link] import GradScaler, autocast
from itertools import cycle

def cosine_lr(step, warmup, total, min_lr, max_lr):

if step < warmup:
return max_lr * step / warmup
t = (step - warmup) / (total - warmup)
return min_lr + 0.5 * (max_lr - min_lr) * (1 + [Link]([Link] * t))

def train(model, loader, device='cuda', total_steps=100_000,

max_lr=3e-4, min_lr=3e-5, warmup=2000, grad_clip=1.0):
[Link](device)
opt = [Link]([Link](), lr=max_lr, weight_decay=0.1,
betas=(0.9, 0.95))
scaler = GradScaler()
for step, (x, y) in enumerate(cycle(loader)):
if step >= total_steps: break
for g in opt.param_groups: g['lr'] = cosine_lr(step, warmup, total_steps,
min_lr, max_lr)
x, y = [Link](device), [Link](device)
with autocast(dtype=torch.bfloat16):
_, loss = model(x, y)
[Link](loss).backward()
scaler.unscale_(opt)
[Link].clip_grad_norm_([Link](), grad_clip)
[Link](opt); [Link](); opt.zero_grad(set_to_none=True)
if step % 500 == 0:
print(f'step {step:6d} | loss {loss:.4f} | lr {opt.param_groups[0]
["lr"]:.2e}')

4.3 Multi-GPU Training with Accelerate

# One-time setup: accelerate config
# Launch: accelerate launch [Link]

from accelerate import Accelerator

accel = Accelerator(mixed_precision='bf16')
model, opt, loader = [Link](model, opt, loader)
# Replace [Link]() with:
[Link](loss)
# Checkpoint:
accel.save_state('checkpoints/step_1000')
Module 5 — Fine-Tuning & Instruction Tuning

5.1 Fine-Tuning vs Training from Scratch

Approach Data Needed Compute Cost

Pre-train from scratch Billions of tokens Very High (weeks, multi-GPU)
Full fine-tuning Millions of examples High (hours to days)
LoRA / QLoRA fine-tuning Hundreds to thousands of Low (minutes to hours, 1 GPU)
examples
Prompt engineering Zero training examples None (inference only)

5.2 Instruction Tuning — ChatML Format

Instruction tuning transforms a base model into a helpful assistant. The standard ChatML format
structures conversations as:

<|im_start|>system
You are a helpful assistant specialised in financial analysis.<|im_end|>
<|im_start|>user
Calculate the P/E ratio for a company with share price $45 and EPS $3.<|im_end|>
<|im_start|>assistant
The P/E ratio = Share Price / EPS = $45 / $3 = 15.0<|im_end|>

5.3 LoRA — Low-Rank Adaptation

LoRA Key Idea
For weight matrix W (shape d×k), add: W' = W + (α/r)×A×B where A is d×r, B is r×k, and r ≪
min(d,k). Only A and B are trained (~0.1–1% of parameters). r (rank) controls capacity —
typically 4–64.

from peft import LoraConfig, get_peft_model, TaskType

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
'meta-llama/Llama-3.2-1B', torch_dtype=torch.bfloat16
)

lora_cfg = LoraConfig(
task_type = TaskType.CAUSAL_LM,
 r = 16,
lora_alpha = 32,
lora_dropout = 0.05,
target_modules = ['q_proj','k_proj','v_proj','o_proj',
'gate_proj','up_proj','down_proj'],
bias = 'none',
)
model = get_peft_model(model, lora_cfg)
model.print_trainable_parameters()
# trainable params: 6.3M || all params: 1.24B || trainable%: 0.51%

5.4 QLoRA — Train a 7B Model on a Consumer GPU

from transformers import BitsAndBytesConfig

bnb_cfg = BitsAndBytesConfig(
load_in_4bit = True,
bnb_4bit_quant_type = 'nf4',
bnb_4bit_compute_dtype = torch.bfloat16,
bnb_4bit_use_double_quant = True,
)
model = AutoModelForCausalLM.from_pretrained(
model_id, quantization_config=bnb_cfg, device_map='auto'
)
# Then apply LoRA exactly as above — now runs on ~6 GB VRAM

5.5 SFTTrainer (TRL) — Full Fine-Tuning Pipeline

from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

dataset = load_dataset('json',
data_files={'train':'[Link]','test':'[Link]'})

trainer = SFTTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset['train'],
eval_dataset = dataset['test'],
peft_config = lora_cfg,
args = SFTConfig(
output_dir = './output',
num_train_epochs = 3,
per_device_train_batch_size = 4,
gradient_accumulation_steps = 4,
learning_rate = 2e-4,
lr_scheduler_type = 'cosine',
warmup_ratio = 0.05,
bf16 = True,
report_to = 'wandb',
 ),
dataset_text_field = 'text',
max_seq_length = 2048,
)
[Link]()

# Merge LoRA adapters back into base model

from peft import AutoPeftModelForCausalLM
merged = AutoPeftModelForCausalLM.from_pretrained('./output')
merged.merge_and_unload().save_pretrained('./merged_model')

5.6 DPO — Direct Preference Optimisation

DPO (Rafailov et al., 2023) trains the model to prefer 'chosen' responses over 'rejected' ones using
paired comparison data, without a separate reward model. This aligns the model with human
preferences efficiently.

from trl import DPOTrainer, DPOConfig

# Dataset format: {prompt, chosen, rejected}

dpo_dataset = load_dataset('json', data_files='dpo_data.jsonl')['train']

dpo_trainer = DPOTrainer(
model = model,
ref_model = None, # None = use frozen copy of initial model
tokenizer = tokenizer,
train_dataset = dpo_dataset,
args = DPOConfig(
beta = 0.1, # KL penalty (higher = closer to reference)
learning_rate = 5e-7,
num_train_epochs = 1,
per_device_train_batch_size = 2,
),
)
dpo_trainer.train()
Module 6 — Domain-Specific Language Models

6.1 The Domain Adaptation Spectrum

Strategy Description & When to Use

RAG (Retrieval-Augmented Inject domain knowledge at inference via vector search. No training.
Generation) Best for factual, document-heavy domains.
Prompt Engineering + Few- Give domain examples in the prompt. Zero training. Good for
shot structured tasks.
Instruction Fine-tuning Fine-tune on (instruction, response) pairs from your domain. Best for
task-specific behaviour.
Continued Pre-training Train on a large unlabelled domain corpus before fine-tuning. Best
when domain vocabulary is different (medicine, law, code).
Pre-train from scratch Full control. Maximum cost. Only when domain is truly unique.

6.2 Continued Pre-Training

Continued pre-training takes an existing checkpoint and continues the next-token prediction
objective on your domain corpus. Use a much smaller learning rate (1e-5) than original pre-training,
and mix in 10–20% general data to prevent catastrophic forgetting.

from datasets import interleave_datasets, load_dataset

# Mix domain + general data to prevent forgetting

domain_ds = load_dataset('text', data_files='domain/*.txt')['train']
general_ds = load_dataset('text', data_files='general/*.txt')['train']

mixed = interleave_datasets(
[domain_ds, general_ds],
probabilities=[0.85, 0.15], # 85% domain, 15% general
seed=42,
)

# Very small learning rate to avoid forgetting

training_args = SFTConfig(
learning_rate = 1e-5, # ~20× lower than instruction FT
warmup_ratio = 0.01,
weight_decay = 0.1,
)
6.3 Custom Vocabulary Extension
For highly specialised domains, adding domain tokens to the vocabulary improves tokenisation
efficiency. A chemistry model might tokenise 'SMILES' notation as single tokens instead of byte-pair
fragments.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained('Qwen/Qwen2.5-1.5B')
new_tokens = ['[GENE]', '[PROTEIN]', 'CRISPR', 'mRNA', 'siRNA']
added = tok.add_tokens(new_tokens)
print(f'Added {added} tokens. New vocab: {len(tok)}')

model.resize_token_embeddings(len(tok))
# New embeddings are random — continued pre-training will learn them
tok.save_pretrained('./domain_tokenizer')

6.4 Domain SLM End-to-End Recipe

1. Collect domain corpus (target: 1B–10B tokens for a 1–3B model).
2. Clean, deduplicate, and quality-filter.
3. Optionally extend tokeniser with domain vocabulary.
4. Choose a foundation model close to your domain (CodeLlama for code, BioMedLM for biology).
5. Continued pre-training (1–3 epochs, LR=1e-5, 85% domain + 15% general mix).
6. Collect instruction/task data (500–50,000 high-quality examples).
7. LoRA instruction fine-tuning (3–5 epochs, LR=2e-4).
8. Optional: DPO alignment on preference data.
9. Evaluate against domain benchmarks.
10. Quantise to 4-bit and serve with vLLM or [Link].
Module 7 — Model Distillation

7.1 Knowledge Distillation — The Core Idea

Knowledge distillation (Hinton et al., 2015) transfers knowledge from a large 'teacher' model to a
small 'student'. The student learns from the teacher's full probability distribution over the vocabulary
— not just the correct answer. This 'soft' distribution encodes richer relational knowledge about
similar concepts.

Dark Knowledge
When a teacher assigns small-but-nonzero probability to wrong answers, it reveals relationships
between concepts. 'cat' and 'kitten' have correlated probabilities when completing sentences
about fluffy animals. Training the student on these distributions is far more informative than hard
labels alone.

7.2 Types of Distillation

Type Description
Offline Response Distillation Use teacher to generate training data, then fine-tune student.
Simplest approach.
Online KL Divergence Teacher and student run simultaneously; student minimises KL
divergence from teacher's live output distributions.
Feature-based Student also learns to match teacher's intermediate hidden state
activations.
Attention-based Student matches teacher's attention patterns layer by layer.
Structural Pruning Create student by removing layers/heads from teacher, then distil.

7.3 Offline Response Distillation (Practical)

The most widely used approach: use GPT-4, Claude, or a large local model to generate high-quality
training examples, then fine-tune your SLM on that data. This is how many of the best-performing
open SLMs are built.

# Step 1: Generate teacher outputs

import anthropic
client = [Link]()

def get_teacher_response(instruction, system=''):

msg = [Link](
 model='claude-opus-4-5-20251101',
max_tokens=1024,
system=system,
messages=[{'role':'user','content':instruction}]
)
return [Link][0].text

# Generate dataset from seed instructions

import json
distilled = []
with open('seed_instructions.jsonl') as f:
for line in f:
item = [Link](line)
resp = get_teacher_response(item['instruction'], [Link]('system',''))
[Link]({'instruction':item['instruction'], 'response':resp})

# Step 2: Fine-tune student SLM on distilled data (use SFTTrainer from Module 5)

7.4 Online KL Divergence Distillation

import [Link] as F

def distillation_loss(s_logits, t_logits, hard_labels, T=4.0, alpha=0.7):

'''
s_logits: student output [B, T, V]
t_logits: teacher output [B, T, V]
hard_labels: ground truth token ids [B, T]
T: temperature (higher = softer distribution)
alpha: weight for soft KL loss
'''
soft_loss = F.kl_div(
F.log_softmax(s_logits / T, dim=-1),
[Link](t_logits / T, dim=-1),
reduction='batchmean'
) * T**2

hard_loss = F.cross_entropy(
s_logits.view(-1, s_logits.size(-1)),
hard_labels.view(-1), ignore_index=-100
)
return alpha * soft_loss + (1 - alpha) * hard_loss

# Training step:
with torch.no_grad():
t_logits, _ = teacher(x) # teacher frozen
s_logits, _ = student(x)
loss = distillation_loss(s_logits, t_logits, y)
7.5 Structural Distillation — Layer Pruning
def create_student_by_layer_selection(teacher, n_student_layers):
n_teacher = len([Link])
stride = n_teacher // n_student_layers
keep = [i * stride for i in range(n_student_layers)]

# Build student config

cfg = [Link].__class__(**[Link].to_dict())
cfg.num_hidden_layers = n_student_layers
student = type(teacher)(cfg)

# Copy shared weights

[Link].embed_tokens.weight = [Link].embed_tokens.weight
student.lm_head.weight = teacher.lm_head.weight
[Link] = [Link]

# Copy evenly-spaced teacher layers

for i, src in enumerate(keep):
[Link][i].load_state_dict(
[Link][src].state_dict()
)
return student

# Halve a 12-layer model:

student = create_student_by_layer_selection(teacher, n_student_layers=6)
# Then fine-tune student with distillation loss
Module 8 — Evaluation & Benchmarking

8.1 Perplexity
@torch.inference_mode()
def perplexity(model, loader, device='cuda'):
[Link]()
total_loss, total_tok = 0.0, 0
for x, y in loader:
x, y = [Link](device), [Link](device)
_, loss = model(x, y)
n = (y != -100).sum().item()
total_loss += [Link]() * n
total_tok += n
return [Link](total_loss / total_tok)

8.2 LM Evaluation Harness — Standard Benchmarks

# pip install lm-eval

lm_eval \
--model hf \
--model_args pretrained=./my_model,dtype=float16 \
--tasks hellaswag,arc_easy,arc_challenge,winogrande,truthfulqa_mc,gsm8k \
--num_fewshot 5 \
--output_path ./eval_results \
--batch_size auto

8.3 Key Benchmarks Reference

Benchmark What It Measures Target for 1B SLM

HellaSwag Commonsense reasoning ≥ 70%
ARC-Challenge Science multiple choice (hard) ≥ 40%
MMLU Multi-domain knowledge (57 ≥ 45%
subjects)
TruthfulQA Resistance to producing false ≥ 40%
beliefs
HumanEval Python code generation (pass@1) ≥ 25%
GSM8K Grade school math word problems ≥ 25%
MT-Bench Multi-turn instruction quality (1–10) ≥ 5.5
8.4 Custom Domain Evaluation
from transformers import pipeline

def evaluate_domain(model_path, eval_file):

pipe = pipeline('text-generation', model=model_path, device_map='auto')
data = [[Link](l) for l in open(eval_file)]
results = {'correct':0, 'total':len(data)}
for item in data:
out = pipe(item['prompt'], max_new_tokens=256, do_sample=False)[0]
['generated_text']
answer = extract_answer(out) # your domain-specific parser
results['correct'] += ([Link]() == item['expected'].strip())
results['accuracy'] = results['correct'] / results['total']
return results
Module 9 — Optimisation for Production

9.1 Quantisation Overview

Format Memory Footprint & Use Case

FP32 1× baseline — Training only
BF16 / FP16 0.5× — Standard GPU inference
INT8 (GPTQ/AWQ) 0.25× — Production inference, minimal quality loss
INT4 (NF4, GGUF Q4_K_M) 0.125× — Edge/CPU deployment, slight quality loss
INT2 (experimental) 0.0625× — Extreme compression, notable quality loss

9.2 GGUF / [Link] — CPU Inference

# Convert to GGUF and quantise (requires [Link])
python convert_hf_to_gguf.py ./merged_model --outfile [Link] --outtype f16
./llama-quantize [Link] model-Q4_K_M.gguf Q4_K_M

# Python inference
from llama_cpp import Llama # pip install llama-cpp-python
llm = Llama(model_path='model-Q4_K_M.gguf', n_ctx=4096, n_threads=8)
out = llm('The quarterly revenue was', max_tokens=200, stop=['\n'])
print(out['choices'][0]['text'])

9.3 GPTQ Quantisation

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

qcfg = BaseQuantizeConfig(bits=4, group_size=128, desc_act=False)

model = AutoGPTQForCausalLM.from_pretrained('./merged_model', qcfg)

# 128 calibration examples from your domain

examples = [tokenizer(t, return_tensors='pt') for t in calibration_texts[:128]]
[Link](examples)
model.save_quantized('./model_gptq_4bit')

9.4 vLLM — Production Serving (OpenAI-Compatible API)

# pip install vllm
 vllm serve ./merged_model \
--host [Link] --port 8000 \
--dtype bfloat16 --max-model-len 4096 \
--gpu-memory-utilization 0.90

# Query via OpenAI SDK:

from openai import OpenAI
client = OpenAI(base_url='[Link] api_key='local')
resp = [Link](
model='./merged_model',
messages=[{'role':'user','content':'Summarise this contract: ...'}],
max_tokens=512,
)
print([Link][0].[Link])

9.5 Structured Output with Outlines

import outlines # pip install outlines
from pydantic import BaseModel

class Entity(BaseModel):
name: str
type: str
confidence: float

model = [Link]('./merged_model')
generator = [Link](model, Entity)

result = generator('Extract entity from: Apple Inc. posted record revenue.')

print(result) # Entity(name='Apple Inc.', type='ORG', confidence=0.98)

9.6 Speculative Decoding — 2–3× Throughput

# vLLM native support:
vllm serve ./main_model \
--speculative-model ./draft_model \
--num-speculative-tokens 5

# HuggingFace assisted generation:

draft = AutoModelForCausalLM.from_pretrained('./tiny_draft')
output = [Link](
**inputs, assistant_model=draft, do_sample=False
)
Module 10 — SLMs in Agentic AI Systems

10.1 What Is an AI Agent?

An AI agent is a system where a language model acts as a reasoning engine that can use tools,
plan multi-step actions, observe results, and iterate until a goal is achieved. Unlike a chatbot, an
agent can browse the web, execute code, read and write files, call APIs, and manage its own
memory.

The Agent Loop

OBSERVE → THINK → ACT → OBSERVE → ... repeat until goal achieved. The LLM is the
'brain' that decides which tool to call and how to interpret results.

10.2 Why SLMs Enable Autonomous Systems

Agents run many inference calls per task. Cloud API costs accumulate rapidly: a complex task
requiring 50 LLM calls costs $1–5 with GPT-4. A local SLM has zero marginal cost, enabling
unlimited agent iterations, full data privacy, and 24/7 operation without budget concerns.

10.3 ReAct Agent Implementation

import json, re
from typing import Callable

SYSTEM = '''You are an autonomous agent. Use this exact format:

Thought: [reason about what to do]

Action: tool_name
Action Input: {"key": "value"}

After each action you receive:

Observation: [result]

When done, output:

Final Answer: [answer]
'''

class ReActAgent:
def __init__(self, llm_fn, tools: dict[str, Callable], max_steps=15):
[Link], [Link], self.max_steps = llm_fn, tools, max_steps

def run(self, task: str) -> str:

history = [
 {'role':'system', 'content':SYSTEM},
{'role':'user', 'content':task},
]
for _ in range(self.max_steps):
resp = [Link](history)
[Link]({'role':'assistant','content':resp})
if 'Final Answer:' in resp:
return [Link]('Final Answer:')[-1].strip()
a_match = [Link](r'Action: (\w+)', resp)
i_match = [Link](r'Action Input: ({.*?})', resp, [Link])
if a_match and i_match:
tool = a_match.group(1)
inp = [Link](i_match.group(1))
obs = str([Link][tool](**inp)) if tool in [Link] else
'Tool not found'
[Link]({'role':'user','content':f'Observation: {obs}'})
return 'Max steps reached.'

10.4 Tool Library

import subprocess, requests
from pathlib import Path

def run_python(code: str) -> str:

r = [Link](['python3','-c',code], capture_output=True, text=True,
timeout=30)
return [Link] + [Link]

def read_file(path: str) -> str:

return Path(path).read_text(errors='replace')[:5000]

def write_file(path: str, content: str) -> str:

Path(path).write_text(content)
return f'Written {len(content)} chars to {path}'

def search_web(query: str) -> str:

r = [Link]('[Link]
params={'q':query,'format':'json'})
return [Link]().get('AbstractText', 'No results found.')[:2000]

def query_db(sql: str, db='[Link]') -> str:

import sqlite3
conn = [Link](db)
rows = [Link](sql).fetchall()
return str(rows[:50])

TOOLS = {'run_python':run_python, 'read_file':read_file,

'write_file':write_file, 'search_web':search_web, 'query_db':query_db}
10.5 Multi-Agent System — Specialised SLM Team
class MultiAgentSystem:
'''Orchestrator delegates to specialised SLM workers.'''
def __init__(self):
[Link] = ReActAgent(llm_fn=llm_3b, tools={
'delegate_coder': self.ask_coder,
'delegate_analyst': self.ask_analyst,
'delegate_writer': self.ask_writer,
})
[Link] = ReActAgent(llm_fn=llm_codellama_1b,
tools={'run_python':run_python})
[Link] = ReActAgent(llm_fn=llm_finance_1b,
tools={'query_db':query_db})
[Link] = ReActAgent(llm_fn=llm_writer_1b,
tools={'write_file':write_file})

def ask_coder(self, task: str) -> str: return [Link](task)

def ask_analyst(self, task: str) -> str: return [Link](task)
def ask_writer(self, task: str) -> str: return [Link](task)
def run(self, task: str) -> str: return [Link](task)

# Example:
system = MultiAgentSystem()
result = [Link](
'Analyse Q3 sales data, plot the top 5 products, write an executive summary.'
)

10.6 Persistent Vector Memory

import chromadb # pip install chromadb
from sentence_transformers import SentenceTransformer # pip install sentence-
transformers

class AgentMemory:
def __init__(self, name='memory'):
[Link] = [Link]('./memory_store')
[Link] = [Link].get_or_create_collection(name)
[Link] = SentenceTransformer('all-MiniLM-L6-v2') # 22M params

def store(self, text: str, meta: dict = None):

emb = [Link]([text])[0].tolist()
[Link](embeddings=[emb], documents=[text],
metadatas=[meta or {}], ids=[str(hash(text))])

def recall(self, query: str, top_k=5) -> list[str]:

emb = [Link]([query])[0].tolist()
return [Link](query_embeddings=[emb], n_results=top_k)
['documents'][0]

# Inject relevant past context into agent:

def build_augmented_prompt(task, memory):
past = [Link](task)
ctx = '\n'.join(f'- {m}' for m in past)
return f'Relevant context from past tasks:\n{ctx}\n\nCurrent task: {task}'
Module 11 — End-to-End Project Walkthrough
This module walks through a complete real-world project: a domain-specific financial analysis SLM
that runs locally, answers questions about earnings reports, and operates as an autonomous agent.

11.1 Project Specification

Aspect Details
Goal Local SLM answering questions about financial documents and
computing ratios
Base Model Qwen2.5-1.5B (strong numerical reasoning)
Domain Financial analysis — earnings, SEC filings, ratios
Deployment Local CPU server, quantised Q4_K_M GGUF
Target Performance <500ms first token on M2 Mac, handles 5 concurrent users

11.2 Project Structure

finance_slm/
├── data/
│ ├── raw/ # SEC EDGAR filings, earnings transcripts
│ ├── cleaned/ # after cleaning pipeline
│ └── instruct/ # (instruction, response) pairs .jsonl
├── src/
│ ├── data_pipeline.py # collection, cleaning, deduplication
│ ├── [Link] # LoRA + SFTTrainer
│ ├── [Link] # offline distillation from GPT-4
│ ├── [Link] # domain benchmarks
│ └── [Link] # financial ReAct agent
├── configs/
│ └── [Link]
├── [Link] # vLLM or [Link] server
└── [Link] # interactive CLI demo

11.3 Financial Domain Tools for the Agent

import yfinance as yf # pip install yfinance
import pandas as pd

def get_financials(ticker: str) -> str:

 stock = [Link](ticker)
info = [Link]
return [Link]({
'market_cap': [Link]('marketCap'),
'pe_ratio': [Link]('trailingPE'),
'revenue': [Link]('totalRevenue'),
'net_income': [Link]('netIncomeToCommon'),
'debt': [Link]('totalDebt'),
})

def calculate_ratios(ticker: str) -> str:

stock = [Link](ticker)
bs = stock.balance_sheet
income = stock.income_stmt
ratios = {
'current_ratio': float([Link]['Current Assets'][0]) /
float([Link]['Current Liabilities'][0]),
'gross_margin': float([Link]['Gross Profit'][0]) /
float([Link]['Total Revenue'][0]),
'debt_to_equity': float([Link]['Total Debt'][0]) /
float([Link]['Stockholders Equity'][0]),
}
return [Link]({k: round(v,4) for k,v in [Link]()})

def search_sec_filings(company: str, form_type: str = '10-K') -> str:

# Use SEC EDGAR full-text search API
url = f'[Link]
q="{company}"&dateRange=custom&startdt=2023-01-01&forms={form_type}'
r = [Link](url, headers={'User-Agent': 'my-app admin@[Link]'})
hits = [Link]().get('hits', {}).get('hits', [])[:3]
return [Link]([[Link]('_source', {}).get('display_date_filed') for h in
hits])

FINANCE_TOOLS = {
'get_financials': get_financials,
'calculate_ratios': calculate_ratios,
'search_sec_filings': search_sec_filings,
'run_python': run_python,
}

11.4 The Complete Agent System Prompt

FINANCE_SYSTEM = '''
You are a financial analyst AI with access to real-time financial data.
Tools available:
- get_financials: Get market cap, P/E, revenue, net income for a ticker
- calculate_ratios: Compute current ratio, gross margin, D/E ratio
- search_sec_filings: Find recent SEC filings for a company
- run_python: Execute Python for custom calculations or data analysis

Always show your reasoning. Be precise with numbers.

Format monetary values as $[Link] (billions) or $[Link] (millions).
'''
11.5 Deployment Config
# Quantise for CPU deployment
python convert_hf_to_gguf.py ./merged_model --outfile finance_slm.gguf
./llama-quantize finance_slm.gguf finance_slm_Q4_K_M.gguf Q4_K_M

# Serve with [Link] server (OpenAI-compatible)

./llama-server \
-m finance_slm_Q4_K_M.gguf \
--host [Link] --port 8080 \
-c 4096 --threads 8 \
--parallel 4 # 4 concurrent users

# Or with vLLM (GPU):

vllm serve ./merged_model --port 8000 --dtype bfloat16
Appendix — Tools, Libraries & Resources

Core Python Libraries

Library Purpose Install

PyTorch Core deep learning framework pip install torch
transformers Pre-trained model hub and training pip install transformers
utilities
datasets Efficient dataset loading and pip install datasets
processing
tokenizers Fast BPE/WordPiece tokeniser pip install tokenizers
training
accelerate Multi-GPU training abstraction pip install accelerate
peft LoRA, QLoRA, and other efficient pip install peft
fine-tuning
trl SFTTrainer, DPOTrainer, PPO for pip install trl
alignment
bitsandbytes 4-bit and 8-bit quantisation for pip install bitsandbytes
training
auto-gptq GPTQ post-training quantisation pip install auto-gptq
vllm High-throughput LLM serving pip install vllm
llama-cpp-python CPU inference with GGUF pip install llama-cpp-python
quantisation
outlines Structured/JSON-mode pip install outlines
constrained generation
lm-eval LLM evaluation harness (60+ pip install lm-eval
benchmarks)
wandb Training monitoring and experiment pip install wandb
tracking
sentence-transformers Embedding models for vector pip install sentence-transformers
memory
chromadb Local vector database for agent pip install chromadb
memory

Pre-Trained Model Checkpoints to Start From

HuggingFace Model ID Best Starting Point For
meta-llama/Llama-3.2-1B General purpose 1B, excellent base
meta-llama/Llama-3.2-3B General purpose 3B, best small model
Qwen/Qwen2.5-1.5B Math, reasoning, and multilingual
microsoft/Phi-3.5-mini-instruct Reasoning, already instruction-tuned
google/gemma-2-2b Code and instruction following
HuggingFaceTB/SmolLM2- On-device and edge deployment
1.7B
codellama/CodeLlama-7b-hf Code specialisation, C++/Python
mistralai/Mistral-7B-v0.3 General purpose 7B, Apache 2.0

Recommended Learning Path

11. Complete Andrej Karpathy's 'Neural Networks: Zero to Hero' YouTube series — builds intuition
from first principles.
12. Read 'Attention Is All You Need' (Vaswani et al., 2017) — the original Transformer paper.
13. Read 'Training Compute-Optimal Large Language Models' (Hoffmann et al., 2022) — Chinchilla
scaling laws.
14. Read 'LoRA: Low-Rank Adaptation of Large Language Models' (Hu et al., 2021).
15. Read 'Knowledge Distillation: A Survey' (Gou et al., 2021) — comprehensive distillation
overview.
16. Work through the HuggingFace NLP Course ([Link]/course) — practical transformers.
17. Study the TinyLlama repository — excellent example of a full pre-training pipeline.
18. Experiment with LLM360 — fully open-source training runs with checkpoints.

Hardware Guide

Hardware Recommended Use Notes

M2/M3 MacBook (16–32 Inference and QLoRA fine-tuning [Link] uses Apple Silicon Metal
GB) up to 7B GPU
RTX 3090 / 4090 (24 GB LoRA fine-tuning 7B; full FT 1– Best single-GPU consumer option
VRAM) 3B
2× A100 (80 GB each) Full fine-tuning 13–70B models Standard research setup
Lambda Labs / RunPod Cloud GPU rental for training ~$1–3/hr for A100
runs
Google Colab Pro+ Quick experiments and T4/A100 available
prototyping
CPU server (32+ cores) GGUF inference up to 7B Use for production if no GPU
available

Hyperparameter Quick Reference

Setting Recommended Value / Range

Pre-training LR 1e-3 to 3e-4 (cosine decay to 1/10 of peak)
Continued pre-training LR 1e-5 (10–30× lower than original)
LoRA fine-tuning LR 2e-4 to 5e-4
DPO fine-tuning LR 5e-7 to 5e-6
Batch size (tokens) 0.5M–2M tokens per step
Gradient accumulation Adjust to reach effective batch size
Weight decay 0.1
AdamW betas (0.9, 0.95)
Gradient clipping 1.0
Warmup steps 1–5% of total steps
LoRA rank (r) 4–64 (start with 16)
LoRA alpha 2× rank is common (alpha=32 for r=16)
Distillation temperature (T) 2–6 (higher = softer)
Distillation alpha 0.5–0.9 (weight for soft loss)