Generative AI — Complete Study Notes

GENERATIVE AI
Complete Study Notes
Foundations • Architecture • Applications • Ethics • Future Trends

2024 Edition

Page 1 of 22
Generative AI — Complete Study Notes

Unit 1: Introduction to Generative AI

1.1 What is Generative AI?
Generative AI (GenAI) refers to artificial intelligence systems that can generate new content —
text, images, audio, video, code, and more — based on patterns learned from training data.

Key Definition
Generative AI: A class of AI models that learn the underlying distribution of training data
and use it to generate novel, realistic outputs that resemble the training data.
Unlike discriminative models (which classify), generative models CREATE new content.

Types of AI (Taxonomy)
• Narrow AI: Performs specific tasks (chess, image recognition)
• General AI (AGI): Hypothetical human-level intelligence across all domains
• Superintelligence: Hypothetical AI surpassing human intelligence
• Generative AI: Subset of AI that creates new content

1.2 History & Evolution

Year / Era Milestone
1950s–60s Perceptrons, early neural nets (Rosenblatt 1958)
1980s Backpropagation, multi-layer networks
1990s Recurrent Neural Networks (RNNs), LSTMs
2014 GANs introduced by Ian Goodfellow
2017 Transformer architecture — 'Attention Is All You Need'
2018–19 BERT, GPT-1, GPT-2 (OpenAI)
2020 GPT-3 (175B parameters), few-shot learning
2021 DALL-E, Codex, GitHub Copilot
2022 ChatGPT, Stable Diffusion, Midjourney
2023 GPT-4, Gemini, Llama, Claude — multimodal era
2024–25 Multimodal agents, reasoning models, edge deployment

1.3 Discriminative vs Generative Models

Aspect Discriminative Generative
Goal P(Y|X) — classify/predict labels given P(X) or P(X|Y) — model data

Page 2 of 22
Generative AI — Complete Study Notes

input distribution
Output Labels, classes, probabilities New data samples
Examples Logistic regression, SVM, BERT GANs, VAEs, GPT, Diffusion models
(classification)
Use case Spam detection, image classification Image synthesis, text generation,
data augmentation

Page 3 of 22
Generative AI — Complete Study Notes

Unit 2: Foundational Architectures

2.1 Neural Networks Recap
Neural networks are the backbone of modern GenAI. Key concepts:
• Neuron (Perceptron): Weighted sum of inputs + bias, passed through activation
function
• Layers: Input → Hidden → Output
• Activation functions: ReLU, Sigmoid, Tanh, GELU, SiLU
• Backpropagation: Gradient of loss w.r.t. weights via chain rule
• Optimizers: SGD, Adam, AdamW, RMSProp
• Loss functions: Cross-entropy (NLP), MSE (regression), BCE (binary)

2.2 Recurrent Neural Networks (RNNs)

RNNs process sequential data by maintaining a hidden state across time steps.
• Vanilla RNN: suffers from vanishing/exploding gradients
• LSTM (Long Short-Term Memory): gates control information flow — forget, input, output
gates
• GRU (Gated Recurrent Unit): simplified LSTM with reset and update gates
• Limitations: slow (sequential), fixed context window, poor long-range dependencies

LSTM Gates Formula (Key Exam Topic)

Forget gate: ft = σ(Wf·[ht-1, xt] + bf)
Input gate: it = σ(Wi·[ht-1, xt] + bi)
Cell state: Ct = ft ⊙ Ct-1 + it ⊙ tanh(WC·[ht-1, xt] + bC)
Output gate: ot = σ(Wo·[ht-1, xt] + bo)
Hidden state: ht = ot ⊙ tanh(Ct)

2.3 Attention Mechanism

Attention allows models to focus on relevant parts of the input when generating each output
token.

Types of Attention
• Additive (Bahdanau): Alignment score = tanh(W1·encoder + W2·decoder)
• Dot-Product: Score = Q·K^T / √dk
• Self-Attention: Query, Key, Value all come from same sequence
• Multi-Head Attention: Run attention h times in parallel, concatenate outputs
• Cross-Attention: Query from decoder, Key/Value from encoder

Page 4 of 22
Generative AI — Complete Study Notes

Scaled Dot-Product Attention

Attention(Q, K, V) = softmax(QK^T / √dk) · V

Q = Query matrix (what we're looking for)

K = Key matrix (what each position offers)
V = Value matrix (actual content to retrieve)
√dk = scaling factor to prevent softmax saturation

Multi-head: MultiHead(Q,K,V) = Concat(head1,...,headh)·WO

2.4 Transformer Architecture

Introduced in 'Attention Is All You Need' (Vaswani et al., 2017). Revolutionized NLP and GenAI.

Encoder Block (used in BERT-like models)

• Multi-Head Self-Attention
• Add & Layer Normalize
• Feed-Forward Network (FFN): two linear layers with ReLU
• Add & Layer Normalize

Decoder Block (used in GPT-like models)

• Masked Multi-Head Self-Attention (causal — can't attend to future tokens)
• Add & Layer Normalize
• Multi-Head Cross-Attention (encoder-decoder models only)
• Add & Layer Normalize
• Feed-Forward Network
• Add & Layer Normalize

Positional Encoding
• Adds position information since Transformers have no inherent order
• Sinusoidal: PE(pos, 2i) = sin(pos/10000^(2i/d))
• Learned positional embeddings (used in GPT, BERT)
• RoPE (Rotary Position Embedding) — used in LLaMA, GPT-NeoX
• ALiBi (Attention with Linear Biases) — better length generalization

Page 5 of 22
Generative AI — Complete Study Notes

Unit 3: Large Language Models (LLMs)

3.1 LLM Fundamentals
LLMs are transformer-based models trained on massive text corpora to predict the next token
(autoregressive) or fill masked tokens (masked language modeling).
Model Family Architecture Training Objective Example Models
GPT-style Decoder-only Causal LM (next token GPT-2/3/4, LLaMA, Mistral,
prediction) Claude
BERT-style Encoder-only Masked LM + NSP BERT, RoBERTa, ALBERT,
DeBERTa
T5/BART-style Encoder- Seq2seq, denoising T5, BART, mT5, Flan-T5
Decoder

3.2 Tokenization
Converting raw text to numerical tokens for model input.
• Word-level: Each word is a token — large vocabulary, OOV issues
• Character-level: Each character is a token — small vocab, long sequences
• BPE (Byte-Pair Encoding): Merge frequent byte pairs iteratively — used in GPT
models
• WordPiece: Similar to BPE but maximizes likelihood — used in BERT
• SentencePiece: Language-agnostic, handles any script
• Tiktoken: OpenAI's fast BPE tokenizer for GPT-3.5/4

3.3 Training LLMs

Pre-training
• Trained on trillions of tokens from web, books, code
• Objective: minimize cross-entropy loss on next-token prediction
• Requires massive compute: thousands of GPUs/TPUs for weeks/months
• Scaling laws (Chinchilla): optimal tokens ≈ 20× model parameters

Fine-tuning
• Supervised Fine-Tuning (SFT): Train on curated instruction-response pairs
• RLHF (Reinforcement Learning from Human Feedback): Human rankings → reward
model → PPO optimization
• RLAIF: AI feedback instead of human feedback
• DPO (Direct Preference Optimization): Directly optimize preferences without RL

Parameter-Efficient Fine-Tuning (PEFT)

• LoRA: Low-Rank Adaptation: train small rank-decomposed matrices only

Page 6 of 22
Generative AI — Complete Study Notes

• QLoRA: Quantized LoRA — 4-bit quantization + LoRA

• Prefix Tuning: Add trainable prefix tokens to input
• Prompt Tuning: Soft prompts — learnable embeddings prepended to input
• Adapter layers: Small bottleneck layers inserted between transformer layers

LoRA Formula (Exam Favourite)

Original weight: W ∈ R^(d×k)
LoRA update: ΔW = BA where B ∈ R^(d×r), A ∈ R^(r×k), rank r << min(d,k)
Forward pass: h = (W + ΔW)x = Wx + BAx
Parameters saved: instead of d×k, train only r×(d+k) — up to 10,000× fewer params
α (scaling): controls contribution of LoRA update: ΔW scaled by α/r

3.4 Prompting Techniques

Basic Prompting
• Zero-shot: No examples — just instruction
• One-shot: One example + instruction
• Few-shot: 2–8 examples demonstrating the task

Advanced Prompting
• Chain-of-Thought (CoT): Ask model to reason step-by-step before answering
• Zero-shot CoT: 'Let's think step by step' appended to prompt
• Tree of Thoughts (ToT): Explore multiple reasoning paths, pick best
• ReAct: Interleave Reasoning and Acting (tool use)
• Self-Consistency: Sample multiple CoT paths, take majority answer
• Least-to-Most: Decompose complex problem into sub-problems
• Role prompting: 'You are an expert in...' — sets persona/context
• System prompts: Instructions given before conversation (e.g., ChatGPT, Claude)

3.5 Inference & Generation Strategies

Decoding Methods
• Greedy: Always pick highest probability token — deterministic, repetitive
• Beam Search: Keep top-k beams — better quality, still deterministic
• Top-k Sampling: Sample from top k tokens — diverse output
• Top-p (Nucleus) Sampling: Sample from smallest set with cumulative prob ≥ p
• Temperature: T<1 → sharper (conservative), T>1 → flatter (creative)

Parameters
• max_tokens / max_length: limits output length
• repetition_penalty: penalizes repeated tokens
• presence_penalty / frequency_penalty (OpenAI): reduce repetition

Page 7 of 22
Generative AI — Complete Study Notes

Page 8 of 22
Generative AI — Complete Study Notes

Unit 4: Generative Models for Images

4.1 Generative Adversarial Networks (GANs)

GAN Architecture
Two networks trained in opposition:
Generator G: Takes random noise z ~ p(z) → generates fake data G(z)
Discriminator D: Classifies real vs. fake → D(x) ∈ [0,1]

Min-Max Objective:
min_G max_D E[log D(x)] + E[log(1 - D(G(z)))]

Training: Alternate between updating D (maximize) and G (minimize)

GAN Variants
• DCGAN: Deep Convolutional GAN — stable training with conv layers
• WGAN: Wasserstein GAN — uses Earth Mover distance, avoids mode collapse
• CycleGAN: Image-to-image translation without paired data
• StyleGAN / StyleGAN2/3: High-quality face synthesis, style mixing
• Pix2Pix: Conditional image translation with paired data
• BigGAN: Large-scale class-conditional image generation

GAN Training Challenges

• Mode collapse: Generator produces limited variety
• Training instability: Discriminator/generator can overpower each other
• Vanishing gradients: If D too good, G gets no gradient signal

4.2 Variational Autoencoders (VAEs)

VAEs learn a latent space where similar items are close together, enabling smooth interpolation
and generation.

VAE Components
Encoder: q(z|x) — maps input x to distribution over latent z (mean μ, variance σ²)
Reparameterization trick: z = μ + σ·ε, ε ~ N(0,I) [makes sampling differentiable]
Decoder: p(x|z) — reconstructs x from z

Loss = Reconstruction Loss + KL Divergence

= E[log p(x|z)] - KL(q(z|x) || p(z))

KL Divergence: forces latent space towards N(0,I), enables smooth generation

Page 9 of 22
Generative AI — Complete Study Notes

4.3 Diffusion Models

Diffusion models learn to reverse a gradual noising process. Currently state-of-the-art for image
generation.

Forward Process (Noising)

• Gradually add Gaussian noise over T steps: q(xt|xt-1) = N(xt; √(1-β)·xt-1, β·I)
• After T steps, image ≈ pure Gaussian noise

Reverse Process (Denoising)

• Learn p_θ(xt-1|xt) — predict and remove noise step by step
• U-Net architecture with attention used as backbone
• Predict noise ε or predict x0 directly

Key Models
• DDPM: Denoising Diffusion Probabilistic Models (Ho et al., 2020)
• DDIM: Faster sampling — 50 steps vs 1000, deterministic
• Stable Diffusion: Latent Diffusion Models (LDM) — compress to latent space first
• DALL-E 2/3: OpenAI — CLIP + diffusion prior + decoder
• Imagen: Google — cascaded diffusion, classifier-free guidance
• Midjourney: Commercial diffusion-based image generation

Guidance Techniques
• Classifier Guidance: Use external classifier gradients to steer generation
• Classifier-Free Guidance (CFG): Train with and without conditioning; scale guidance at
inference
• CFG scale: higher = more faithful to prompt, less diverse (typical: 7–12)

4.4 Comparison: GAN vs VAE vs Diffusion

Aspect GAN VAE Diffusion
Training Adversarial (unstable) Maximizes ELBO Denoising MSE
(stable) (stable)
Sample Quality High (but mode Moderate (blurry) Highest (SOTA)
collapse)
Latent Space Not explicit/structured Smooth, interpretable No explicit latent
Speed Fast (single forward Fast Slow (many steps)
pass)
Control Limited Good interpolation Excellent with
guidance
Applications Faces, style transfer Anomaly detection, Text-to-image SOTA
VAE-LM

Page 10 of 22
Generative AI — Complete Study Notes

Unit 5: Multimodal Models & Vision-Language

5.1 CLIP (Contrastive Language-Image Pre-training)
OpenAI's CLIP learns joint embeddings of images and text via contrastive learning on 400M
image-text pairs.

CLIP Training Objective

Encode N images: {I1...IN} → image embeddings
Encode N texts: {T1...TN} → text embeddings
Maximize cosine similarity of correct (image, text) pairs
Minimize similarity of incorrect pairs (N²-N negatives)
Uses temperature-scaled cross-entropy (InfoNCE loss)

Zero-shot classification: encode class labels as text, find nearest image embedding

5.2 Vision-Language Models (VLMs)

• BLIP / BLIP-2: Bootstrap language-image pre-training, Q-Former bridges vision-
language
• LLaVA: Large Language and Vision Assistant — connects CLIP encoder to LLaMA
• GPT-4V: Vision-capable GPT-4 — accepts image + text inputs
• Gemini: Google's natively multimodal model — text, image, audio, video
• Claude 3+: Anthropic's multimodal models with vision capabilities
• Flamingo: DeepMind — few-shot visual question answering

5.3 Text-to-Video Models

• Sora (OpenAI): Diffusion transformer for realistic long video generation (2024)
• Runway Gen-2/3: Commercial text-to-video and video editing
• Pika: Fast text/image-to-video
• Lumiere (Google): Space-time diffusion model for video

5.4 Text-to-Speech & Audio Generation

• WaveNet (DeepMind): Autoregressive raw waveform generation
• Tacotron / Tacotron2: Text → Mel spectrogram → WaveNet vocoder
• FastSpeech2: Non-autoregressive TTS — much faster
• VALL-E (Microsoft): Zero-shot voice cloning from 3s audio
• MusicLM / MusicGen: Text-to-music generation
• Whisper (OpenAI): Automatic speech recognition (ASR)

Page 11 of 22
Generative AI — Complete Study Notes

Page 12 of 22
Generative AI — Complete Study Notes

Unit 6: RAG & Agents

6.1 Retrieval-Augmented Generation (RAG)
RAG combines the knowledge retrieval capability of search with the generation capability of
LLMs to produce grounded, factual responses.

RAG Pipeline
1. INDEXING: Chunk documents → Embed chunks → Store in vector database
2. RETRIEVAL: Embed query → Find top-k similar chunks (cosine/MMR similarity)
3. AUGMENTATION: Prepend retrieved chunks to LLM prompt
4. GENERATION: LLM generates answer conditioned on retrieved context

Advantages: No retraining needed | Reduces hallucination | Fresh knowledge | Citable

Vector Databases
• Pinecone: Managed vector DB, scalable
• Weaviate: Open-source, hybrid search
• Chroma: Lightweight, Python-native
• FAISS (Facebook): Efficient similarity search library
• Qdrant: Open-source, Rust-based, fast
• pgvector: PostgreSQL extension for vectors

Embedding Models
• text-embedding-3-small/large: OpenAI embeddings
• E5, BGE, GTE: Open-source embedding models
• sentence-transformers: HuggingFace library for dense embeddings
• Cohere Embed: High-quality commercial embeddings

Advanced RAG Techniques

• HyDE: Generate hypothetical document → use its embedding for retrieval
• Reranking: Use cross-encoder to rerank retrieved chunks (Cohere, BGE-Reranker)
• Multi-query: Generate multiple queries from one question
• Parent-child chunking: Retrieve small chunks, return parent context
• GraphRAG (Microsoft): Build knowledge graph → graph-aware retrieval

6.2 LLM Agents

Agents use LLMs as a reasoning engine to take actions, use tools, and complete multi-step
tasks autonomously.

Agent Components
• Brain/LLM: Core reasoning — decides what to do next

Page 13 of 22
Generative AI — Complete Study Notes

• Memory: Short-term (conversation), long-term (vector store), episodic

• Tools: Web search, code interpreter, APIs, database queries, file I/O
• Planning: Decompose tasks, handle sub-goals, reflect on progress

Agent Frameworks
• ReAct: Reasoning + Acting — interleaved thought and tool-use
• LangChain: Popular Python/JS framework for chains and agents
• LlamaIndex: Data framework for LLM applications
• AutoGen (Microsoft): Multi-agent conversation framework
• CrewAI: Role-based multi-agent collaboration
• OpenAI Assistants API: Built-in code interpreter, retrieval, function calling

Tool Use / Function Calling

• Define tools as JSON schemas (name, description, parameters)
• LLM decides when to call which tool with what arguments
• Tool result fed back into context, LLM continues
• OpenAI: function_calling → tool_choice API

Page 14 of 22
Generative AI — Complete Study Notes

Unit 7: Evaluation Metrics

7.1 NLP Generation Metrics
Metric Description & Formula
Perplexity 2^H(p,q) = exp(-1/N Σ log p(xi)). Lower = better LM. NOT for quality.
BLEU Precision of n-gram overlap with references. BP·exp(Σ wn·log pn). 0–1.
ROUGE-N Recall of n-gram overlap. Used for summarization.
ROUGE-L Longest Common Subsequence (LCS) based recall.
METEOR Aligns hypotheses with references, considers synonyms.
BERTScore Contextual embeddings similarity — correlates better with human judgment.
MoverScore Earth Mover Distance between BERT embeddings.
MAUVE Measures distribution overlap between generated and real text.

7.2 Image Generation Metrics

Metric Description
FID (Fréchet Inception Compares real vs. generated image distributions in Inception feature
Distance) space. Lower = better.
IS (Inception Score) Measures quality and diversity using conditional label distributions.
Higher = better.
CLIP Score Cosine similarity between CLIP embeddings of image and caption.
Measures text alignment.
Precision/Recall (GenAI) Precision: fidelity (are generated images real-looking?). Recall:
diversity (cover real distribution?).
SSIM Structural Similarity Index — pixel-level similarity metric.

7.3 LLM Benchmark Leaderboards

• MMLU: Massive Multitask Language Understanding — 57 subjects, multiple choice
• HellaSwag: Commonsense inference — complete sentence
• HumanEval / MBPP: Code generation benchmarks (Python functions)
• GSM8K: Grade-school math word problems
• MATH: Competition-level mathematics
• GPQA: Graduate-level Q&A in STEM
• MT-Bench: Multi-turn conversation evaluation by GPT-4 judge
• LMSYS Chatbot Arena: Human ELO-based ranking via blind pairwise comparisons

Page 15 of 22
Generative AI — Complete Study Notes

Unit 8: Ethics, Safety & Governance

8.1 Key Risks & Harms
• Hallucination: LLMs generate plausible but factually incorrect content
• Bias & Fairness: Models reflect training data biases — race, gender, culture
• Privacy: Memorization of PII, copyright content
• Disinformation: Deepfakes, synthetic media, fake news at scale
• Dual Use: Same technology can be used for good or harm
• Misuse: Phishing, fraud, malware generation, academic dishonesty
• Environmental impact: Massive energy consumption for training & inference
• Labor displacement: Automation of cognitive tasks
• Concentration of power: Few organizations control most powerful models

8.2 AI Safety Approaches

• Constitutional AI (Anthropic): Train on AI-generated critiques and revisions based on
a constitution
• RLHF / RLAIF: Align outputs with human/AI preferences
• Red-teaming: Adversarial testing — try to elicit harmful outputs
• Guardrails: Input/output filters — NeMo Guardrails, Llama Guard
• Interpretability: Mechanistic understanding of model internals (circuits, features)
• Scalable oversight: Methods to supervise AI on tasks humans can't easily evaluate

8.3 Hallucination
Hallucination is a major failure mode where models generate confident but wrong information.

Types
• Factual hallucination: Wrong facts about real entities
• Faithfulness hallucination: Contradicts provided context
• Intrinsic hallucination: Contradicts source document
• Extrinsic hallucination: Cannot be verified from source

Mitigation
• RAG — ground in retrieved facts
• Self-consistency / majority voting
• Chain-of-thought reasoning
• Calibration — model should know what it doesn't know
• FACTS grounding — attribution-based evaluation

Page 16 of 22
Generative AI — Complete Study Notes

8.4 Regulatory Landscape

Framework / Law Key Provisions
EU AI Act (2024) Risk-based tiers; foundation models must register; GPAI providers
must comply
GDPR (EU) Data privacy; right to explanation; restricts training on EU personal
data
Executive Order (US, 2023) Safety testing, watermarking, NIST AI Risk Management Framework
China AI Regulations Algorithm registration; generative AI must apply security assessment
NIST AI RMF Govern, Map, Measure, Manage — voluntary framework
IEEE Ethics Guidelines Transparency, accountability, explainability principles

Page 17 of 22
Generative AI — Complete Study Notes

Unit 9: Deployment & MLOps

9.1 Model Serving & Optimization
Quantization
• FP32 → FP16/BF16: 2× memory reduction, minimal quality loss
• INT8 quantization: 4× memory, slight quality loss — LLM.int8()
• INT4 / GPTQ: 8× memory — GPTQ, AWQ algorithms
• GGUF ([Link]): CPU-friendly quantized format for local inference

Model Parallelism
• Data Parallelism: Same model on N GPUs, different data batches
• Tensor Parallelism: Split individual layers across GPUs (Megatron-LM)
• Pipeline Parallelism: Different layers on different GPUs
• Mixture of Experts (MoE): Route tokens to sparse subset of expert FFN layers

Inference Optimization
• KV Cache: Cache Key/Value matrices to avoid recomputation
• Flash Attention: IO-aware attention — 2–4× faster, less memory
• Speculative Decoding: Small model drafts tokens, large model verifies
• Continuous batching: Dynamic batching for high-throughput serving (vLLM)
• PagedAttention (vLLM): Paged memory management for KV cache — 24× throughput

9.2 Frameworks & Tools

Category Tools
Training PyTorch, TensorFlow, JAX/Flax, DeepSpeed, FSDP, Megatron-LM
Fine-tuning HuggingFace Transformers, PEFT library, Axolotl, LLaMA-Factory
Inference vLLM, TensorRT-LLM, ONNX Runtime, [Link], Ollama, TGI
LLM Apps LangChain, LlamaIndex, Haystack, Semantic Kernel
Evaluation HELM, lm-evaluation-harness, Promptfoo, Ragas (RAG)
Monitoring Weights & Biases, MLflow, LangSmith, Arize AI
Deployment AWS SageMaker, Google Vertex AI, Azure ML, Modal, Replicate

Page 18 of 22
Generative AI — Complete Study Notes

Unit 10: Cutting-Edge Topics

10.1 Mixture of Experts (MoE)
MoE models activate only a fraction of parameters per token, enabling massive scale at lower
compute cost.
• Router network: selects top-k experts for each token
• Sparse activation: only k of N experts compute for each token
• GPT-4 rumoured to be MoE (~8 experts, 2 active)
• Mixtral 8x7B: 46.7B total params, but only 12.9B active per token
• Load balancing loss: ensure all experts are used equally

10.2 Long Context & Extended Attention

• Context window evolution: GPT-2: 1K → GPT-4: 128K → Gemini 1.5: 1M tokens
• RoPE + YaRN: Extends RoPE position encoding to longer sequences
• Ring Attention: Distributes long sequences across multiple devices
• Mamba (SSM): State Space Model — linear complexity vs O(n²) attention
• RWKV: RNN-like but parallelizable — constant memory per token

10.3 Multimodal & Emerging Modalities

• Any-to-Any models: Single model handles text, image, audio, video I/O
• 3D generation: NeRF, Gaussian Splatting, Point-E, Shap-E
• Code generation: Codex, StarCoder2, DeepSeek-Coder, GitHub Copilot
• Protein generation: ESMFold, RFdiffusion, AlphaFold 3
• Drug discovery: Generative models for molecular design (SMILES, graphs)

10.4 Reasoning & Planning

• OpenAI o1/o3: Test-time compute scaling — think longer = better accuracy
• DeepSeek-R1: Open-source reasoning model using GRPO (RL without critic)
• Process Reward Models (PRM): Reward each reasoning step, not just final answer
• Monte Carlo Tree Search: Applied to LLM generation for systematic search
• Self-play / Constitutional AI: Iterative self-improvement via AI feedback

10.5 Edge & Efficient GenAI

• Small Language Models (SLMs): Phi-3 (3.8B), Gemma (2B/7B), Qwen2.5 — near-
GPT-4 quality at tiny scale
• Model distillation: Small student learns from large teacher model

Page 19 of 22
Generative AI — Complete Study Notes

• On-device inference: Apple Neural Engine, Qualcomm NPU, MediaTek AI processors

• ONNX / CoreML / TFLite: Cross-platform deployment formats

Page 20 of 22
Generative AI — Complete Study Notes

Quick Reference: Key Formulas

Transformer / Attention
Attention(Q,K,V) = softmax(QK^T/√dk)V
FFN(x) = max(0, xW1+b1)W2+b2
LayerNorm(x) = (x-μ)/σ · γ + β
Positional Encoding: PE(pos,2i)=sin(pos/10000^(2i/d)), PE(pos,2i+1)=cos(...)

Training Objectives
Causal LM: L = -Σ log P(xi | x1...xi-1)
Masked LM: L = -Σ log P(xi | x\xi) for masked tokens
Seq2Seq: L = -Σ log P(y | x)
ELBO (VAE): L = E[log p(x|z)] - KL(q(z|x) || p(z))
GAN: min_G max_D E[log D(x)] + E[log(1-D(G(z)))]
Diffusion: L = E[||ε - ε_θ(xt,t)||²]

Evaluation Metrics Summary

Perplexity: exp(-1/N Σ log p(xi)) → Lower is better (language modeling)
BLEU: BP · exp(Σwn·log pn) → Higher is better (translation, 0–1)
ROUGE-L: F1 based on LCS → Higher is better (summarization)
FID: ||μr-μg||² + Tr(Σr+Σg-2(ΣrΣg)^0.5) → Lower is better (image generation)
BERTScore: cosine sim of BERT embeddings → Higher is better (semantic similarity)

Cheat Sheet: Model Families

Model Org Type Key Feature
GPT-4 / 4o OpenAI Decoder LLM Multimodal, 128K context, RLHF
Claude 3.x Anthropic Decoder LLM Constitutional AI, 200K context
Gemini Ultra/Pro Google Multimodal Natively multimodal, 1M context
LLaMA 3.1 Meta Decoder LLM Open weights, 405B, 128K context
Mistral / Mixtral Mistral AI Decoder / MoE Efficient, open, sliding window attn
Falcon TII UAE Decoder LLM Multi-query attention, open weights
DALL-E 3 OpenAI Diffusion Integrated with ChatGPT, instruction following
Stable Diffusion Stability AI Latent Diffusion Open source, high quality images
XL
Midjourney v6 Midjourney Diffusion Aesthetic quality, photorealism

Page 21 of 22
Generative AI — Complete Study Notes

Sora OpenAI Video Diffusion 60s photorealistic video from text

Whisper OpenAI ASR Transformer Multilingual speech recognition
CLIP OpenAI Contrastive Shared image-text embedding space

— End of Generative AI Complete Notes —

Page 22 of 22