Course Syllabus

Semester – VI
Generative AI with Large
Course Name : Course Code : MCSGA301
Language Models
Number of Lecture
: 03 CCE Marks : 50
Hours / Week
Number of Tutorial
/ Practical Hours / : 00 SEE Marks : 50
Week
Total Number of
Lecture + Tutorial/ : 40 SEE Duration : 03 Hrs
Practical Hours
L:T:P : 3:0:0 CREDITS : 03

Course Prerequisites:
Linear Algebra (matrices), Calculus (gradient descent), and Probability (cross-entropy). Advanced
Python (OOP) and PyTorch (custom training loops). Deep Learning architectures (RNNs, LSTMs) and
basic NLP concepts (tokenization).
Course Overview:
Generative AI and Large Language Models is a core postgraduate-level course designed for Master
of Technology ([Link]) students specializing in Artificial Intelligence, Computer Science, or Data
Science.
The curriculum transitions students from foundational deep sequence modeling to engineering
production-ready, large-scale language systems. It emphasizes architectural execution from scratch,
distributed cloud infrastructure configurations, hardware-aware optimization, memory
augmentation strategies, and automated agent orchestration.
Course Learning Objectives (CLOs):
This course will enable students to:
• Understand Architectural Fundamentals: Comprehend the mathematical and structural
mechanics of transformers, self-attention, and language tokenization paradigms.
• Apply Advanced Engineering: Execute pre-training scaling laws, distributed multi-GPU
training, and custom parameter-efficient fine-tuning (PEFT) implementations.
• Analyze System Constraints: Evaluate performance tradeoffs between model scale,
quantization compression, retrieval accuracy, and resource utilization in deployment
environments.
• Synthesize Enterprise Architectures: Design complete production systems integrating
memory-augmented data layers, multi-agent reasoning paths, and alignment frameworks.

Teaching
Modules
Hours
Module 1
Foundations of Transformer Architectures:
Scaled dot-product attention, multi-head attention, positional encodings
(Absolute, Relative, RoPE), Encoder vs. Decoder-only structures.

Textbook Mapping: 08
Textbook 2: Chapter 2 (Text Data Tokenization) & Chapter 3 (Coding
Attention Mechanisms).
Textbook 1: Chapter 1 (Transformer Overview) & Chapter 3 (Anatomy of a
Transformer).
Module 2
Large-Scale Pre-training & Hardware Scaling:
Data curation, tokenization paradigms, compute scaling laws (Chinchilla),
distributed training (Data, Tensor, Pipeline parallelism), memory optimization
(ZeRO, FlashAttention).

08
Textbook Mapping:
Textbook 2: Chapter 4 (Implementing GPT Architecture) & Chapter 5 (Pre-
training Systems).
Textbook 1: Chapter 10 (Training Transformers from Scratch).
Module 3
Advanced Fine-Tuning & Model Alignment:
Instruction Tuning (SFT), Parameter-Efficient Fine-Tuning (PEFT, LoRA,
QLoRA), Preference Optimization (RLHF, DPO, ORPO).

08
Textbook Mapping:
Textbook 2: Chapter 6 (Fine-Tuning for Classification) & Chapter 7 (Instruction
Tuning).
Textbook 1: Chapter 11 (Alignment and Future Directions).
Module 4
Memory Augmentation, Vector DBs & Agents:
Retrieval-Augmented Generation (RAG), vector databases (HNSW indexing),
reasoning frameworks (CoT, ReAct), multi-agent orchestration.

08
Textbook Mapping:
Textbook 3: Chapter 4 (RAG Architectures), Chapter 5 (Vector Databases), &
Chapter 7 (Advanced Orchestration and Agents).
Module 5
Compression, Evaluation & Multimodality:
Post-Training Quantization (AWQ, GPTQ), evaluation frameworks (MMLU,
HELM), Vision-Language Models (CLIP, LLaVA).
08

Textbook Mapping:
Textbook 1: Chapter 8 (Model Compression and Speed Optimization).
Textbook 3: Chapter 9 (Multimodal Models and Vision).
Textbooks
1. [Tunstall] Natural Language Processing with Transformers (Revised Edition) – Tunstall, von
Werra, & Wolf (O'Reilly).
2. [Raschka] Build a Large Language Model (From Scratch) – Sebastian Raschka (Manning).
3. [Auffarth] Generative AI Architectures – Ben Auffarth (Packt).
Reference Books
1. Jurafsky, D., & Martin, J. H. (2024). *Speech and Language Processing* (3rd ed. Draft).
Stanford University.
2. Banerjee, S. D. (2024). *Large Language Machine Learning Systems: Scalability and
Architecture*. Packt Publishing.
3. 3. Kumar, R. (2025). *Generative AI Systems: Architecture and Engineering*. O'Reilly Media.
Course Outcomes (COs):
Upon completion of the course students will be able to:
CO1 Understand: Explain the mathematical operations of scaled dot-product attention,
positional encodings, and diverse LLM architectural variants.
CO2 Apply: Implement an end-to-end causal language model and execute domain-
specific instruction tuning using LoRA in PyTorch.
CO3 Analyze: Diagnose GPU memory bottlenecks during scaling and analyze
performance losses when compressing models via post-training quantization.
CO4 Analyze: Critique retrieval quality inside hybrid RAG architectures and evaluate
multi-agent orchestration frameworks for reasoning errors.