Essential Math for Machine Learning
1. Linear Algebra - Foundation of ML computations
- Scalars, Vectors, Matrices, Tensors: Data representation in ML
- Matrix operations (add, multiply): Core of ML model computation
- Dot product, Cross product: Similarity, projection, attention
- Transpose, Inverse, Identity matrix: Used in normalization, linear systems
- Rank, Span, Basis, Linear Independence: Model complexity and solvability
- Systems of linear equations (Ax = b): Underpins least squares, regression
- Determinants: Understanding invertibility
- Eigenvalues & Eigenvectors: PCA, covariance, stability of systems
- Diagonalization, Spectral decomposition: Used in PCA, SVD
- Singular Value Decomposition (SVD): Dimensionality reduction
- Projections & Orthogonality: Linear regression, loss minimization
- Norms (L1, L2) & distances: Regularization, feature scaling
2. Calculus - Optimization and gradients
- Functions and Limits: Basics of learning functions
- Derivatives (1D): Rate of change, slope of loss
- Partial derivatives: Multi-variable loss functions
- Chain Rule: Backpropagation in neural nets
- Gradients: Steepest descent, optimization
- Directional derivatives: How loss changes across directions
- Jacobian & Hessian: Multivariable optimization, curvature
Essential Math for Machine Learning
- Gradient Descent: Training of ML/DL models
- Multivariate calculus: Needed in backpropagation
- Taylor Expansion (optional): Optimization, model approximation
3. Probability & Statistics - Handling uncertainty in ML
- Random Variables: Represent predictions, events
- Discrete vs Continuous Variables: Different ML problems
- Probability Distributions: Model assumptions
- Joint, Marginal, and Conditional Probabilities: Bayesian models
- Bayes' Theorem: Probabilistic inference
- Expectation, Variance, Std. Deviation: Loss functions and model analysis
- Covariance, Correlation: Feature relationships, PCA
- Law of Large Numbers, CLT: Justify assumptions in data modeling
- Maximum Likelihood Estimation (MLE): Parameter estimation
- Log-Likelihood: Loss functions in probabilistic models
- Hypothesis Testing (optional): Model evaluation and comparison
4. Discrete Math & Logic - Foundations for algorithms
- Sets, Relations, Functions: Data mapping, function properties
- Logic & Propositional Calculus: Algorithmic reasoning
- Combinatorics: Probabilistic models, generative models
- Boolean algebra: Logic in model evaluation
- Graphs (Intro only): For neural network connectivity
Essential Math for Machine Learning
- Recurrence relations (optional): Algorithm complexity
5. Optimization - Training ML models
- Cost/Loss functions: Basis for model learning
- Gradient Descent variations: Core training algorithms
- Momentum, RMSProp, Adam: Modern optimization methods
- Learning rate scheduling: Performance tuning
- Convex vs Non-convex functions: Optimization difficulty
- Saddle points, Local vs Global minima: Training challenges
- Constraints & Lagrange Multipliers: Regularized models
- KKT Conditions (optional): Advanced constrained optimization
6. Information Theory - Understanding learning from information
- Entropy: Measure of uncertainty
- Cross Entropy: Classification loss function
- KL Divergence: Distribution comparison
- Mutual Information: Feature selection, attention mechanisms
- Data Compression basics (optional): Information bottleneck theory
7. Numerical Methods - Making ML computation practical
- Floating-point precision & errors: Model stability
- Matrix inversion (numerical): Efficient computation
- Iterative methods: Solving large systems
- Numerical differentiation: Analytical gradient is hard
Essential Math for Machine Learning
- Gradient checking: Debugging backpropagation
- Matrix decompositions: Efficient computation in ML
8. Graph Theory (Optional) - Used in GNNs, PageRank, etc.
- Graph Representation: GNN input format
- BFS/DFS: Graph traversal
- Shortest path algorithms: Routing, RL
- Connected Components, Trees: Hierarchical models
- Centrality, PageRank: Ranking, influence models
- Spectral Graph Theory (optional): GNNs and Laplacian matrices