Machine Learning Formulas

Complete Formula Reference Guide

Har formula ka matlab, kaam, aur practical use in simple language

Neural Networks | SVM | KNN | Decision Trees | k-Means | RNN | VAE | GAN

# Topic / Formula Group What it covers

1 Backpropagation & Weight Update Neural network learning — error correction

2 Weight Initialization Xavier, He — smart starting weights

3 Convolution Output Size CNN output dimensions calculation

4 RNN / Recurrent Network Sequential data memory formulas

5 Reconstruction Loss (VAE) How well decoder recreates input

6 VAE Loss Function Total VAE training objective

7 GAN Minimax Loss Generator vs Discriminator game

8 Distance Metrics Euclidean, Manhattan, Minkowski

9 KNN Weighted Prediction Neighbor-weighted voting

10 SVM Hyperplane & Hinge Loss Maximum margin classifier

11 Kernel Functions (SVM) Polynomial and RBF kernels

12 SVR Epsilon Loss Regression version of SVM

13 Decision Tree Splits Entropy, Gini, MSE criteria

14 k-Means Cost Function Clustering objective

PART 1: Neural Network Learning

Training, Initialization & Sequence Models

Backpropagation & Weight Update

1
Neural network seekhna kaise karta hai

delta[L] = w[L+1] * delta[L+1] * a'[L]

w[L] = w[L] - eta * (dJ/dw[L]) = w[L] - eta * a[L-1] * delta[L]

Backpropagation neural network ka core learning mechanism hai. Jab network koi galat prediction
karta hai, to ye formula us galti ko peeche ki layers tak pohonchata hai. Phir weight update rule us
galti ki wajah se weights ko thoda adjust karta hai — jaise koi student apni galtiyon se seekhta hai.

› delta[L] = Layer L ka error signal — kitna galat tha ye layer

› w[L+1] = Agali layer ke weights — galti wahan se aati hai
› a'[L] = Activation function ka derivative — sensitivity measure
› eta (learning rate) = Kitna bada step lena hai adjustment ka
› a[L-1] = Pichli layer ka output — weight update calculate karne ke liye

Practical Use: Har training step mein har weight thoda adjust hota hai taake next time prediction better ho.

Xavier / Glorot & He Weight Initialization

2
Weights ko smart tareeqe se start karna

Xavier: W ~ Uniform[ -1/sqrt(f_in), +1/sqrt(f_in) ]

Xavier (tanh): W ~ Uniform[ -sqrt(6)/sqrt(f_in+f_out),

+sqrt(6)/sqrt(f_in+f_out) ]

He Init: W ~ Gaussian[ 0, sqrt(2/f_in) ]

Training shuru karne se pehle weights random set hote hain — lekin soch ke. Agar weights bahut
bade hoon to signals explode kar jaate hain, bahut chhote hoon to vanish ho jaate hain. Xavier
initialization sigmoid/tanh activations ke liye best hai, jabke He initialization ReLU activations ke saath
use hoti hai (Facebook/He et al. ki research se).

› f_in = Input features/neurons ki tadaad is layer mein

› f_out = Output neurons ki tadaad (Xavier extended mein)
› Uniform distribution = Dono taraf symmetrically random
› Gaussian = Normal distribution — bell curve se random values
› sqrt(2/f_in) = He init ka scaling factor ReLU ke liye optimize

Practical Use: Deep networks mein proper init se training 10x faster ho sakti hai — vanishing/exploding
gradient problem solve hoti hai.

Convolutional Layer Output Size

3
CNN ki output dimensions calculate karna

Output (with padding): floor((n + 2p - f)/s + 1) x floor((n + 2p -

f)/s + 1) x d_out

Output (no padding): floor((n - f)/s + 1) x floor((n - f)/s + 1) x

d_in

Jab CNN mein ek convolutional layer se image guzarti hai, to output image ki size change hoti hai. Ye
formula exactly batata hai ke output kitna bada hoga — height, width, aur depth. Ye sochna zaroori
hai taake layers ek doosre se match karein.
 › n = Input image ka size (e.g., 28x28 pixels ke liye n=28)
› f = Filter/kernel ka size (e.g., 3x3 ke liye f=3)
› p = Padding — kitne zeros border ke around add kiye
› s = Stride — filter kitne pixels jump karta hai
› d_out = Number of filters (output depth/channels)
› floor() = Round down to nearest integer

Practical Use: Ye formula use karo taake pata chale ke model mein kisi layer ki output size kya hogi aur FC
layer ke liye flatten size kya hogi.

Recurrent Neural Network (RNN)

4
Sequential data — text, speech, time-series

h_t = af(x_t, h_{t-1})

h^ = af( W_h * h^ + W_x * x^ + b_h )

y^ = af( W_y * h^ + b_y )

RNN wo network hai jo sequence data process karta hai — jaise sentence mein words ek ek karke
aate hain. Har time step t par, network current input x_t bhi dekhta hai aur apni 'memory' h_{t-1} bhi.
Ye memory ise pichle context yaad rakhne deti hai — jaise hum English sentence mein har word ka
context yaad rakhte hain.

› h_t = Hidden state — network ki current 'memory'

› x_t = Current time step ka input (e.g., current word)
› h_{t-1} = Previous time step ki memory
› W_h, W_x, W_y = Trainable weight matrices
› b_h, b_y = Bias terms
› af = Activation function (usually tanh ya ReLU)
› y^ = Current time step ka output/prediction

Practical Use: Language models, speech recognition, machine translation — sab mein RNN ya uske variants
(LSTM, GRU) use hote hain.

PART 2: Generative Models

VAE — Variational Autoencoder & GAN

Reconstruction Loss
5
VAE — decoder ne kitna sahi recreate kiya

||x - x_hat||^2
Ye Euclidean squared distance hai original input x aur reconstructed output x_hat ke darmiyan. VAE
(Variational Autoencoder) mein encoder input ko compress karta hai, decoder wapas reconstruct
karta hai. Ye loss measure karta hai ke reconstruction kitni accurate hai — smaller = better.

› x = Original input data (e.g., original image)

› x_hat = Decoder ka reconstructed output
› || ||^2 = Squared L2 norm — har dimension ka squared difference ka sum
› Value 0 = Perfect reconstruction, value bade = zyada difference

Practical Use: Image generation, denoising, anomaly detection — jab kuch unusual hai to reconstruction error
zyada hoga.

VAE Total Loss Function

6
Reconstruction + KL Divergence

L(phi, theta, x) = ||x - x_hat||^2 - (1/2) * SUM[ 1 + log(sigma_i^2) -

sigma_i^2 - mu_i^2 ]

VAE ki full training loss do parts se banti hai. Pehla part reconstruction loss hai jo decoder quality
check karta hai. Doosra part KL divergence hai jo ensure karta hai ke latent space (compressed
representation) ek organized Gaussian distribution follow kare — taake naye samples generate kiye
ja sakein meaningfully.

› ||x - x_hat||^2 = Reconstruction quality (encoder-decoder accuracy)

› KL Divergence term = Latent space ko normal distribution ke qareeb rakhna
› mu_i = Latent variable ka mean
› sigma_i^2 = Latent variable ki variance
› log(sigma_i^2) = Log-variance (stability ke liye log use hota hai)
› 1/2 SUM = Average over all latent dimensions k

Practical Use: VAE se naye realistic data samples generate kiye jaate hain — faces, handwriting, etc. Latent
space interpolation bhi possible hai.

GAN Minimax Loss

7
Generator vs Discriminator — ek ajeeb game

min_G max_D V(G,D) = E[log D(x)] + E[log(1 - D(G(z)))]

GAN mein do networks ek dusre ke against train hote hain. Discriminator D chahta hai real (x) ko real
identify kare aur fake (G(z)) ko fake. Generator G chahta hai ke Discriminator uski fake images ko
real samjhe. Ye adversarial training hai — ek forger aur ek detective ki tarha. Jab tak Generator itna
acha na ho jaye ke Discriminator confused ho jaye.

› G = Generator — noise z se fake data banana

› D = Discriminator — real vs fake pehchaanna
› x ~ P_data(x) = Real training data se sampled
› z ~ P_z(z) = Random noise vector (generator ka input)
 › D(x) = Probability ke x real hai (D chahta hai ye 1 ho)
› D(G(z)) = Fake image ke real hone ki probability (G chahta hai ye 1 ho, D chahta hai 0)
› E[ ] = Expected value over distribution

Practical Use: GAN se photorealistic faces, artwork, deepfakes, data augmentation — yeh sab generate hote
hain.

PART 3: Classical ML Algorithms

KNN, SVM, Decision Trees, k-Means

Distance Metrics
8
Do points ke darmiyan faasla measure karna

Euclidean: d(X,Y) = sqrt( SUM_{i=1}^{m} (x_i - y_i)^2 )

Manhattan: d(X,Y) = SUM_{i=1}^{m} |x_i - y_i|

Minkowski: d(X,Y) = ( SUM_{i=1}^{m} |x_i - y_i|^p )^(1/p)

Machine learning mein do data points ke darmiyan 'similarity' ya 'distance' measure karna bohat
common kaam hai. Teen main distance measures hain: Euclidean (seedhi line ka faasla — jaise ruler
se napna), Manhattan (city blocks mein chalna — sirf up/down/left/right), aur Minkowski (generalized
formula jisme Euclidean p=2 aur Manhattan p=1 special cases hain).

› X = (x1, x2, ...xm) aur Y = (y1, y2, ...ym) = Do data points m dimensions mein
› Euclidean = Straight-line 'crow flies' distance — most common
› Manhattan = Taxicab distance — grid layout mein useful
› Minkowski p=1 = Manhattan, p=2 = Euclidean, p=inf = Chebyshev
› Real applications: KNN, k-Means, SVM kernel, anomaly detection

Practical Use: KNN algorithm mein distance se nearest neighbors dhunde jaate hain. k-Means mein centroid
se distance se cluster assignment hoti hai.

KNN Weighted Prediction

9
Kareeb wale neighbors zyada vote dete hain

w_i = 1 / d(x, x_i) (Weight of neighbor i)

y_hat = SUM_{i=1}^{m} w_i * y_i / SUM_{i=1}^{m} w_i

Standard KNN mein sab K neighbors ka equal vote hota hai. Weighted KNN mein jo neighbor jitna
kareeb hai uska vote utna zyada count hota hai. Ek neighbor jo bahut door hai uska influence bahut
kam hoga. Regression ke liye ye weighted average hai, classification ke liye weighted voting hota hai.
 › w_i = Neighbor i ka weight — distance ka inverse
› d(x, x_i) = Query point x aur neighbor x_i ke darmiyan distance
› y_i = Neighbor i ki known label ya value
› Numerator = Weighted sum of neighbor values
› Denominator = Total weight (normalization ke liye)
› Jitna kareeb, utna important — intuitive idea

Practical Use: Recommendation systems mein: similar users ka zyada influence hota hai. Real estate price
prediction mein kareeb ghar zyada relevant hote hain.

SVM: Hyperplane, Hinge Loss & Objective

10
Maximum margin separator

Hyperplane: w^T * x + b = 0

Prediction: h(x) = sign(w^T * x + b)

Margins: w^T*x + b = +1 and w^T*x + b = -1

Minimize: (1/2)||w||^2

Hinge Loss: L(y, y_hat) = max(0, 1 - y * (w^T*x + b))

SVM Objective: J(w,b) = (1/2)||w||^2 + C * SUM max(0, 1 -

y^(i)*(w^T*x^(i)+b))

SVM (Support Vector Machine) do classes ko separate karne wali ek optimal line/hyperplane dhundta
hai — lekin sirf koi bhi line nahi, balke woh line jo dono classes ke sabse kareeb points se maximum
distance par ho (maximum margin). Hinge loss penalize karta hai un points ko jo galat side par hain
ya margin ke andar hain.

› w = Weight vector — hyperplane ki direction

› b = Bias — hyperplane ka position
› w^T*x + b >= 0 = Class +1, < 0 = Class -1
› ||w||^2 = w ki magnitude squared — margin = 2/||w||, isliye minimize karo
› Hinge loss = 0 jab correctly classified outside margin, positive otherwise
› C = Regularization parameter — margin vs misclassification trade-off
› Support Vectors = Woh points jo margin par ya andar hain

Practical Use: Email spam detection, image classification, medical diagnosis — SVM structured data par bohat
effective hai.

SVM Kernel Functions

11
Non-linear boundaries ke liye feature transformation
 Kernel SVM Prediction: h(x) = sign( SUM_{i=1}^n alpha_i * y^(i) *
k(x^(i), x) + b )

Polynomial Kernel: k(x, l) = (x^T * l + c)^d

RBF Kernel: k(x, l) = exp( -||x - l||^2 / (2*sigma^2) )

RBF (gamma form): k(x, l) = exp( -gamma * ||x - l||^2 ) where gamma =
1/(2*sigma^2)

Kabhi kabhi data linearly separable nahi hota — matlab seedhi line se separate nahi ho sakta. Kernel
trick data ko implicitly ek higher-dimensional space mein map karta hai jahan linear separation
possible ho. RBF kernel data points ke darmiyan Gaussian similarity measure karta hai.

› alpha_i = Dual coefficients (support vector ke liye non-zero)

› k(x, l) = Kernel function — two points ki similarity
› Polynomial: d = degree, c = constant hyperparameter
› RBF = Radial Basis Function — most popular kernel
› sigma = RBF bandwidth — Gaussian ki width
› gamma = 1/(2*sigma^2) — zyada gamma = tighter boundary

Practical Use: RBF kernel almost kisi bhi dataset par kaam karta hai — hyperparameter tuning (C aur gamma)
se performance optimize hoti hai.

Support Vector Regression (SVR)

12
SVM ka regression version

Epsilon Loss: L(y, y_hat) = max(0, |y - y_hat| - epsilon)

SVR Objective: J(w,b) = (1/2)||w||^2 + C * SUM max(0, |y^(i) -

y_hat^(i)| - epsilon)

SVR regression ke liye SVM adapt karta hai. Ek 'epsilon tube' banai jaati hai predicted values ke
around. Agar actual value tube ke andar hai to koi loss nahi. Sirf tab loss hota hai jab prediction tube
se bahar ho. Ye outliers ke against robust hai.

› epsilon = Tube ki width — allowed error margin

› |y - y_hat| = Actual aur predicted value ka absolute difference
› max(0, ...) = Tube ke andar hone par zero loss
› C = Trade-off: bada tube (flexible) vs chhota tube (tight fit)
› Bada epsilon = zyada tolerance, chhota epsilon = tighter fitting

Practical Use: Stock price prediction, demand forecasting — jahan exact value nahi, approximate range
acceptable ho.
 PART 4: Tree Models & Clustering
Decision Trees & k-Means

Decision Tree Split Criteria

13
Best split dhundhna — Entropy, Gini, MSE

Entropy: H(D) = - SUM_{i=1}^C P_i * log2(P_i)

Info Gain: IG(D,S) = H(D) - [ (D1/D)*H(D1) + (D2/D)*H(D2) ]

Gini Impurity: gini(D) = 1 - SUM_{i=1}^C P_i^2

Gini (split): gini(D,S) = (D1/D)*gini(D1) + (D2/D)*gini(D2)

MSE (regress): MSE(D) = (1/n) * SUM_{i=1}^n (y^(i) - y_bar)^2

MSE (split): MSE(D,S) = (D1/D)*MSE(D1) + (D2/D)*MSE(D2)

Decision tree banate waqt sabse important sawal hai: 'Kaunsi feature par aur kaunsi value par split
karein?' Teen popular criteria hain. Entropy/Information Gain batata hai ke split ke baad data kitna
pure ho gaya (classification). Gini Impurity faster calculate hoti hai aur similarly purity measure karti
hai. MSE regression trees ke liye — split ke baad values kitni close hain mean ke.

› P_i = Class i ki probability node mein

› Entropy = 0 (pure node), max jab equal distribution ho
› Information Gain = Parent entropy - Weighted child entropy — maximize karo
› Gini = 0 (pure), max = 1 - 1/C — lower is better
› D1, D2 = Subsets after split, D = Total samples
› y_bar = Mean of all target values in node
› Split choose karo jo maximum IG ya minimum Gini/MSE de

Practical Use: Random Forests aur Gradient Boosting inhi criteria use karte hain — ek powerful ensemble ka
base yahi simple splits hain.

k-Means Clustering Cost Function

14
Clusters ko minimize karo — similar cheezein ikathha karo

J = SUM_{j=1}^K SUM_{i=1}^{c_j} ||x^(i) - mu_j||^2

k-Means clustering unsupervised algorithm hai — koi labels nahi hote. Ye data ko K groups mein
divide karta hai taake har group mein points apne centroid ke qareeb hoon. Cost function J total
distance hai — har point ki apne assigned centroid se distance squared. Algorithm is J ko minimize
karne ki koshish karta hai iteratively.
 › K = Number of clusters (user decide karta hai)
› mu_j = Cluster j ka centroid (mean position)
› x^(i) = i-th data point jo cluster j mein assign hai
› ||x^(i) - mu_j||^2 = Point aur centroid ke darmiyan squared Euclidean distance
› c_j = Cluster j mein points ki tadaad
› Outer sum = Sab clusters par, inner sum = cluster ke andar sab points
› Algorithm: assign → update centroids → repeat jab tak converge

Practical Use: Customer segmentation, image compression, document clustering — k-Means fast aur simple,
large datasets par effective.

Quick Reference Summary

Sab formulas at a glance

Formula Algorithm Use / Purpose

delta[L] = w[L+1]*delta[L+1]*a'[L] Backpropagation Error peeche bhejta hai

w = w - eta*(dJ/dw) Gradient Descent Weights improve karta hai

W ~ U[-1/sqrt(f_in), 1/sqrt(f_in)] Xavier Init Sigmoid/tanh ke liye init

W ~ G[0, sqrt(2/f_in)] He Init ReLU ke liye init

floor((n+2p-f)/s + 1) CNN Output size calculate

h_t = af(W_h*h_{t-1} + W_x*x_t) RNN Sequential memory update

||x - x_hat||^2 VAE Reconstruction quality

||x-x_hat||^2 - (1/2)SUM(1+log(s^2)-s^2-u^2) VAE Loss Total training objective

min_G max_D E[logD(x)]+E[log(1-D(G(z)))] GAN Adversarial training

sqrt(SUM(x_i-y_i)^2) Euclidean Straight-line distance

SUM|x_i-y_i| Manhattan City-block distance

(SUM|x_i-y_i|^p)^(1/p) Minkowski Generalized distance

w_i = 1/d(x,x_i) KNN Weight Closer = more influence

y_hat = SUM(w_i*y_i)/SUM(w_i) KNN Predict Weighted neighbor average

w^T*x + b = 0 SVM Decision hyperplane

max(0, 1 - y*(w^T*x+b)) Hinge Loss SVM classification loss

(1/2)||w||^2 + C*SUM(hinge) SVM Objective Max margin optimization

exp(-||x-l||^2 / 2sigma^2) RBF Kernel Non-linear similarity

max(0, |y-y_hat| - epsilon) SVR Loss Regression tube loss

H(D) = -SUM(P_i * log2(P_i)) Entropy Node impurity

IG = H(D) - weighted H(children) Info Gain Best split selection

gini = 1 - SUM(P_i^2) Gini Index Alternative impurity

MSE = (1/n)SUM(y-y_bar)^2 Tree MSE Regression split criterion

J = SUM_j SUM_i ||x^(i) - mu_j||^2 k-Means Cluster cost minimize

Machine Learning Formula Reference Guide

Prepared with detailed explanations | All 14 formula groups covered | Neural Networks • SVMs • Trees • Clustering •
Generative Models