Module-5 Parametric and Non-Parametric Methods for Classifica on
and Regression
Parametric Methods
Parametric methods assume a specific form or distribu on for the underlying data and use this
assump on to make predic ons. These methods require es ma ng a fixed number of
parameters.
1. Maximum Likelihood Es ma on (MLE)
MLE is a method of es ma ng the parameters of a sta s cal model that maximizes the
likelihood func on.
Steps:
1. Define the likelihood func on based on the assumed data distribu on.
2. Derive the likelihood func on with respect to the parameters.
3. Solve the equa ons to find the parameter values that maximize the likelihood.
Applica ons: Used in logis c regression, Gaussian models, and many other parametric
models.
2. Bayes’ Es mator
Bayes’ Es mator incorporates prior knowledge (prior probability) into parameter es ma on
using Bayes' theorem.
Formula:
where:
o : Posterior probability.
o : Likelihood.
o : Prior probability.
o : Evidence.
Applica ons: Bayesian regression, Naïve Bayes classifier.
3. Parametric Classifica on and Regression
Classifica on: Algorithms like logis c regression and linear discriminant analysis assume
a specific func onal form for class boundaries.
Regression: Linear regression assumes a linear rela onship between independent and
dependent variables.
Non-Parametric Methods
Non-parametric methods make fewer assump ons about the data’s distribu on and do not
assume a fixed number of parameters. They are flexible and can model complex rela onships.
1. Kernel Es mator
Kernel methods es mate the probability density func on or regression func on without
assuming a specific distribu on.
Kernel Density Es ma on (KDE): where:
o : Kernel func on (e.g., Gaussian, Epanechnikov).
o : Bandwidth parameter controlling smoothness.
Applica ons: Smoothing data, probability density es ma on.
2. K-Nearest Neighbor (KNN) Es mator
KNN is a lazy learning algorithm that uses the data points closest to the input to make
predic ons.
Steps:
1. Choose the number of neighbors .
2. Compute the distance between the input and all data points.
3. Select the nearest neighbors.
4. For classifica on: Use majority vo ng. For regression: Use the average of
neighbors’ values.
Strengths: Simple, no training phase.
Weaknesses: Sensi ve to noise and the choice of .
3. Decision Trees
Decision trees split data into subsets based on feature values, crea ng a tree-like structure for
decision-making.
Univariate Trees
Univariate trees use a single feature for each split.
Example: Spli ng based on age in a dataset about health outcomes.
Classifica on Trees
Classifica on trees predict categorical outcomes by spli ng data to maximize class separability.
Spli ng Criterion:
o Gini index
o Entropy (informa on gain)
Example: Predic ng whether a customer will churn.
Regression Trees
Regression trees predict con nuous outcomes by spli ng data to minimize variance within
subsets.
Spli ng Criterion: Mean squared error (MSE).
Example: Predic ng house prices.
4. Rules Extrac on from Trees
Decision trees can be transformed into rule-based systems.
Process:
1. Trace paths from the root to leaves.
2. Create rules for each path.
3. Combine rules to form a rule set.
Example: “If age > 30 and income > 50k, then loan approval = Yes.”
Advantages: Interpretability and simplicity.
Comparison of Parametric and Non-Parametric Methods
Aspect Parametric Methods Non-Parametric Methods
Assump ons Strong assump ons about data distribu on. Minimal assump ons.
Complexity Simpler, faster to compute. Can be computa onally intensive.
Flexibility Limited by predefined model. Highly flexible.
Performance Works well with structured data. Suitable for unstructured data.
Applica ons
Parametric Methods: Used when data follows a known distribu on or model.
o Example: Predic ng stock prices using linear regression.
Non-Parametric Methods: Used for complex datasets without prior knowledge of the
distribu on.
o Example: Image classifica on using KNN.