Scribd
Upload
3 views2 pages

Tutorial 2

This document is a tutorial on the basics of probability for machine learning, covering topics such as probability models, the law of large numbers, concentration inequalities, and probability and information theory. Key concepts include various probability distributions (Binomial, Normal, Beta, Dirichlet), the weak and strong laws of large numbers, and the central limit theorem. Additionally, it discusses entropy and Kullback–Leibler divergence with relevant proofs and computational examples.

Uploaded by

Ishaan Seth
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
3 views2 pages

Tutorial 2

This document is a tutorial on the basics of probability for machine learning, covering topics such as probability models, the law of large numbers, concentration inequalities, and probability and information theory. Key concepts include various probability distributions (Binomial, Normal, Beta, Dirichlet), the weak and strong laws of large numbers, and the central limit theorem. Additionally, it discusses entropy and Kullback–Leibler divergence with relevant proofs and computational examples.

Uploaded by

Ishaan Seth
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Tutorial 2 EE-5180 Machine Learning: Basics of Probability for

Machine Learning
Rahul Meshram
March 6, 2026

0.1 Probability Models


1. Explain properties of probability density function for continuous random variable.
2. Explain properties of cumulative distribution function (CDF).
3. Explain Binomial distribution.
4. Explain Normal distribution
5. Explain Beta distribution and Study Beta distribution when true probability is unknown but only
success and failures are observed.
import numpy as np
import [Link] as plt
from [Link] import beta

# parameters
alpha = 300
beta_param = 200

# x values
x = [Link](0, 1, 200)

# beta pdf
pdf = [Link](x, alpha, beta_param)

# plot
[Link](x, pdf)
[Link]("x")
[Link]("Density")
[Link]("Beta Distribution PDF")
[Link]()
6. Study Dirichlet Distribution and write python program for Distribution for this.

0.2 Law of Large Numbers


1. State weak law of large numbers (WLLNs) for independent random variables. Provide proof for WLLNs
2. State strong law of large numbers (WLLNs) for independent random variables.
3. State Law of Large numbers for Markov chains. Ergodic Theorem.
4. Explain Uniform Law of Large Numbers (ULLN).

1
5. Explain Glivenko–Cantelli Theorem.
6. Explain the central limit theorem.

0.3 Concentration inequalities


1. Read chapter 5 of Bandit Algorithms book by Csaba Szepeswari on Concentration Measure.
2. Provide proof for 1−subgaussian and confidence bound in Theorem 5 and Corollary 5.5

0.4 Probability and Information Theory


1. Explain the entropy definition and explain entropy for Bernouilli distribution (p). Is entropy function
convex or concave?
2. Explain the Kullback–Leibler divergence. (This is also referred as KL distance or divergence). State
the properties of KL divergeence.

3. Compute the KL divergence for two Bernuoilli distributions p and q.

You might also like