The Bright Side of Mathematics

The following pages cover the whole Probability Theory course of the
Bright Side of Mathematics. Please note that the creator lives from
generous supporters and would be very happy about a donation. See
more here: [Link]
Have fun learning mathematics!

1
 Probability Theory - Part 1

(Stochastic, stochastic processes, statistics,...)

Probability measures Random variables Central limit theorem

Probability distributions Random processes

Statistical tests

Example: Probability of getting an even number?

number of throws with an even outcome

number of total throws

 Probability Theory - Part 2

Probability measures: measures with total mass

area

sample space

subset collection of subsets

subset

We want:

if are disjoint

if we have pairwise disjoint sets

for

power set

Definition: Let be a set. A collection of subsets is called

a sigma algebra if:
(a)
algebra
(b) If then
elements
are called events (c) If , then

Definition: Let be a algebra A map is called a

probability measure if: (a)

(b)

if we have pairwise disjoint sets for

Example: 1 throw:

number of elements in a set

For example:

Exercise: Prove:
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 3

events form
-algebra event

sample space

(absolutely) mixed and

discrete case
continuous case other cases

"finitely many outcomes" "uncountably many outcomes"

"countably many outcomes"

-additivity:
if we have pairwise disjoint sets

discrete (abs.) continuous

sample space finite or countable set sample space uncountable

(Borel set)
(Example: Heads, Tails (Example:

-algebra -algebra

probability measure probability measure

is completely determined by for all can be described by

probability density function: with

probability mass function: with

measurable!

Define: Define:

Example: unfair die Example:

throw point into interval

with

Hence:

Lebesgue measure
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 4

Coin tossing:

Probability for

(Fair coin:

times
Drawing a ball
times
H T
H
H T

H
T
H
Probability for H
T

In both cases:

Binomial distribution: tosses of the same coin and counting the heads

(no order!)
draw balls with replacement and count the heads
H H
T H
size unordered , with replacement

two parameters

H T

H T H
T

T H T H T H T
H

In R: times
times
H T
H
H T
T
H T H
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 5

Probability space

sample space -algebra probability measure

Example: first throw a die then throw a point into the interval

possible outcome: probability?

First probability space:

Second probability space:

event
new probability space

product -algebra product

measure

satisfies for
Definition: Probability spaces:

Product space: defined by:

elements:

cylinder sets

product -algebra

product measure

Example: throw a die infinitely many times:

Product space: product -algebra product measure

event: "At the 100th throw, we get a six for the first time"

times
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 6

Hypergeometric distribution (multivariant)

size unordered , without replacement

draw balls at once

urn model

colours: finite set

for example:

one possible outcome:

Sample space:
function or

For our example:

number of balls for colour in the urn

total number of balls

(multivariant) hypergeometric distribution:

Hypergeometric distribution for two colours:

0 1
0
count the 1 0 1
1

0 1 0
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 7

Conditional probability: probability space

subset

with

new probability space:

only with

new probability space:

Definition: probability space with

is called the conditional probability of under

is called the conditional probability measure given

Property: For set

Example: urn model: ordered, without replacement First ball

Second ball

possible sample:

given by probability mass function

g r

g r g r

event: "first ball is green"

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 8

Bayes's theorem:

Law of total probability: probability space

disjoint union

disjoint union

Case with countably many sets: for with

disjoint union

disjoint union
Example: Monty Hall problem
3 doors:
- 1 car
First: You pick a door
- 2 goats

Second: Show master opens a door with a goat (never the door you picked)

Third: Stay or switch

car is behind door j show master opens door j (in the second step)

We know:

Bayes's
Law of total
theorem probability
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 9

Independence (for events)

events

independent?

events

independent!

We want: and

Example:

independent!

Recall:

Definition: Let be a probability space.

Two events are called independent if

A family with is called independent if

for all finite

Example: 2 throws with order:
uniform distribution

"first throw gives "

"sum of both throws is "

are independent

Example: throw a point into unit interval

uniform distribution
density function
for
with
and

indicator function:
else

For two independent events we have:

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 10

Random variables with some properties.

Example: Throwing two dice

uniform distribution

random variable gives sum of

the numbers the dice show

Definition: Let and be measurable spaces (= event spaces).

A map is called a random variable if

for all

Examples:(a) and

for all is a
random variable

(b) and

is not a
random variable

Notation: Let and be measurable spaces (= event spaces).

probability measure random variable

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 11

event spaces,

random variable

(abstract)

Definition: Let be a probability space, be a random variable.

with Borel sigma algebra

Then defined by

is called probability distribution of

is a
Proposition: is a probability measure.
probability measure

Proof:

For additivity: Choose pairwise disjoint.

Then:

So: pairwise disjoint.

And:

is a
probability measure
Notation: If probability measure and , then

Example: tosses of the same coin

BERNOULLI
s s

part 4

number of in
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 12

Cumulative distribution function (cdf)

random variable
(abstract)

Definition: Let be a probability space, be a random variable.

with Borel sigma algebra

is called the cumulative distribution function of

Properties:

is monotonically increasing

is right-continuous

Example: NORMAL probability density function

cdf:
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 13

two independents events

two independent random variables?

Definition: Let be a probability space and let

be two random variables.

Then are called independent if for all

and are independent events.

cdf of random variable

Example: Product space:

are independent random variables

Definition: A family is called independent if

for all

for all finite

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 14

probability space

random variable

expectation of (expected value, mean, expectancy...)

continuous case: probability density function of

Definition: probability space random variable

(abstract integral)

Change of variables:

new random variable

(for example: )

continuous case

pdf of

pmf of
discrete case
Remember:
continuous case

discrete case

Example:
throwing a fair die
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 15

Example: (exponential distribution)

pdf

Properties: probability space random variables,

where and exist.

(a)
for all

(b) If are independent, then:

(c) If , then:

(d)
If almost surely

then:
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 16

variance?

Definition: probability space random variable

new random variable

linearity

We need to assume that exists

change-of-variables
continuous case

discrete case
Examples:
(a)
Uniform discrete case with

arithmetic
mean

(b)
(exponential distribution)

integration by parts
The Bright Side of Mathematics - [Link]
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 18

Properties of variance and standard deviation:

Let be independent random variables where and exist.

Then: (a) Var Var Var

(b) Var Var for every

(c)
for every

Proof: (a)
Var

Var Var

independence

(b)
Var

Var

(c) (b)

Var
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 19

Definition: probability space

random variables

are finite

Cov

linearity

is called the covariance of and

Remember: independent Cov uncorrelated

only in special situations

(for example: normally distributed)

Property: Cov Cov Cov

Cov
Definition: correlation coefficient

Example: uniform on

Cov

for all
Independence?
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 20

random vector

random variable

projection to
first component

marginal
distribution

Definition: is called the marginal distribution of

with respect to the first component.

marginal cumulative distribution function

Two important cases:

(1) (abs.) continuous: has a probability density function

projection

projection

marginal probability density function

 (2) discrete: has a probability mass function

(only countably many are non-zero)

marginal probability mass function with

Example: uniformly distributed on

marginal probability density function

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 21

conditional probability:

is probability measure

Definition: probability space with

probability space

For a random variable , we define:

(expectation of )

(conditional expectation of given )

Remember:

indicator function:

Example: NORMAL( )
General example:

Example: Throw one die:

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 22

Recall: discrete event with

Consider discrete

joint pmf of and

and

Define:

is called the conditional expectation of given

and denoted by

Example: die throw checks if number is even

else

checks if number is the highest

else

and

and
Definition for (abs.) continuous case: with pdf

conditional density

is called the conditional expectation of given

Properties: (a) independent and

(b)

(c)
(Law of total probability)
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 23

Stochastic processes: "random variables in a row"

random experiment with time evolution

(discrete timesteps, continuous time)

board game:

coin game: toss a coin again and again until two successive heads occur

discrete time
no two successive heads two successive heads
in the first tosses in the first tosses
and th toss is "tails"

no two successive heads

in the first tosses
and th toss is "heads"

Definition: set often

For each define: (random variable/ vector)

Then: is called a stochastic process.

For the map is called path.

Example from before:

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 24

Definition: Let be a stochastic process with or

discrete-time continuous-time

We call Markov process or Markov chain if

for all

and we have:

for discrete-time Markov chain:

depends only on

transition probability
from to at time
time time

If does not depend on , then we say:

the Markov chain is time-homogeneous

Example: toss a coin again and again until two successive heads occur

H T

discrete time
no two successive heads two successive heads
in the first tosses in the first tosses
and th toss is "tails"

no two successive heads

in the first tosses
and th toss is "heads"

transition matrix

Here:

one time-step

Start the game with

one time-step

(vector-matrix-multiplication)

Law of total probability

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 25

stochastic process: subset of or

discrete-time Markov chains + time-homogeneous:

depends only on and

independent of

transition matrix

Important: entries of lie in

acts on row vectors from the right

General example:

start at probability mass function of pmf of

is given by a row vector

at
disjoint union!

law of total probability

 by induction:

Definition: is called a stationary distribution for the Markov chain if

and

Note:

eigenvalue
column vector eigenvector

Example:

Ker Ker

row operations

Ker Span

only stationary distribution

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 26

probability space

Markov's inequality: random variable.

Then satisfies:

for any

picture for

Proof:
We have: indicator function

And:

Chebyshev's inequality: random variable where

Var
Then: for any

Proof: Define: Hence Var Var

Var

Markov's inequality for

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 27

Assumption: random variable with

both should exist!

Var

probability density function

Var
Chebyshev's inequality:

-intervals:

Chebyshev's inequality
Var

For 75%

For 88.8%
-intervals for the normal distribution:

0.682...

0.954...

0.997...
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 28

law of large numbers relative

(theoretical) empirical
frequency
probability of event probability of event of the event
number of outcomes in

total number

Example:
coin toss:

repeat random experiment:

product measure
( in th toss)

define random variables:

let's look at tosses:

(relative frequency of heads in the first tosses)

we expect:
What does this convergence mean?
Weak law of large numbers: random variables.

Let be independent and identically distributed = i.i.d.

for all for all

for all finite for all Borel

sets

and

Then for and for all

We say converges in probability to the expected value

Proof: for the case Var

We have:

Var Var Var

By Chebyshev's inequality:

Var
for any
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 29

law of large numbers: repetitions

"Monte Carlo method"

Monte Carlo integration:

random points...

random variables:

Weak law of large numbers: random variables.

Let be i.i.d. and

Then for and for all

Monte Carlo integration: Given: integrable

We want:

Take: picks a point (randomly = uniformly distributed)

from the interval

What is ?
change of
area: variables
pdf
 procedure: i.i.d.+uniformely distributed on [0,1]

approximates

Example:
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 30

repeating a random experiment: i.i.d.

should lead to:

weak law of large numbers: is unlikely for large

pointwise convergence?
?

we could have:

How many have such "bad" behaviour?

Strong law of large numbers: random variables.

Let be i.i.d. and

Then for
for
almost surely

This means:

(we could have but the probability is zero)

Remark: almost sure convergence convergence in probability

strong law of large numbers weak law of large numbers

 The Bright Side of Mathematics - [Link]

Probability Theory - Part 31

Assumptions of the central limit theorem: i.i.d. with Var

part 28
Var
satisfies Var

Example:
1 1 urn model without replacement hypergeometric distribution
0
0 1
(part 6)

picks 3 balls and counts numbers of 1

What is the distribution?

close to normal distribution!

Var
Var
Standardize the random variable: Var

(1) expectation should be zero:

(2) variance should be one:

Central limit theorem: For i.i.d. with Var define:

where Var

Then the cdf of converges to the cdf of Normal( , ) :

for every
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 32

De Moivre–Laplace theorem (special case of central limit theorem)

approximation of binomial distribution Bin( )

times Bernoulli
Galton board

Result: For large and close to , we get:

pmf of binomial distribution pdf of normal distribution

De Moivre–Laplace theorem: Let Then for any

max
increase

with
 The Bright Side of Mathematics - [Link]

Probability Theory - Part 33

Statistics descriptive statistics inferential statistics

"all the theoretically possible data"

The rough idea:
sample population
real world
finite set of data points
prediction

analyzing
inferential statistics:
summarizing
finding the underlying
visualizing
probability distribution

descriptive statistics

Definition: sample in with

sample in with

can be ordered:

visualization

histogram
(grouping)

subset in
Definition: For a given sample in and subset

we define:

absolute frequency of
abs

relative frequency of
rel

Definition: For a given sample in

we define the sample mean

really far off

from the other points

median: middle point of the sample same size same size

odd
odd
same size same size

even even

average

unbiased sample variance:

makes it unbiased