Multivariate transformations

The remaining part of the probability course is centered around

x
2
. p.1/21
Convolution of probability measures
Let and be probability measures on (R, B).
Consider the functions : R
2
R dened as
(x, y) = x +y for (x, y) R
2
Denition: The image measure
= ( )
is called the convolution of and .
Let Pr(R, B) be the set of probability measures on (R, B).
As an algebraic operation on Pr(R, B) the convolution, , is commutative
and associative.
. p.2/21
Interpretation
Let X and Y be two real valued stochastic variables dened on (, F, P).
Theorem: If X and Y are independent then
(X +Y )(P) = X(P) Y (P).
Proof: We use the denition and observe that
(X +Y )(P) =
_
(X, Y )
_
(P) =
_
(X, Y )(P)
_
=
_
X(P) Y (P)
_
= X(P) Y (P)
The distribution of a sum of two independent stochastic variables is the
convolution of their marginal distributions.
. p.3/21
Convolutions of discrete measures
Theorem: If X and Y are concentrated on Z with point probabilities
P(X = z) = p(z) , P(Y = z) = r(z) for all z Z
then X +Y is concentrated on Z with point probabilities
P(X +Y = z) =

m=
p(z m) r(m), for all z Z.
Proof: We use a classic; divide and conquer. We split the computation of
P(X +Y = z) according to the value of Y and use -additivity.
The probability functions is thus:
P(X +Y = z) =

m=
P(X = z m, Y = m)=

m=
p(z m) r(m).
. p.4/21
Convolution of binomials
Example: Let X and Y be independent and
X bin(n, p) , Y bin(m, p)
Then X +Y is concentrated on N
0
with point probabilities
p(i) =
i

j=0
_
n
i j
_
p
ij
(1 p)
ni+j
_
m
j
_
p
j
(1 p)
mj
= p
i
(1 p)
n+mi
i

j=0
_
n
i j
__
m
j
_
=
_
n +m
i
_
p
i
(1 p)
n+mi
for all i N
0
. That is,
X +Y bin(n +m, p)
. p.5/21
Convolution of Poisson distribution
Example: If X and Y are independent
X pois() , Y pois()
then X +Y is concentrated on N
0
with point probabilities
p(i) =
i

j=0

ij
(i j)!
e

j
j!
e

= ( +)
i
e
(+)
i!
i

j=0
_
i
j
__

+
_
ij
_

+
_
j
= ( +)
i
e
(+)
i!
_

+
+

+
_
i
=
( +)
i
i!
e
(+)
for all i N
0
. That is, X +Y is Poisson distributed with parameter +.
. p.6/21
Convolution from distribution functions
Theorem: Let X have distribution with distribution function F. Let Y
have distribution and distribution function G. If X and Y are independent
then X +Y have distribution function H given by
H(z) =
_

F(z y) d(y) =
_

G(z x) d(x).
Proof: Take z R. Then
H(z) = P(X +Y z) = (X, Y )(P)
_
{(x, y) R
2
| x +y z}
_
=
_
{(x, y) R
2
| x +y z}
_
.
Tonellis theorem gives that
H(z) =
_

_
{x R | x +y z}
_
d(y)=
_

F(z y) d(y)
. p.7/21
Convolution from densities
Corollary: Let X and Y be independent real stochastic variables. Assume
that
P(X A) =
_
A
f(x) dx , P(Y B) =
_
B
g(y) dy for all A, B B,
then X +Y has density h w.r.t. m, where
h(z) = f g(z) :=
_

f(z y)g(y) dy for z R

_
(, z]
_
=
_
z

h(x) dx =
_
z

f(x y)g(y) dy dx
=
_

_
z

f(x y) dxg(y) dy=

F(z y) d(y),
which is in fact the distribution function for X +Y .
. p.8/21
Convolution of normal distributions
Example: If X and Y are independent and if
X N(,
2
), Y N(,
2
)
then
X +Y N( +,
2
+
2
)
The important content: X +Y has a normal distribution.
By induction, if X
i
N(
i
,
i
2
) for i = 1, . . . , n are independent then
n

i=1
X
i
N
_
n

i=1

i
,
n

i=1

i
2
_
.
. p.9/21
Convolution of -distributions
Example: If X and Y are independent and if
X (, ), Y (, )
then
X +Y ( +, )
The important content: X +Y is -distributed.
By induction, if X
i
(
i
, ) for i = 1, . . . , n are independent then
n

i=1
X
i

_
n

i=1

i
,
_
.
. p.10/21
Sums of squares
Example: If X
1
, . . . , X
n
are independent N(0, 1)-distributed variables then
n

i=1
X
i
2

2
, df = n
This holds because X
1
2
, . . . , X
n
2
are independent
2
-distributed each
with 1 degree of freedom.
That is, they are -distributed with shape parameter 1/2 and scale
parameter 2.
. p.11/21
Transformation effects
A substantial part of probabilistic modeling consists of starting with certain
assumptions on independence and the derive via transformations a range
of more interesting/complicated distributions.
Bold claim: All dependence and every complicated distribution is created
by transformations from simple, independent variables.
. p.12/21
Regression model
Let Z, Y
1
, . . . , Y
n
,
1
, . . . ,
n
denote 2n + 1 independent, real stochastic
variables. Dene
X
i
= Z +Y
i
+
i
.
Observe that
Cov(X
i
, X
j
) = V Z,
thus the variables are dependent if Z is not degenerate.
We will in practice observe the Xs, perhaps the Y s but neither the Z nor
the s.
The Z represents an unobserved, shared component in all the n
measurements from the experiment and the s the unobserved, random
uctuations (noise).
. p.13/21
Tabulation
Let X
1
, . . . , X
n
be independent, identically distributed stochastic variables
with values in {1, . . . , N}. Assume
P(X
i
= j) = p
j
for j = 1, . . . , N
We regard the Xs as classication variables. Dene
Y
j
=
n

i=1
1
(X
i
=j)
for j = 1, . . . , N,
which is simply counting how many of the Xs that have the value j.
These Y s correspond to a tabulation of the Xs.
Note the Y
j
Bin(n, p
j
).
Challenge: What is the joint distribution of (Y
1
, . . . , Y
N
)?
. p.14/21
Tabulation
Construct the maps t
n
: {1, . . . , N}
n
N
0
N
as follows:
t
n
(x
1
, . . . , x
n
) =
_
n

i=1
1
{1}
(x
i
), . . . ,
n

i=1
1
{N}
(x
i
)
_
for (x
1
, . . . , x
n
) X
n
.
and dene
(Y
1
, . . . , Y
N
) = t
n
(X
1
, . . . , X
n
)
Remark: It always holds that
N

j=1
Y
j
= n.
Thus, if we know the distribution of the rst N 1 of the Y s we know the
last in particular, the Y s can impossibly be independent.
. p.15/21
Tabulation
Theorem: Let X = (X
1
, . . . , X
n
) be independent identically distributed
random variables, dened on (, F, P), with values in {1, . . . , N}. Assume
that
P(X
i
= j) = p
j
, j = 1, . . . , N, i = 1, . . . , n.
Let Y = (Y
1
, . . . , Y
N
) denote the tabulation of the Xs then
P(Y = y) =
n!

N
j=1
y
j
!
N

j=1
p
y
j
j
.
for y = (y
1
, . . . , y
N
) N
N
0
with

N
j=1
y
j
= n.
. p.16/21
Tabulation
Proof: With y = (y
1
, . . . , y
N
) we nd that
P(Y
1
= y
1
, . . . , Y
N
= y
N
) = P(t
n
(X) = y)
where
|t
n
1
(y)| =
n!

N
j=1
y
j
!
=
_
n
y
1
. . . y
N
_
.
. p.17/21
The multinomial distribution
Denition: The multinomial distribution on
S(N, n) = {(x
1
, . . . , x
N
) N
N
0
| x
1
+. . . +x
N
= n}
has point probabilities
p(x) =
_
n
x
1
. . . x
N
_
N

j=1
p
j
x
j
for x = (x
1
, . . . , x
N
) S(N, n).
From the multinomial formula we have that

xS(N,n)
p(x) = (p
1
+. . . +p
N
)
n
= 1.
. p.18/21
Re-labeling and collapsing
For any multinomial distribution we can always invent an underlying
sequence X
1
, . . . , X
n
of independent variables with values in {1, . . . , N}
whose tabulation gives this multinomial distribution.
If
s : {1, . . . , N} {1, . . . , K}
is surjective, this is a re-labeling and collapsing of the categories and
s(X
1
), . . . , s(X
n
) are independent. Dening
q
i
=

j:s(j)=i
p
j
, i = 1, . . . , K
we observe that this re-labeling and collapsing of a multinomial distribution
results in a multinomial distribution with parameters (n, q
1
, . . . , q
K
).
. p.19/21
Convolution
Same idea for convolution. If Y
1
and Y
2
are two multinomial distributions
with parameters (n, p
1
, . . . , p
N
) and (n

, p
1
, . . . , p
N
) let
X
1
, . . . , X
n
, X
n+1
, . . . , X
n+n
be independent, identically distributed with
values in {1, . . . , N} and
p(X
i
= j) = p
j
, j = 1, . . . , N.
Then Y
1
has the same distribution as the tabulation of X
1
, . . . , X
n
and Y
2
has the same distribution as the tabulation of X
n+1
, . . . , X
n+n
.
In conclusion, Y
1
+Y
2
has the same distribution as the tabulation of
X
1
, . . . , X
n+n
, thus a multinomial distribution with parameters
(n +n

, p
1
, . . . , p
N
).
. p.20/21
Covariance
If Y = (Y
1
, . . . , Y
N
) has a multinomial distribution with parameters
(n, p
1
, . . . , p
N
) then (using the representation as tabulation without even
mentioning it)
Cov(Y
i
, Y
j
) = Cov
_

k
1
{i}
(X
k
),

r
1
{j}
(X
r
)
_
=

k
P(X
k
= i, X
k
= j) P(X
k
= i)P(X
k
= j) = np
i
p
j
,
for i = j where we used independence of the Xs for the second equality.
For i = j we nd
V Y
i
= np
i
(1 p
i
).
The correlations become
corr(Y
i
, Y
j
) =
np
i
p
j
_
np
i
(1 p
i
)
_
np
j
(1 p
j
)
=
_
p
i
p
j
(1 p
i
)(1 p
j
)
.
. p.21/21