Statistics:

No more than 10 pages for the project. The techniques used must be explained
at a theoretical level.

Many interrelated data = high dimensional data

We want to analyse the relationship among these variables, we do not consider

the relationship between each and every variable separately, rather we
consider them jointly.

The trace of a matrix is the sum of its diagonal elements.

GAUSSIAN RANDOM VECTORS:

The likelihood function approach allows us to estimate the values of the mean
and the variance?

The maximum likelihood estimate of the covariance matrix of a normal

distribution is the biased covariance matrix.

PRINCIPAL COMPONENT ANALYSIS:

Technique used to reduce the dimensionality of the data. We consider linear

combinations of variables in such a way as to find new variables that explain as
much variability in the data as possible. These new variables will be
uncorrelated amongst themselves.

The sum of the variability of all components will have to be equal to the sum of
the variability of initial variables.

If we sum two variables that are correlated, their combined variance will be
higher than the variance of each. Vice-versa if we subtract them.

Uncorrelated variables are orthogonal to one another. We want to have a

principal component such that the mean squared error is minimised (this
ensures maximal variability, we don’t study the proof). The mean squared error
is the mean squared distance between each observation and the principal
component.

The component will lie closer to the axis where the most variance is found. The
higher the correlation between X1 and X2, the better the component will work
(it will explain more variance). In other words, the variance of the first principal
component is high if the original variables are highly correlated and this means
that the PCA is working well.

Even with more dimensions, each component is orthogonal to the other, but we
can’t visualise it.

In doing the PCA, we need to be careful about unit of measurement. If different

variables have different units of measurement, their variance will not be
comparable and the PCA technique will be distorted. In such a case we need to
standardise the variables (this implies using correlations instead of
covariances).

PCA theoretical framework:

The weights are typically called “loadings”. The sum of squared loadings of a
certain component has to be equal to one, and the sum of the products of the
loading has to be equal to zero (loadings are uncorrelated).

The method to find the loadings involves finding the eigenvalues and the
eigenvectors of the covariance (or correlation) matrix. The principal component
can be expressed as the product between the transpose of the vector of
loadings and the vector of variables (we call it capital X).

See the addendum on Virtuale for the mathematics. The covariance matrix of
the principal components can be expressed as a function of the original
covariance matrix.

The variance of the first principal component is: eta(1) ‘ S eta(1), where eta(1)
is the vector of loadings for the first component and S is the original covariance
matrix.

To this we need to add the constraint that the sum of the squared loadings is
one. In matrix form, this requires eta(1)’ * eta(1) to be equal to one.

So now we have a function to maximise (the variance of the principal

component) and a constraint, we set up a Lagrangian to solve the problem. I
get a linear equation and the solutions to this linear equation are the
eigenvalues and the eigenvectors of the original matrix S. The Lagrangian
multiplier is the eigenvalue, and this is equal to the variance of the principal
component. The eigenvector is the vector of loadings.

The first eigenvalue of the original covariance matrix is the variance of the first
principal component; the first eigenvector is the vector of loadings.

In the project, we must aim to choose components that have a meaningful

interpretation (not easy to do).

17.02.2026

High dimensional data imply a larger number of variables but still a larger
number of observations. High-dimension low sample size data (HDLSS), more
variables than observations, we focus on these situations.

Illicit drug market example:

Variables are the number of months, for each month we have 17 observations
(which represent the various things we observe: heroin addiction, crimes, etc.).
So, conclusion: there is an epidemic cycle, but it is not the most relevant factor
in explaining the decline. The number of components extracted can’t be larger
than the number of observations.

FACTOR ANALYSIS:

Factor analysis is used when both the latent variable (income) and the
observed variables (consumption) are continuous.

If the covariance amongst observed variables conditional on the latent variable

of interest is not zero, we can add additional latent variables. The important
things it that the number of factors, or latent variables, is much smaller than
the number of observations.

If we want to use the MLE we need to assume the distribution of the latent
variables and the unique factors (errors), in particular we assume it is normal.

Matrix Sigma contains the lambdalambda’ matrix that has the variances on the
diagonal (communalities), outside of the main diagonal there are the
covariances among observed variables + the psi matrix. If in the psi matrix the
elements outside of the main diagonal are not zero, it may be necessary to add
more factors.

In factor analysis, starting from the correlation matrix instead of the covariance
matrix doesn’t change the end results (differently from what happened with the
PCA).

The R command assumes that the variables are normally distributed. Compare
the values of the likelihood function at each step to check that the command
worked, if the difference between the last step and the one before is very
small, the solution is stable and the operation was performed correctly.

18.02.2026

It is necessary that the observed variables be correlated, otherwise factor

analysis can’t be used. Overidentification is preferable, because it allows for
dimensionality reduction.

For orthogonal matrices, the transpose is equal to the inverse. Consider a

matrix v that is an orthogonal transformation of the original matrix f. This
transformed matrix is normally distributed with mean zero and variance I
(identity matrix). The proof is uncomprehensible.

Λ∗ = ΛG′ is the transformed matrix of loadings.

The bottom line is that rotating the matrix of loadings doesn’t change anything,
we always end up with the same covariance matrix. In other words, we have
infinite possible solutions that change only because we rotate the matrix.
Different people may get different factor models because of rotation, but they
are all equivalent, there is no unique solution. We should choose the solution
that is easier to interpret.

The factor score is the value of the factor. The aim of factor analysis is to
explain the correlation among observed variables, this is different from PCA
which aims to explain as much of the total variance as possible with linear
combinations of variables.

NEVER ROTATE WHEN DOING THE PCA.

Example:

We need to decide how many factors to introduce in the model. In the example,
first of all, we observe that correlations are positive. Even a correlation of 0.3 is
not bad, so we start by adding a singles factor (it may be enough as all
variables are correlated amongst each other). Let us then apply a test to
evaluate the goodness of fit. The null hypothesis is that the theoretical
covariance matrix is well decomposed in to the usual form. The assumption is
that one factor is enough to explain all the correlation among variables.

The test is a likelihood ratio test, the numerator is the value under the null
hypothesis and the denominator is the value under the alternative hypothesis
(this is more or less the idea).

W = log likelihood under the null hypothesis / log likelihood under the
alternative hypothesis = likelihood of the decomposed sigma / likelihood of the
sample sigma. The higher the value we get, the better our model perform.

Unfortunately, the value of w we get with just one factor is too small, so we add
additional factors and we re-estimate the model.

In the example, the overall variance is equal to the variance of each variable
(which is equal to one because they are standardised), so we get a variance of
7.

There is a mistake in the slide on the reproduced correlation matrix.

In this case it makes sense to use oblique transformations, as political

democracy is likely correlated with industrialisation.

Confirmatory factor models: they impose constraints on the loadings (e.g. by

imposing that some be equal to zero). The solution we obtain is unique, we
cannot rotate it. Among the constraints, we have that one loading is fixed to
one (it is used as an anchoring measure). Furthermore, the loadings
corresponding to the same variables over time must be equal in time (this is
important for panel data and longitudinal analysis).

In factor analysis we can’t estimate 15 factors and then choose for example
two, the solution is not the same as fitting two factors to start with. So the
procedure is different from PCA. What we must do is add one factor at a time to
decide.

DISCRIMINANT ANALYSIS:

Discriminant analysis is a statistical technique employed to classify

observations into different classes. The procedure starts with an initial set of
observations whose class is already known; from the analysis of these
observations it is then possible to extract information on the characteristics of
the observations in each class. The procedure requires the use of discriminant
rules which best capture differences between classes.

Consider a random variable that takes on different values based on a

probability density f(x), and whose realisations can be classified into k different
classes depending on the value of x. A discriminant rule tries to divide the set
of values that x can take into k disjoint regions, which correspond to the classes
in which the random variable is defined.

The main question is to determine in which region the data x falls into.

The discriminant methods we decide to use depend on three different

conditions:

1. The distribution f(x) of the d-dimensional random vector X is completely

known.
2. The parameters of f(x) are unknown.
3. The distribution of f(x) is completely unknown.

We will concentrate on cases 1 and 2. We will also assume that the groups in
which the observations are divided are known a priori.

Assume that f1(x) and f2(x) are the known distributions of x in groups C1 and
C2, for example x could represent the number of years spent in education and
C1 and C2 could represent respectively Italian citizens and foreign residents in
Italy. It is likely that the two probability distributions overlap in some areas, that
is because there will be Italian citizens and foreign residents with more or less
the same years of education. Because of this fact, each point of the domain of
X can belong to both C1 and C2, with different probabilities.

The goal is to determine a partition of R = R1 ∪ R2 such that if the unit belongs

to region R1 it is assigned to C1 and if the unit belongs to region R2 it is
assigned to C2. The partition is obtained in such a way that the probability of
misclassification is minimised.

It is possible to prove that P(R) is minimised by choosing R1

such that:

The left-hand side of this formula is the likelihood ratio; it tells us how much
more compatible is x with class 1 than with class 2. The right-hand side on the
other hand is the ratio between the probability that an observation belongs to
class 2 and the probability that an observation belongs to class 1 a priori
(without having seen the realisation of the random variable). Intuitively, if π 1 is
very small, then the ratio on the right will be large, implying that a lot of
evidence will be needed to assign the observation to class one (a very large
likelihood ratio will be necessary).

If π 1 and π 2 are unknown or equal, the decision rule becomes:

This decision rule is

called the likelihood ratio rule.

Linear discriminant analysis is the application of the technique explained thus

far to situations in which the distributions are known to be normal. In this case,
assuming homoskedasticity, we obtain a formula that looks as such:

With h(x) being the discrimination function. If h(x) is larger than zero, the
observation is allocated in class 1, whereas if h(x) is smaller than zero, the
opposite occurs. Since the parameters of the model are unknown, the means
are estimated using the sample means and the covariance matrix is estimated
using the sample covariance matrix.

If homoskedasticity doesn’t hold, the discrimination rule becomes:

This is called quadratic discriminant analysis, and the decision rule is

unchanged.

If we know that the classes are Gaussian, as long as the estimates of the
means and covariance matrices are accurate enough, the quadratic
discriminant analysis method and the Bayesian methods are equivalent.

Fisher’s Linear Discriminant Analysis:

The key idea is to partition the data in such a way that the variability within
classes is as small as possible and the variability across classes is as large as
possible. Suppose we observe two samples of size n1 and n2 with n = n1 + n2
from C1 and C2 that are supposed to be homoscedastic. The central idea is to
find a new variable Y that is a linear combination of the X that maximizes the
distance between the two populations.

This is similar to the PCA, but here rather than looking for the coefficients that
maximise the variance of the component, we look for the coefficients that
minimise within-class variance and maximise across-class variance.

The ideal vector gamma is found by maximising the ratio between the between
variability and the within variability, this leads to the identification of the
following formula:

Where W is the covariance

matrix.

This is
called Fisher’s discriminant rule. Example: dd
CLUSTER ANALYSIS:

Classification technique.

Appropriability effect

Intertemporal spillover effect: the private obsolescence-adjusted interest rate

in the decentralised equilibrium is higher than the social one.

Business stealing effect: innovators don’t internalise the welfare cost that they
cause to incumbents through their innovation

Don’t do a cluster analysis to assign labels and then use discriminant analysis.
With discriminant analysis, the labels have to be 100% true and cluster
analysis doesn’t provide truth.

IMPORTANT:

When interpreting the principal components, use the standardised scores! Not
just the eigenvalues.

In confirmatory factor analysis, only keep scores of the same sign!!

Use “lavaan” to add points(?)

Explain the methods with formula as well, really explain the theory in detail.

Explain the tests, the null hypothesis.

18.03.2026

BANK MARKETING:

Predict whether clients subscribe after the marketing campaign.

PENN WORLD TABLE (cross sectional):

Similar to one of the datasets done at the labs.

PENN WORLD TABLE (Italy, time series):

Also similar to one dataset seen in the labs.

WORLD DEVELOPMENT INDICATORS DATASET(2022):

Region and income specification may be used to filter the data into more
relevant groups.

WORLD DEVELOPMENT INDICATORS DATASET (2024):

Smaller dataset, more filtering was done.