The mathematical procedures in PCA begin with computing the covariance matrix of the data, capturing variances within dimensions and covariances between them. Next, eigenvectors and eigenvalues of this matrix are calculated, with eigenvectors indicating the direction of maximum variance and eigenvalues quantifying their significance. The eigenvectors are then sorted based on their associated eigenvalues in descending order. The top k eigenvectors, corresponding to the largest eigenvalues, are chosen to form the new basis, defining the reduced-dimensionality space. Finally, the original dataset is projected onto this new basis, transforming the data into k dimensions while preserving critical variance information, thus achieving dimensionality reduction .

Zero covariance between dimensions in PCA indicates that the variables are linearly independent, meaning changes in one dimension do not predict or affect changes in another. This lack of linear relationship leads the transformed axes to be orthogonal, as orthogonal axes by definition regard dimensions that are uncorrelated. This relationship is significant because it ensures the transformed data space is aligned for maximum variance capture independently across each axis, reducing redundancy. Orthogonality simplifies analyzing multivariate data, making interpretation more intuitive and reducing noise .

The covariance matrix plays a central role in PCA as it quantifies the pairwise covariances between the dimensions of the dataset, revealing how variations in one dimension predict or affect another. To identify the principal components, PCA seeks a transformation of the original data such that the covariance matrix becomes diagonal, meaning all off-diagonal covariance terms become zero, indicating no linear correlation between dimensions. This is crucial because PCA aims to derive dimensions that are linearly independent, allowing for simpler and more interpretable data structures. The diagonal elements of this matrix, which are the variances of the transformed data, help in understanding the amount of variance captured by each principal component .

In PCA, eigenvectors represent the directions in which the data variance is maximized, meaning they indicate the principal axes of the transformed feature space. Eigenvalues, on the other hand, measure the amount of variance captured by each eigenvector, representing the importance or significance of each axis. The transformation process involves projecting the original data onto the space defined by the top k eigenvectors, which are selected based on their corresponding eigenvalues. This ensures that most of the variability in the dataset is retained while reducing its dimensionality .

The rationale for selecting only the top k eigenvectors in PCA is to focus on the dimensions that capture the most variance within the data, thereby maximizing information retention while minimizing dimensionality. This selection impacts data reconstruction by ensuring that the transformed data mostly preserves the structure of the original dataset, allowing for effective approximation with reduced complexity. However, choosing k requires balancing between retaining critical information and achieving simplification; too few dimensions may lead to significant information loss, while more dimensions may retain redundancies. This careful selection facilitates interpretation by highlighting the most influential dimensions without noise .

Not transforming the covariance matrix into a diagonal matrix before performing PCA means the dimensions remain correlated, inhibiting the independence needed for effective principal component analysis. This can lead to components that are not orthogonal, diminishing PCA’s ability to isolate the directions of maximum variance and causing redundant information to persist in the data. The final component structure may then inadequately reflect the true variability within the dataset, impairing interpretability and efficacy in model building or data compression .

Data normalization is important before applying PCA because it ensures that all features contribute equally to the analysis, particularly when they are measured on different scales. Not normalizing the data may lead PCA to be biased towards dimensions with larger ranges of values, thus skewing the component extraction towards those features and potentially producing misleading or erroneous principal components. By normalizing, each feature is on the same scale, ensuring the PCA identifies the dimensions that truly reflect the inherent structure and variance of the data .

If the dataset includes features of different scales, PCA can be adjusted by normalizing the data or by using a correlation matrix instead of a covariance matrix. Normalization scales all features to a common scale, preventing the PCA from favoring dimensions with inherently larger ranges and ensuring all dimensions contribute equally. Using a correlation matrix inherently accounts for differing scales by focusing on standardized covariance, underlining proportionate relationships over absolute ones. Such adjustments are necessary to avoid skewed representation of variance across dimensions, maintaining the robustness and accuracy of the PCA outcome .

The covariance matrix is symmetric in PCA because the covariance of any two dimensions is equal regardless of their order, i.e., cov(X,Y) = cov(Y,X). This symmetry implies that the eigenvectors of the covariance matrix form an orthogonal set and that it can be diagonalized, which is fundamental for transforming the original dataset into a set of uncorrelated principal components. This transformation helps in simplifying the data structure, leading to a more efficient representation with reduced dimensions while maintaining the maximum amount of variability .

The primary goal of PCA in data dimensionality reduction is to find a new set of dimensions, or principal components, that capture the maximum variance in the data while ensuring that these dimensions are orthogonal (linearly independent). PCA achieves this by calculating the covariance matrix of the data, then deriving its eigenvectors and eigenvalues. The eigenvectors, ranked by the magnitude of their corresponding eigenvalues in decreasing order, serve as the orthogonal axes of the transformed space. Each principal component is a linear combination of the original dimensions, aligned along directions of maximum variance, and are orthogonal because the eigenvectors are orthogonal, as per matrix properties .