Data Science and Statistical Learning

Comprehensive Table of Contents

1. Statistical Foundations and Probability
2. Exploratory Data Analysis (EDA)
3. Supervised Learning Regression
4. Classification Algorithms
5. Unsupervised Learning and Clustering
6. Feature Engineering and Selection
7. Model Evaluation and Cross-Validation
8. Ensemble Methods and Meta-Learning
9. Dimensionality Reduction
10. Bayesian Methods and Inference
11. Time Series Analysis and Forecasting
12. Production ML Systems and Monitoring

Chapter 1: Statistical Foundations

1.1 Probability Basics
Random Variables:

Definition:
�� Function mapping outcomes to numbers
�� Discrete: Finite or countable values
�� Continuous: Any value in interval
�� Example: Die roll (discrete), Height (continuous)

Probability Distribution:

Properties:
�� P(X=x) � 0 for all x
�� Sum of all probabilities = 1
�� P(a � X � b) = area under curve

Common Distributions:

Normal (Gaussian):
�� Bell curve shape
�� Mean �, standard deviation �
�� 68% within 1�, 95% within 2�
�� Most common in nature
�� Notation: X ~ N(�, �²)

1
Uniform:
�� Equal probability everywhere
�� Example: Dice roll (1/6 each)
�� Simple but rare in real data

Exponential:
�� Time until event
�� Memoryless property
�� Decay function

Poisson:
�� Count of events in fixed time
�� Example: Emails per hour
�� Rare events

Binomial:
�� Number of successes in N trials
�� Each trial: Success with probability p
�� Example: Coin flips

Statistical Moments:

Mean (Expected Value):

�� E[X] = Σ x·P(x)
�� Average value
�� Center of distribution

Variance:
�� Var(X) = E[(X - E[X])²]
�� Spread around mean
�� �² notation

Standard Deviation:
�� � = √Var(X)
�� Same units as X
�� More interpretable than variance

Skewness:
�� Asymmetry of distribution
�� Positive: Right tail
�� Negative: Left tail
�� Symmetric: Skewness = 0

Kurtosis:
�� Tail heaviness
�� High kurtosis: Heavy tails

2
�� Low kurtosis: Light tails

1.2 Hypothesis Testing

Null vs Alternative:

Null Hypothesis (H�):

�� Default assumption
�� "No effect" or "no difference"
�� Trying to reject it

Alternative Hypothesis (H�):

�� Contradicts null
�� "Effect exists"
�� Accept if evidence strong

Type I Error (False Positive):

�� Reject H� when actually true
�� Probability = � (significance level)
�� Example: COVID test positive, actually negative

Type II Error (False Negative):

�� Fail to reject H� when false
�� Probability = �
�� Example: COVID test negative, actually positive

Power:
�� Power = 1 - �
�� Probability of detecting effect if exists
�� Want power > 0.8

P-value:

Definition:
�� Probability of result under H�
�� If p < �, reject H�
�� Common �: 0.05 (5%)

Interpretation:
� p = 0.01: Very strong evidence against H�
� p = 0.05: Strong evidence against H�
� p = 0.10: Moderate evidence
� p > 0.05: Not enough evidence

Common Tests:

3
T-test:
�� Compare means of groups
�� Assumes normal distribution
�� Example: Is drug better than placebo?

Chi-square:
�� Categorical data
�� Independence test
�� Example: Are preferences gender-dependent?

ANOVA:
�� Compare 3+ groups
�� Extension of t-test
�� Example: Do 5 fertilizers differ?

Correlation:
�� Pearson: Linear relationship
�� Spearman: Rank correlation
�� Kendall: Another rank method
�� Range: -1 to +1

Causation vs Correlation:

Critical Distinction:
�� Correlation: Relationship exists
�� Causation: One causes the other
�� Don't confuse them!

Example:
�� Ice cream sales correlate with drowning deaths
�� Both caused by warm summer
�� No direct causation
�� Confounding variable: Temperature

Chapter 2: Exploratory Data Analysis

2.1 Univariate Analysis
Numerical Data:

Summary Statistics:
```python
import pandas as pd
import numpy as np

4
df = pd.read_csv('[Link]')

# Basic statistics
[Link]() # count, mean, std, min, 25%, 50%, 75%, max

# Specific metrics
df['age'].mean() # Central tendency
df['age'].median() # Robust average
df['age'].std() # Spread
df['age'].skew() # Asymmetry
df['age'].kurtosis() # Tail heaviness
Visualizations:
Histogram: �� Shows distribution shape �� Bin width affects appearance �� Identify
outliers, skewness
Box Plot: �� Shows quartiles and outliers �� Median, Q1, Q3 �� Whiskers and
outlier points �� Compare distributions
Violin Plot: �� Kernel density estimate �� Shows full distribution shape �� Better
than box plot for complex distributions
Categorical Data:
Value Counts:
df['category'].value_counts()
df['category'].value_counts(normalize=True) # Proportions
Bar Chart: �� Frequency of each category �� Sorted or unsorted �� Identify domi-
nant categories
Pie Chart: �� Proportion visualization �� Limited to ~5 categories �� Less preferred
than bar chart
Missing Data:
Assessment:
[Link]().sum() # Count missing per column
[Link]().sum() / len(df) # Proportion missing

# Visualize
import missingno as msno
[Link](df)
[Link](df)
Handling:
Delete: �� Remove rows with missing �� Simple but loses data �� Only if missing
< 5%

5
Imputation: �� Mean/median for numerical �� Mode for categorical �� Advanced:
KNN imputation, model-based
Forward/Backward Fill: �� Time series data �� Use last/next value �� Preserves
temporal structure

### 2.2 Multivariate Analysis

Correlation Analysis:
Correlation Matrix:
[Link]() # Pearson correlation all pairs

# Visualization
import [Link] as plt
import seaborn as sns

[Link]([Link](), annot=True, cmap='coolwarm')

Correlation Types:
Strong Positive (r > 0.7): �� Variables increase together �� Example: Height and
weight
Moderate Positive (0.3 < r < 0.7): �� Some positive relationship �� Example:
Study hours and grades
Weak (|r| < 0.3): �� Little relationship �� Example: Shoe size and IQ
Multicollinearity:
Definition: �� High correlation between predictors �� Inflates regression coeffi-
cients �� Makes interpretation difficult
Detection: �� Correlation > 0.8 suspicious �� VIF (Variance Inflation Factor) >
5 problematic �� Condition number analysis
Remedies: �� Drop one variable �� Combine variables (PCA) �� Regularization
(Ridge/Lasso)
Dimensionality Reduction:
Purpose: �� Reduce number of features �� Visualize high-dimensional data ��
Remove noise �� Speed up models
Principal Component Analysis (PCA): �� Find directions of maximum variance
�� Project data onto principal components �� Example: 100 features → 10 com-
ponents
t-SNE: �� Non-linear dimensionality reduction �� Good for visualization �� Pre-
serves local structure �� Slow for large datasets
UMAP: �� Faster than t-SNE �� Better global structure preservation �� Scalable

6
Bivariate Plots:
Scatter Plot: �� X vs Y relationship �� Pattern reveals relationship �� Color by
group for multivariate
Hexbin Plot: �� Density visualization �� Better for large datasets (overplotting
problem) �� Shows concentration areas
Contour Plot: �� 2D density contours �� Like topographic map �� Reveals multi-
modal distributions

---

## Chapter 3: Regression Analysis

### 3.1 Linear Regression

Simple Linear Regression:
Model: �� ŷ = �� + ��x �� ��: Intercept �� ��: Slope
Least Squares Estimation:
Minimize: �� SSE = Σ(y - ŷ)² �� Finds best-fit line �� Closed-form solution exists
from sklearn.linear_model import LinearRegression

X = df[['feature']].values
y = df['target'].values

model = LinearRegression()
[Link](X, y)

# Coefficients
print(f"Intercept: {model.intercept_}")
print(f"Slope: {model.coef_}")

# Predictions
y_pred = [Link](X)
Evaluation:
R² (Coefficient of Determination): �� Proportion of variance explained �� Range:
0 to 1 �� 0.8 is good, 0.9 is excellent �� Formula: 1 - (SSE / SST)
RMSE (Root Mean Square Error): �� √(Σ(y - ŷ)² / n) �� Same units as y ��
Penalizes large errors
MAE (Mean Absolute Error): �� Σ|y - ŷ| / n �� Less sensitive to outliers �� Easier
to interpret
Assumptions:

7
Linearity: �� Relationship is linear �� Check scatter plot �� If curved, consider
transformation
Homoscedasticity: �� Constant variance of errors �� Errors don’t fan out �� Plot
residuals vs fitted
Normality: �� Errors normally distributed �� Q-Q plot check �� Histogram of
residuals
Independence: �� Observations independent �� No serial correlation �� Durbin-
Watson test
Multiple Linear Regression:
# Multiple features
X = df[['feat1', 'feat2', 'feat3']]
y = df['target']

model = LinearRegression()
[Link](X, y)

# Interpretation
for i, col in enumerate([Link]):
print(f"{col}: {model.coef_[i]}")
# Coefficient = change in y per unit change in x
Challenges:
Multicollinearity: �� Correlated features �� Unstable coefficients �� Solution: Fea-
ture selection or regularization
Overfitting: �� Too many features �� Fits noise in data �� Solution: Regularization
(Ridge/Lasso)
Non-linear Relationships: �� Linear model insufficient �� Try polynomial features
�� Or use non-linear models

### 3.2 Advanced Regression

Polynomial Regression:
Idea: �� ŷ = �� + ��x + ��x² + ��x³ + … �� More flexible than linear �� Still linear
in parameters
from [Link] import PolynomialFeatures

poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

model = LinearRegression()
[Link](X_poly, y)

8
Caution: �� Higher degree = more overfitting risk �� Visual inspection important
�� Cross-validation to find optimal degree
Ridge Regression:
Idea: �� Add penalty for large coefficients �� Shrinks coefficients toward zero ��
Handles multicollinearity
Loss Function: �� SSE + �·Σ(�²) �� �: Regularization strength �� Higher � = more
shrinkage
Lasso Regression:
Idea: �� Add L1 penalty instead of L2 �� Can shrink some coefficients to exactly
zero �� Feature selection built-in
Loss Function: �� SSE + �·Σ|�| �� Sparse solution (many zeros) �� Good for
high-dimensional data
Ridge vs Lasso:
Ridge: �� All features retained �� Good for correlated features �� Continuous
shrinkage
Lasso: �� Some features dropped �� Good for feature selection �� Sparse solutions
Elastic Net: �� Combines Ridge and Lasso �� Balance between two approaches ��
Most flexible
Robust Regression:
Purpose: �� Handles outliers better �� Less sensitive to extreme values �� Alter-
native to OLS
Methods: �� Huber regression �� Quantile regression (median instead of mean) ��
RANSAC: Random sample consensus
Quantile Regression:
Idea: �� Predict quantiles, not just mean �� Example: 10th percentile, median,
90th percentile �� More information about distribution �� Robust to outliers
Use Cases: �� Insurance: Claims at different percentiles �� Growth curves: Dif-
ferent rates �� Risk assessment: Downside scenarios “‘

Chapters 4-12 (Abbreviated)

[Continued sections on Classification, Clustering, Feature Engineering, Model
Evaluation, Ensemble Methods, Dimensionality Reduction, Bayesian Methods,
Time Series, and Production ML - maintaining same detailed technical pattern]

9
Conclusion
Data science combines statistics, machine learning, and domain knowledge. Un-
derstanding the statistical foundations ensures robust models and correct inter-
pretation.
Key takeaways: - Probability underpins everything - EDA reveals data structure
- Linear regression baseline - Assumptions matter - Feature engineering critical -
Ensemble methods powerful - Cross-validation prevents overfitting - Time series
different - Production requires monitoring - Interpretability important - Domain
knowledge essential - Ethical considerations
Data science is art and science - balance rigor with practical intuition.

10