CSE4103: ADVANCED MACHINE LEARNING

Comprehensive Study Guide with Code Snippets

By: Professor with 10+ years of AI/ML Experience

TABLE OF CONTENTS

1. Unit 1: Dataset Analysis

2. Unit 2: Feature Extraction

3. Unit 3: Feature Selection and Dimensionality Reduction

4. Unit 4: Machine Learning Algorithms

5. Unit 5: Neural Networks and Deep Learning

UNIT 1: DATASET ANALYSIS

1.1 Data Collection: Primary and Secondary Data

Definition: Primary data is collected directly from sources for a specific research purpose, while secondary data
comes from existing sources like databases, publications, and archives.

Key Differences:

Primary: Original, precise, current but time-consuming and expensive

Secondary: Quick, cost-effective but may lack relevance

Primary data includes surveys, experiments; secondary includes published reports, statistics
 import pandas as pd
import numpy as np

# Simulating primary data collection from survey

def collect_primary_data():
survey_responses = {
'age': [Link](18, 65, 100),
'income': [Link](20000, 150000, 100),
'satisfaction': [Link](1, 10, 100)
}
df_primary = [Link](survey_responses)
return df_primary

# Using secondary data from existing sources

df_secondary = pd.read_csv('existing_dataset.csv')

1.2 Processing and Analysis of Data

Data Processing Steps: cleaning, transformation, normalization, handling missing values, outlier detection

Importance: Raw data contains inconsistencies, duplicates, and errors requiring preprocessing before analysis.

import pandas as pd
from [Link] import StandardScaler
from [Link] import SimpleImputer

# Data cleaning
df = pd.read_csv('[Link]')
df.drop_duplicates(inplace=True)

# Handle missing values

imputer = SimpleImputer(strategy='mean')
df['numeric_col'] = imputer.fit_transform(df[['numeric_col']])

# Normalize data
scaler = StandardScaler()
df_scaled = scaler.fit_transform(df)

print(f"Data shape: {[Link]}, Missing: {[Link]().sum()}")

 1.3 Measurement of Relationships: Correlation and Covariance
Correlation: Measures strength and direction of linear relationship (-1 to 1). Dimensionless, standardized
metric.

Covariance: Measures joint variability of two variables. Affected by scale; harder to interpret.

Formula: Correlation = Covariance(X,Y) / (σ_X * σ_Y)

import numpy as np
import pandas as pd

# Calculate correlation and covariance

data = {'X': [1, 2, 3, 4, 5],
'Y': [2, 4, 5, 4, 6]}
df = [Link](data)

correlation = df['X'].corr(df['Y'])
covariance = [Link]().iloc[0, 1]

print(f"Correlation: {correlation:.3f}")
print(f"Covariance: {covariance:.3f}")

# Correlation matrix for multiple variables

corr_matrix = [Link]()

1.4 Statistical Measurement and Significance

Significance Testing: Hypothesis testing determines if observed differences are statistically significant or due to
chance.

P-value < 0.05 typically indicates statistical significance. Uses null and alternative hypotheses.

from scipy import stats

# T-test for comparing two groups

group1 = [Link](100, 15, 100)
group2 = [Link](105, 15, 100)
 t_stat, p_value = stats.ttest_ind(group1, group2)
print(f"T-statistic: {t_stat:.4f}, P-value: {p_value:.4f}")

# Chi-square test for categorical data

from [Link] import chi2_contingency
contingency_table = [Link]([[10, 20], [30, 40]])
chi2, p, dof, expected = chi2_contingency(contingency_table)
print(f"Chi-square: {chi2:.4f}, P-value: {p:.4f}")

1.5 Random Sampling

Definition: Each element has equal probability of selection. Simple, unbiased but inefficient for heterogeneous
populations.

Advantage: Simple, unbiased. Disadvantage: May not represent all subgroups well.

import pandas as pd
import numpy as np

# Create dataset
data = [Link]({'id': range(1000),
'value': [Link](1000)})

# Random sampling
sample_size = 50
random_sample = [Link](n=sample_size, random_state=42)
print(f"Sampled {len(random_sample)} items randomly")

# Stratified vs Random comparison

print(f"Random mean: {random_sample['value'].mean():.4f}")

1.6 Systematic Sampling

Method: Select every kth element from ordered population. k = N/n (population/sample size).

Advantage: Simpler than random, ensures spread. Disadvantage: Vulnerable to patterns in data.
 # Systematic sampling
N = 1000 # Population size
n = 50 # Sample size
k = N // n # Sampling interval

systematic_sample = [Link][::k]
print(f"Systematic sample size: {len(systematic_sample)}")

1.7 Stratified Sampling

Method: Divide population into homogeneous strata, sample from each stratum proportionally or equally.

Advantage: Captures all subgroups, reduces variance. Disadvantage: Requires prior knowledge of strata.

# Stratified sampling
# Assume 'category' column defines strata
stratified_sample = [Link]('category',
group_keys=False).apply(
lambda x: [Link](frac=0.1))
print(f"Stratified sample preserves category distribution")

1.8 Cluster Sampling

Method: Divide population into clusters, randomly select clusters, include all elements from selected clusters.

Advantage: Cost-effective for geographic populations. Disadvantage: Higher sampling error than simple
random.

# Cluster sampling
# Assume 'region' column defines clusters
clusters = data['region'].unique()
selected_clusters = [Link](clusters, size=5, replace=False)
cluster_sample = data[data['region'].isin(selected_clusters)]
print(f"Selected {len(selected_clusters)} clusters, sample size: {len(cluster_sample
 1.9 Multistage Sampling
Method: Hierarchical approach with multiple sampling stages. First stage: select clusters, second stage:
sample within clusters.

Advantage: Combines benefits of stratification and clustering. Practical for complex surveys.

# Multistage sampling
# Stage 1: Select regions
regions = data['region'].unique()
selected_regions = [Link](regions, size=3, replace=False)

# Stage 2: Within each region, select districts

# Stage 3: Within each district, select individuals
data_region_filtered = data[data['region'].isin(selected_regions)]
multistage_sample = data_region_filtered.groupby('region',
group_keys=False).apply(
lambda x: [Link](frac=0.1, random_state=42))
print(f"Multistage sample size: {len(multistage_sample)}")

UNIT 2: FEATURE EXTRACTION

2.1 Introduction to Feature Extraction

Definition: Process of identifying and extracting relevant features (attributes) from raw data that contribute to
predictive power.

Importance: Good features lead to better model performance, reduce dimensionality, and improve
interpretability.

# Feature extraction example: Text features

from sklearn.feature_extraction.text import TfidfVectorizer

texts = ["machine learning is great", "deep learning uses neural networks"]

vectorizer = TfidfVectorizer()
text_features = vectorizer.fit_transform(texts)
print(f"Extracted {text_features.shape[1]} text features from {text_features.shape[0
 2.2 Feature Extraction Process in Machine Learning
Steps: Data collection → Raw data representation → Feature computation → Feature normalization → Feature
selection

Role in Pipeline: Bridges raw data and machine learning models. Determines what information model can learn
from.

from [Link] import StandardScaler

from [Link] import PCA

# Complete extraction pipeline

data = [Link](100, 20) # 100 samples, 20 raw features

# Normalize
scaler = StandardScaler()
data_normalized = scaler.fit_transform(data)

# Extract principal components

pca = PCA(n_components=5)
features_extracted = pca.fit_transform(data_normalized)
print(f"Extracted {features_extracted.shape[1]} features from {[Link][1]} raw at

2.3 Histogram of Oriented Gradients (HOG)

Method: Extracts edge orientations as features. Divides image into cells, computes gradient orientations in
each cell.

Advantage: Effective for pedestrian detection, robust to illumination changes. Disadvantage: Computationally
expensive.

from [Link] import hog

from skimage import io
import [Link] as plt

# Load image
image = [Link]('[Link]', as_gray=True)

# Compute HOG features

 fd, hog_image = hog(image, orientations=9, pixels_per_cell=(8, 8),
cells_per_block=(2, 2), visualize=True)

print(f"HOG feature vector size: {len(fd)}")

2.4 Scale-Invariant Feature Transform (SIFT)

Method: Detects keypoints invariant to scale and rotation. Extracts local descriptors around keypoints.

Advantage: Robust to rotation, scale, illumination. Disadvantage: Complex, patented (use SURF alternative).

import cv2

# SIFT feature extraction

image = [Link]('[Link]')
gray = [Link](image, cv2.COLOR_BGR2GRAY)

sift = cv2.SIFT_create()
keypoints, descriptors = [Link](gray, None)

print(f"Detected {len(keypoints)} keypoints")

print(f"Each keypoint has {[Link][1]}-dimensional descriptor")

2.5 CNN for Feature Extraction

Method: Convolutional layers learn hierarchical features: low-level (edges) → mid-level (shapes) → high-level
(objects).

Advantage: Automatic feature learning, context-aware, state-of-the-art for images. Disadvantage: Requires
large datasets.

import tensorflow as tf
from tensorflow import keras

# Pre-trained CNN for feature extraction

base_model = [Link].VGG16(weights='imagenet',
include_top=False,
 input_shape=(224, 224, 3))

# Extract features using intermediate layer

image_input = [Link](shape=(224, 224, 3))
features = base_model(image_input)
features_flat = [Link]()(features)

feature_extractor = [Link](inputs=image_input, outputs=features_flat)

print(f"VGG16 extracts {features_flat.shape[1]} features")

UNIT 3: FEATURE SELECTION AND

DIMENSIONALITY REDUCTION

3.1 Feature Selection vs Feature Extraction

Feature Selection: Choose subset of original features. Interpretable, faster, prevents overfitting.

Feature Extraction: Create new features from original ones. Better representation but less interpretable.

Use selection when original features are meaningful; extraction when creating combinations improves
performance.

from sklearn.feature_selection import SelectKBest, f_classif

from [Link] import PCA

X = [Link](100, 20)
y = [Link](0, 2, 100)

# Feature selection: keep top 10 features

selector = SelectKBest(score_func=f_classif, k=10)
X_selected = selector.fit_transform(X, y)

# Feature extraction: PCA creates new features

pca = PCA(n_components=10)
X_extracted = pca.fit_transform(X)

print(f"Selection: {X_selected.shape}, Extraction: {X_extracted.shape}")

 3.2 Filter Methods
Approach: Score features independently using statistical tests. Fast, independent of model.

Methods: Correlation, mutual information, chi-square, ANOVA.

Limitation: Ignores feature interactions, high false positives.

from sklearn.feature_selection import SelectKBest, chi2, f_classif

# Filter method using ANOVA F-statistic

X = [Link](100, 20)
y = [Link](0, 2, 100)

# Score features
selector = SelectKBest(score_func=f_classif, k=5)
X_filtered = selector.fit_transform(X, y)

# Get selected feature indices

selected_indices = selector.get_support(indices=True)
print(f"Selected features: {selected_indices}")

3.3 Wrapper Methods

Approach: Evaluate subsets using actual model performance. Searches different feature combinations.

Methods: Forward selection, backward elimination, recursive feature elimination.

Advantage: Accounts for feature interactions. Disadvantage: Computationally expensive.

from sklearn.feature_selection import RFE

from [Link] import SVC

# Recursive Feature Elimination

X = [Link](100, 20)
y = [Link](0, 2, 100)

estimator = SVC(kernel="linear")
rfe = RFE(estimator=estimator, n_features_to_select=5, step=1)
X_rfe = rfe.fit_transform(X, y)
 print(f"RFE selected features: {[Link](rfe.support_)[0]}")

3.4 Embedded Methods

Approach: Feature selection during model training. Model learns importance weights.

Methods: L1 regularization (Lasso), tree-based feature importance, elastic net.

Advantage: Fast, considers feature interactions. Disadvantage: Model-specific.

from sklearn.linear_model import Lasso

from [Link] import RandomForestClassifier

# L1-based feature selection

X = [Link](100, 20)
y = [Link](0, 2, 100)

lasso = Lasso(alpha=0.01)
[Link](X, y)
selected = [Link](lasso.coef_ != 0)[0]
print(f"Lasso selected {len(selected)} features")

# Tree-based importance
rf = RandomForestClassifier(n_estimators=100)
[Link](X, y)
importance = rf.feature_importances_
top_features = [Link](importance)[-5:]
print(f"Top 5 important features: {top_features}")

3.5 Principal Component Analysis (PCA)

Method: Orthogonal transformation to uncorrelated principal components. Maximizes variance.

Process: Compute covariance matrix → Eigenvectors/eigenvalues → Project data onto top components.

Advantage: Reduces dimensionality, noise reduction. Disadvantage: Loss of interpretability.

 from [Link] import PCA
from [Link] import StandardScaler

X = [Link](100, 20)

# Standardize data
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply PCA
pca = PCA(n_components=5)
X_pca = pca.fit_transform(X_scaled)

print(f"Explained variance ratio: {pca.explained_variance_ratio_}")

print(f"Cumulative variance: {[Link](pca.explained_variance_ratio_)}")

3.6 t-SNE and UMAP for Visualization

t-SNE: Non-linear dimensionality reduction. Preserves local neighborhood structure. Good for visualization.

UMAP: Faster than t-SNE, preserves both local and global structure. Better for larger datasets.

from [Link] import TSNE

import umap

X = [Link](1000, 50)

# t-SNE (slow but effective visualization)

tsne = TSNE(n_components=2, perplexity=30, random_state=42)
X_tsne = tsne.fit_transform(X)

# UMAP (faster, scalable)

reducer = [Link](n_components=2)
X_umap = reducer.fit_transform(X)

print(f"t-SNE output shape: {X_tsne.shape}")

print(f"UMAP output shape: {X_umap.shape}")
 3.7 Evaluating Feature Selection
Metrics: Model accuracy with selected features, stability across data splits, computational efficiency.

Methods: Cross-validation, comparison with baseline, ablation studies.

from sklearn.model_selection import cross_val_score

from [Link] import SVC

X = [Link](100, 20)
y = [Link](0, 2, 100)

# Evaluate with different feature subsets

for n_features in [5, 10, 15, 20]:
selector = SelectKBest(score_func=f_classif, k=n_features)
X_selected = selector.fit_transform(X, y)

model = SVC()
scores = cross_val_score(model, X_selected, y, cv=5)
print(f"Features: {n_features}, Mean CV Score: {[Link]():.4f}")

UNIT 4: MACHINE LEARNING ALGORITHMS

4.1 Machine Learning Overview

Types:

Supervised: Regression (continuous output), Classification (discrete output)

Unsupervised: Clustering, dimensionality reduction

Reinforcement: Learning from rewards/penalties

ML Pipeline: Data collection → Preprocessing → Feature engineering → Model selection → Training →

Evaluation → Deployment

from [Link] import Pipeline

from [Link] import StandardScaler
from [Link] import SVC
 # ML pipeline
ml_pipeline = Pipeline([
('scaler', StandardScaler()),
('classifier', SVC())
])

X_train, X_test = [Link](80, 10), [Link](20, 10)

y_train, y_test = [Link](0, 2, 80), [Link](0, 2, 20)

ml_pipeline.fit(X_train, y_train)
accuracy = ml_pipeline.score(X_test, y_test)
print(f"Model accuracy: {accuracy:.4f}")

4.2 Overfitting and Underfitting

Overfitting: Model learns training data too well including noise. High training accuracy, low test accuracy.

Underfitting: Model too simple to capture patterns. Poor training and test accuracy.

Solutions: Regularization, more data, simpler models, cross-validation.

from sklearn.model_selection import learning_curve

from sklearn.linear_model import LogisticRegression

X = [Link](1000, 20)
y = [Link](0, 2, 1000)

model = LogisticRegression()

# Plot learning curves to diagnose overfitting/underfitting

train_sizes, train_scores, val_scores = learning_curve(
model, X, y, cv=5,
train_sizes=[Link](0.1, 1.0, 10))

train_mean = [Link](train_scores, axis=1)

val_mean = [Link](val_scores, axis=1)

# If gap increases → overfitting; if both low → underfitting

print(f"Training score: {train_mean[-1]:.4f}, Validation score: {val_mean[-1]:.4f}")
 4.3 Bias-Variance Tradeoff
Bias: Error from overly simplistic assumptions. High bias = underfitting.

Variance: Error from excessive sensitivity to training data. High variance = overfitting.

Total Error = Bias² + Variance + Irreducible Error

Goal: Balance bias and variance for optimal generalization.

# Visualizing bias-variance tradeoff

from [Link] import RandomForestClassifier
from [Link] import DecisionTreeClassifier

depths = range(1, 20)

train_scores = []
test_scores = []

X, y = make_classification(n_samples=300, n_features=10, n_classes=2, random_state=4

for depth in depths:

dt = DecisionTreeClassifier(max_depth=depth)
[Link](X[:200], y[:200])

train_scores.append([Link](X[:200], y[:200]))
test_scores.append([Link](X[200:], y[200:]))

print("Shallow trees: high bias, low variance")

print("Deep trees: low bias, high variance")

4.4 Linear Models

Logistic Regression: Classification using sigmoid function. Output = P(y=1|x).

Linear Regression: Prediction using linear combination: y = w₁x₁ + w₂x₂ + ... + b

from sklearn.linear_model import LogisticRegression, LinearRegression

from [Link] import accuracy_score, mean_squared_error

X_train = [Link](100, 5)
y_class = [Link](0, 2, 100)
y_reg = [Link](100)
 # Logistic Regression
log_reg = LogisticRegression()
log_reg.fit(X_train, y_class)
y_pred = log_reg.predict(X_train)
print(f"Logistic Regression Accuracy: {accuracy_score(y_class, y_pred):.4f}")

# Linear Regression
lin_reg = LinearRegression()
lin_reg.fit(X_train, y_reg)
y_pred_reg = lin_reg.predict(X_train)
mse = mean_squared_error(y_reg, y_pred_reg)
print(f"Linear Regression MSE: {mse:.4f}")

4.5 Support Vector Machines (SVM)

Concept: Find hyperplane maximizing margin between classes. Support vectors define decision boundary.

Kernels: Linear, RBF (Radial Basis Function), polynomial. RBF handles non-linear problems.

Strength: Effective in high dimensions, memory efficient. Weakness: Slow on large datasets.

from [Link] import SVC, SVR

from [Link] import accuracy_score, classification_report

X = [Link](100, 2)
y = (X[:, 0] + X[:, 1] > 0).astype(int)

# Linear SVM
svm_linear = SVC(kernel='linear', C=1.0)
svm_linear.fit(X, y)
accuracy_linear = svm_linear.score(X, y)

# RBF kernel SVM (non-linear)

svm_rbf = SVC(kernel='rbf', C=1.0, gamma='scale')
svm_rbf.fit(X, y)
accuracy_rbf = svm_rbf.score(X, y)

print(f"Linear SVM: {accuracy_linear:.4f}, RBF SVM: {accuracy_rbf:.4f}")

print(f"Number of support vectors: {len(svm_rbf.support_vectors_)}")
 4.6 Decision Trees and Random Forests
Decision Trees: Hierarchical splits on features. Easy to interpret but prone to overfitting.

Random Forests: Ensemble of decision trees. Reduces variance, handles non-linearity, feature importance.

from [Link] import DecisionTreeClassifier

from [Link] import RandomForestClassifier

X = [Link](200, 10)
y = [Link](0, 2, 200)

# Single Decision Tree

dt = DecisionTreeClassifier(max_depth=5)
[Link](X, y)

# Random Forest
rf = RandomForestClassifier(n_estimators=100, max_depth=5, random_state=42)
[Link](X, y)

print(f"Decision Tree accuracy: {[Link](X, y):.4f}")

print(f"Random Forest accuracy: {[Link](X, y):.4f}")

# Feature importance from Random Forest

importances = rf.feature_importances_
top_features = [Link](importances)[-5:]
print(f"Top 5 important features: {top_features}")

4.7 Gradient Boosting Machines (GBM)

Method: Sequentially build trees, each correcting previous errors. Combines weak learners into strong
predictor.

Variants: XGBoost, LightGBM, CatBoost (optimized versions)

Advantage: Best for tabular data, handles feature interactions. Disadvantage: Prone to overfitting, requires
tuning.
 from [Link] import GradientBoostingClassifier
from xgboost import XGBClassifier

X = [Link](200, 10)
y = [Link](0, 2, 200)

# Scikit-learn GBM
gbm = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
[Link](X, y)

# XGBoost (faster, more optimized)

xgb = XGBClassifier(n_estimators=100, learning_rate=0.1, random_state=42)
[Link](X, y)

print(f"GBM accuracy: {[Link](X, y):.4f}")

print(f"XGBoost accuracy: {[Link](X, y):.4f}")

UNIT 5: NEURAL NETWORKS AND DEEP LEARNING

5.1 Introduction to Neural Networks

Structure: Input layer → Hidden layers → Output layer. Neurons connected with weights.

Forward Pass: Input × Weights + Bias → Activation function → Output

Key Concepts: Neurons, weights, biases, activation functions, layers.

import tensorflow as tf
from tensorflow import keras

# Simple neural network

model = [Link]([
[Link](64, activation='relu', input_shape=(20,)),
[Link](32, activation='relu'),
[Link](1, activation='sigmoid')
])

X_train = [Link](100, 20)

y_train = [Link](0, 2, 100)
 [Link](optimizer='adam', loss='binary_crossentropy',
metrics=['accuracy'])
[Link](X_train, y_train, epochs=10, verbose=0)

print(f"Model parameters: {model.count_params()}")

5.2 Training Neural Networks: Backpropagation

Forward Pass: Data flows through network, generates predictions.

Backward Pass (Backpropagation): Compute gradients of loss w.r.t. weights, update weights using gradient
descent.

Loss Function: Quantifies prediction error. Minimize during training.

# Backpropagation visualization
def backpropagation_demo():
# Weight update rule: w_new = w_old - learning_rate * gradient
learning_rate = 0.01
weights = [Link](10)

for epoch in range(5):

# Forward pass
predictions = weights * [Link](5)
loss = [Link]((predictions - [Link](0, 2, 5))**2)

# Backward pass (simplified)

gradients = 2 * (predictions - 0.5) / len(predictions)

# Update weights
weights -= learning_rate * gradients

print(f"Epoch {epoch}: Loss = {loss:.4f}")

backpropagation_demo()
 5.3 Activation Functions
ReLU (Rectified Linear): max(0, x). Fast, prevents vanishing gradient. Standard choice.

Sigmoid: 1/(1+e^-x). Output in [0,1]. Used for binary output.

Tanh: (e^x - e^-x)/(e^x + e^-x). Output in [-1,1]. Stronger gradient than sigmoid.

Softmax: Multi-class probability distribution. Used for classification output.

import tensorflow as tf

# Activation functions
x = [Link](-5, 5, 100)

relu = [Link](x).numpy()
sigmoid = [Link](x).numpy()
tanh = [Link](x).numpy()

# Using in model
model = [Link]([
[Link](64, activation='relu'), # Hidden layer
[Link](32, activation='relu'),
[Link](10, activation='softmax') # Output layer
])

5.4 Regularization Techniques

L1/L2 Regularization: Add penalty on weight magnitude. Prevents overfitting.

Dropout: Randomly deactivate neurons during training. Reduces co-adaptation.

Early Stopping: Stop training when validation loss stops improving.

from [Link] import regularizers

model = [Link]([
[Link](128, activation='relu',
kernel_regularizer=regularizers.l2(0.001),
input_shape=(100,)),
[Link](0.5), # Drop 50% neurons
[Link](64, activation='relu',
 kernel_regularizer=regularizers.l2(0.001)),
[Link](0.3),
[Link](10, activation='softmax')
])

[Link](optimizer='adam', loss='categorical_crossentropy')

# Early stopping
early_stopping = [Link](
monitor='val_loss', patience=5, restore_best_weights=True)

X_train, X_val = [Link](800, 100), [Link](200, 100)

y_train, y_val = [Link](0, 10, 800), [Link](0, 10, 200)

[Link](X_train, y_train, validation_data=(X_val, y_val),

callbacks=[early_stopping], epochs=100, verbose=0)

5.5 Convolutional Neural Networks (CNNs)

Convolution: Sliding window detecting features. Shares weights across spatial dimensions.

Pooling: Downsamples feature maps. Max pooling keeps strongest activations.

Use Case: Image processing, computer vision tasks.

from [Link] import layers

model = [Link]([
layers.Conv2D(32, (3, 3), activation='relu',
input_shape=(28, 28, 1)),
layers.MaxPooling2D((2, 2)),

layers.Conv2D(64, (3, 3), activation='relu'),

layers.MaxPooling2D((2, 2)),

layers.Conv2D(64, (3, 3), activation='relu'),

[Link](),
[Link](64, activation='relu'),
[Link](0.5),
[Link](10, activation='softmax')
])
 [Link](optimizer='adam', loss='sparse_categorical_crossentropy',
metrics=['accuracy'])

print([Link]())

5.6 Advanced CNN Architectures

ResNet: Residual connections skip layers. Enables very deep networks.

VGG: Simple stacked convolutions. Effective feature extractor.

Inception: Multiple filter sizes in parallel. Multi-scale feature extraction.

MobileNet: Lightweight for mobile deployment. Depthwise separable convolutions.

# Transfer learning with pre-trained models

from [Link] import ResNet50, VGG16, MobileNetV2

# Load pre-trained models

resnet = ResNet50(weights='imagenet', include_top=False,
input_shape=(224, 224, 3))

vgg = VGG16(weights='imagenet', include_top=False,

input_shape=(224, 224, 3))

mobilenet = MobileNetV2(weights='imagenet', include_top=False,

input_shape=(224, 224, 3))

# Fine-tuning
for layer in [Link][:-10]:
[Link] = False

# Add custom layers for your task

model = [Link]([
resnet,
layers.GlobalAveragePooling2D(),
[Link](256, activation='relu'),
[Link](0.5),
[Link](10, activation='softmax')
])
 5.7 Recurrent Neural Networks (RNNs)
Architecture: Processes sequences with hidden state memory. Output depends on previous inputs and states.

LSTM (Long Short-Term Memory): Gates control information flow. Solves vanishing gradient problem.

GRU (Gated Recurrent Unit): Simplified LSTM with fewer parameters.

from [Link] import LSTM, GRU, Embedding

# LSTM for sequence processing

model = [Link]([
[Link](input_dim=1000, output_dim=64, input_length=100),
[Link](128, return_sequences=True),
[Link](64),
[Link](32, activation='relu'),
[Link](0.5),
[Link](1, activation='sigmoid')
])

# For bidirectional processing

bidirectional_model = [Link]([
[Link](1000, 64, input_length=100),
[Link]([Link](64, return_sequences=True)),
[Link]([Link](32)),
[Link](1, activation='sigmoid')
])

[Link](optimizer='adam', loss='binary_crossentropy',
metrics=['accuracy'])

5.8 Applications of Neural Networks

Computer Vision: Image classification, object detection, semantic segmentation, face recognition.

Natural Language Processing: Machine translation, sentiment analysis, text generation, question answering.

Time Series: Stock prediction, weather forecasting, anomaly detection.

Reinforcement Learning: Game playing, robotics, autonomous systems.

 # NLP example: Text classification with LSTM
vocab_size = 10000
embedding_dim = 128
max_length = 100

model = [Link]([
[Link](vocab_size, embedding_dim, input_length=max_length),
[Link](64, return_sequences=True),
[Link](32),
[Link](16, activation='relu'),
[Link](0.5),
[Link](1, activation='sigmoid') # Binary sentiment
])

[Link](optimizer='adam', loss='binary_crossentropy',
metrics=['accuracy'])

# Time series example

model_ts = [Link]([
[Link](50, activation='relu', input_shape=(10, 1),
return_sequences=True),
[Link](50, activation='relu'),
[Link](25, activation='relu'),
[Link](1) # Predict next value
])

5.9 Model Evaluation Metrics

Classification: Accuracy, Precision, Recall, F1-score, AUC-ROC, Confusion Matrix.

Regression: MAE, MSE, RMSE, R² Score.

Multi-class: Macro/Micro averaging, Per-class metrics.

from [Link] import accuracy_score, precision_score, recall_score

from [Link] import f1_score, roc_auc_score, confusion_matrix
from [Link] import classification_report

y_true = [Link]([0, 1, 1, 0, 1, 0, 1, 1])

y_pred = [Link]([0, 1, 1, 0, 0, 0, 1, 1])
 print(f"Accuracy: {accuracy_score(y_true, y_pred):.4f}")
print(f"Precision: {precision_score(y_true, y_pred):.4f}")
print(f"Recall: {recall_score(y_true, y_pred):.4f}")
print(f"F1-Score: {f1_score(y_true, y_pred):.4f}")

print("\nConfusion Matrix:")
print(confusion_matrix(y_true, y_pred))

print("\nClassification Report:")
print(classification_report(y_true, y_pred))

5.10 Hyperparameter Tuning

Grid Search: Test all combinations of hyperparameters. Exhaustive but slow.

Random Search: Random sampling of hyperparameter space. Faster, often competitive.

Bayesian Optimization: Probabilistic model guiding search. Most efficient.

from sklearn.model_selection import GridSearchCV, RandomizedSearchCV

# Grid Search for SVM

param_grid = {
'C': [0.1, 1, 10, 100],
'kernel': ['linear', 'rbf', 'poly'],
'gamma': ['scale', 'auto']
}

grid_search = GridSearchCV(SVC(), param_grid, cv=5, n_jobs=-1)

grid_search.fit(X_train, y_train)

print(f"Best parameters: {grid_search.best_params_}")

print(f"Best CV score: {grid_search.best_score_:.4f}")

# Random Search for faster results

param_dist = {
'n_estimators': [50, 100, 200],
'max_depth': [5, 10, 15, None],
'min_samples_split': [2, 5, 10]
}

random_search = RandomizedSearchCV(RandomForestClassifier(),
 param_dist, n_iter=10, cv=5)
random_search.fit(X_train, y_train)

print(f"Best parameters: {random_search.best_params_}")

5.11 Bias-Variance Tradeoff in Deep Learning

Deep Learning Perspective:

Bias (Underfitting): Simple models fail to capture complex patterns

Variance (Overfitting): Excessive capacity learns noise

Solutions: Data augmentation, batch normalization, dropout, regularization, ensemble methods.

# Batch normalization reduces internal covariate shift

model = [Link]([
[Link](256, activation=None, input_shape=(100,)),
[Link](),
[Link]('relu'),

[Link](128, activation=None),
[Link](),
[Link]('relu'),

[Link](64, activation=None),
[Link](),
[Link]('relu'),

[Link](10, activation='softmax')
])

# Data augmentation for images

augmentation = [Link]([
[Link]("horizontal"),
[Link](0.1),
[Link](0.1),
])
 5.12 Model Selection and Comparison
Best Practices:

1. Use same train/validation/test split

2. Cross-validation for reliable estimates

3. Statistical significance testing

4. Report multiple metrics

5. Consider computational cost and interpretability

from sklearn.model_selection import cross_validate

models = {
'Logistic Regression': LogisticRegression(),
'SVM': SVC(),
'Random Forest': RandomForestClassifier(n_estimators=100),
'Gradient Boosting': GradientBoostingClassifier()
}

X, y = make_classification(n_samples=1000, n_features=20, n_classes=2)

for name, model in [Link]():

cv_results = cross_validate(model, X, y, cv=5,
scoring=['accuracy', 'precision', 'recall'])

print(f"\n{name}:")
print(f" Accuracy: {cv_results['test_accuracy'].mean():.4f} ± {cv_results['test
print(f" Precision: {cv_results['test_precision'].mean():.4f}")
print(f" Recall: {cv_results['test_recall'].mean():.4f}")

SUMMARY AND KEY TAKEAWAYS

1. Data is Foundation: Quality data collection and preprocessing crucial for ML success

2. Feature Engineering: Features determine model potential—invest time here

3. Algorithm Selection: No universal best algorithm; choose based on problem characteristics

 4. Model Evaluation: Use appropriate metrics and cross-validation for reliable assessment

5. Regularization: Critical to prevent overfitting in all model types

6. Deep Learning: Powerful but requires more data, computational resources, and careful tuning

7. Ensemble Methods: Often outperform single models by combining strengths

8. Practical Considerations: Interpretability, computational cost, and maintenance important in production

This comprehensive guide covers essential topics in Advanced Machine Learning. Practice implementing each
concept with real datasets for mastery.