Master Project Roadmap — Batch 2: Machine Learning Projects

MASTER PROJECT ROADMAP

AI & Data Science Students

BATCH 1 BATCH 2 BATCH 3

Data Analytics ✓ Machine Learning ML + GenAI
30 Projects

This document covers all 30 Machine Learning project roadmaps — from Customer Churn Prediction
to Employee Attrition Prediction — spanning NLP, Computer Vision, Tabular ML, Time Series,
Healthcare AI, FinTech, and Recommender Systems.

Generated: 18 March 2026 · Batch 2 of 3 · For Academic Use

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 1

Master Project Roadmap — Batch 2: Machine Learning Projects

BATCH 2
MACHINE LEARNING PROJECTS
This batch covers 30 comprehensive Machine Learning projects spanning supervised learning,
unsupervised learning, deep learning (CNNs, LSTMs, Transformers), NLP, computer vision,
recommender systems, time-series forecasting, and healthcare AI.

Each project follows the 9-phase implementation roadmap from data collection through to deployment,
with emphasis on production-grade practices: proper validation methodology, explainability (SHAP),
fairness analysis, and API deployment.

Quick Reference — All 30 ML Projects

# Project Title Domain / Type
1 Customer Churn Prediction CRM & Retention
2 Sales Forecasting Model Business Forecasting
3 Credit Card Fraud Detection Risk & Compliance
4 Product Recommendation System Personalisation
5 Customer Lifetime Value Prediction Customer Analytics
6 Fake News Detection NLP / Media
7 Spam Email Classifier NLP / Security
8 Product Review Sentiment Analysis NLP
9 Resume Screening System HR Tech
10 News Topic Classification NLP
11 Face Mask Detection Computer Vision
12 Handwritten Digit Recognition Computer Vision
13 Traffic Sign Recognition Computer Vision / Autonomous
14 Object Detection using YOLO Computer Vision
15 Plant Disease Detection AgriTech
16 Diabetes Prediction System Healthcare AI
17 Heart Disease Prediction Healthcare AI
18 Breast Cancer Detection Healthcare AI
19 Medical Image Classification Healthcare AI
20 Stock Price Prediction FinTech
21 Loan Approval Prediction Credit Tech
22 Credit Risk Modelling Credit Tech
23 Movie Recommendation System Personalisation

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 2

Master Project Roadmap — Batch 2: Machine Learning Projects

24 Music Recommendation System Personalisation

25 House Price Prediction Real Estate
26 Energy Consumption Forecasting Utilities / Energy
27 Dynamic Pricing Prediction Revenue Management
28 Retail Demand Forecasting Supply Chain
29 Customer Purchase Prediction E-Commerce
30 Employee Attrition Prediction HR Analytics

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 3

Master Project Roadmap — Batch 2: Machine Learning Projects

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 4

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 1 of 30 · Machine Learning

1. Customer Churn Prediction

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Customer churn — when subscribers cancel or stop purchasing — is one of the most expensive
problems in business. Acquiring a new customer costs 5–7× more than retaining an existing one. This
project builds a full end-to-end churn prediction system using Telco or SaaS data, training binary
classifiers that flag at-risk customers before they leave, enabling targeted retention interventions.

2 Objectives
▸ Train and compare multiple binary classifiers for churn prediction
▸ Handle class imbalance using SMOTE and class weighting
▸ Perform SHAP-based feature importance analysis to explain model predictions
▸ Build a customer risk scoring API using FastAPI
▸ Create a retention priority dashboard showing churn probability scores

3 Real-World Applications
▸ Telecom operators targeting retention offers to high-risk subscribers
▸ SaaS companies identifying accounts showing disengagement signals
▸ Banks preventing high-value account attrition
▸ Insurance companies managing policyholder renewal risk

4 Suggested Datasets
▸ Kaggle: 'Telco Customer Churn' (IBM, 7,043 customers, 21 features) — canonical dataset
▸ Kaggle: 'Bank Churners Dataset' (10,127 credit card customers)
▸ Kaggle: 'E-Commerce Churn Dataset'
▸ Kaggle: 'SaaS Customer Churn Dataset'

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost LightGBM
SHAP imbalanced-learn Streamlit FastAPI
(SMOTE)

6 System Architecture & Pipeline

📥 Customer Data (demographics, usage, service, account)
↓
🧹 Clean & Encode (label encode, one-hot encode categoricals)

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 5

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
⚖️Handle Class Imbalance (SMOTE / class_weight='balanced')
↓
🔧 Feature Engineering (tenure bins, charge-per-service ratio)
↓
🤖 Train: Logistic Regression, RF, XGBoost, LightGBM
↓
📊 Evaluate: ROC-AUC, F1, Precision-Recall Curve
↓
🔍 SHAP Feature Importance Explanation
↓
🚀 FastAPI Scoring Endpoint + Streamlit Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Define churn binary label. Identify business cost of false negatives (missed churners) vs. false positives
(wasted retention spend). Set threshold strategy based on cost matrix.
Phase 2: Dataset Collection
Load Telco Churn CSV. Columns: gender, SeniorCitizen, tenure, Contract, MonthlyCharges,
TotalCharges, Churn. Churn rate ≈ 26.5%.
Phase 3: Data Cleaning
Fix TotalCharges (object → numeric). One-hot encode: Contract, InternetService, PaymentMethod.
Label encode: gender, Partner, Dependents. Create churn_binary (0/1).
Phase 4: EDA
Churn rate by Contract, tenure group, and MonthlyCharges band. Correlation heatmap. Box plot:
MonthlyCharges by churn status.
Phase 5: Feature Engineering
tenure_group bins. avg_charge_per_month = TotalCharges / tenure. services_count (count of active
add-ons). has_tech_support binary.
Phase 6: Handle Imbalance
Apply SMOTE from imbalanced-learn on training set only. Alternatively, set class_weight='balanced' in
classifiers. Compare both approaches.
Phase 7: Model Training
Split 80/20 train-test. Train: LogisticRegression, RandomForestClassifier, XGBClassifier,
LGBMClassifier. Use GridSearchCV for hyperparameter tuning on RF and XGB.
Phase 8: Evaluation

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 6

Master Project Roadmap — Batch 2: Machine Learning Projects

Compute: Accuracy, Precision, Recall, F1, ROC-AUC for all models. Plot ROC curves. Plot Precision-
Recall curves (better for imbalanced data). Confusion matrix.
Phase 9: Deployment
Best model → pickle. FastAPI endpoint: POST /predict accepts customer JSON → returns
churn_probability. Streamlit dashboard shows top 100 at-risk customers sorted by churn probability ×
monthly revenue.

8 Algorithms & Models

▸ Logistic Regression (baseline + interpretability)
▸ Random Forest Classifier
▸ XGBoost (gradient boosted trees)
▸ LightGBM (fast gradient boosting)
▸ SHAP (SHapley Additive exPlanations) for model explanation
▸ SMOTE (Synthetic Minority Oversampling Technique)

9 Evaluation Metrics
ROC-AUC Score F1 Score (weighted) Precision @ threshold

Recall @ threshold Matthews Correlation Coefficient Lift and Gain Charts

(MCC)

Business Cost Matrix Score

10 Expected Output

A churn prediction system that assigns a churn probability score (0–100%) to every
customer, ranks them by risk × revenue value, and exposes a FastAPI scoring endpoint. A
Streamlit dashboard shows the top 200 at-risk customers with their key churn drivers
(SHAP values) — enabling retention teams to act before customers leave.

11 Optional Advanced Enhancements

▸ Add survival analysis (Cox Proportional Hazards) for time-to-churn prediction
▸ Build a real-time churn scoring pipeline using Kafka + model serving
▸ Add counterfactual explanations: 'Customer would stay if contract type changed to Annual'
▸ Train on longitudinal data to detect churn early signals at Day 30, 60, 90

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 7

Master Project Roadmap — Batch 2: Machine Learning Projects

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 8

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 2 of 30 · Machine Learning

2. Sales Forecasting Model

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Accurate sales forecasting is mission-critical for inventory planning, cash flow management, workforce
scheduling, and promotional budgeting. This project builds a multi-horizon sales forecasting system that
combines classical time-series models (SARIMA), gradient boosting with lag features (LightGBM), and
the Facebook Prophet library — comparing their accuracy and building an ensemble forecast.

2 Objectives
▸ Build and compare SARIMA, Prophet, and LightGBM forecasting models
▸ Generate 7-day, 30-day, and 90-day sales forecasts with confidence intervals
▸ Incorporate external features: holidays, promotions, weather, economic indicators
▸ Evaluate forecast accuracy using MAE, RMSE, and MAPE
▸ Build a forecast dashboard with scenario planning capability

3 Real-World Applications
▸ Retail chains planning inventory replenishment based on demand forecast
▸ FMCG companies coordinating manufacturing with downstream demand signals
▸ E-commerce platforms forecasting GMV for financial planning
▸ Restaurants predicting daily covers for staffing and ingredient ordering

4 Suggested Datasets
▸ Kaggle: 'Store Sales — Time Series Forecasting' (Favorita, 3M+ rows)
▸ Kaggle: 'Walmart Recruiting: Store Sales Forecasting' (45 stores, weekly)
▸ Kaggle: 'Rossman Store Sales' (1,115 German drug stores)
▸ Kaggle: 'M5 Forecasting — Accuracy' (Walmart, 42,840 series)

5 Recommended Tech Stack

Python Pandas NumPy statsmodels Prophet (Facebook)
(SARIMA)
LightGBM Scikit-learn Plotly Streamlit

6 System Architecture & Pipeline

📥 Historical Sales Time Series (daily/weekly by store/SKU)
↓
🧹 Clean: Fill gaps, handle store closures, align time index

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 9

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
📊 Decompose: Trend + Seasonality + Residual
↓
🔧 Feature Engineering: Lags, Rolling Means, Holiday Flags
↓
🤖 Model 1: SARIMA / SARIMAX
↓
🤖 Model 2: Facebook Prophet
↓
🤖 Model 3: LightGBM with lag features
↓
📐 Evaluate & Ensemble
↓
📈 Forecast Dashboard with Confidence Intervals

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Define forecast horizon (7/30/90 days). Identify granularity: store-level, SKU-level, or aggregate.
Understand what external regressors are available (holidays, promotions).
Phase 2: Dataset Collection
Rossman dataset: 1,017,209 rows, 1,115 stores, 2.5 years daily sales. Columns: Store, DayOfWeek,
Date, Sales, Customers, Open, Promo, StateHoliday, SchoolHoliday.
Phase 3: Data Cleaning
Remove rows where Open=0 (store closed). Forward-fill missing dates. Merge store metadata
(StoreType, Assortment, CompetitionDistance). Compute log(Sales+1) for normality.
Phase 4: Feature Engineering for LightGBM
Lag features: lag_1, lag_7, lag_14, lag_28. Rolling stats: rolling_mean_7, rolling_mean_28,
rolling_std_7. Date features: day_of_week, week_of_year, month, is_weekend. Holiday dummies.
Phase 5: SARIMA Modelling
Use auto_arima (pmdarima) to find optimal (p,d,q)(P,D,Q,m) parameters. Fit on training set. Generate
forecast with prediction intervals. Compute MAE and RMSE on test set.
Phase 6: Prophet Modelling
Fit Prophet with daily_seasonality=True, weekly_seasonality=True. Add custom holiday DataFrame.
Generate forecast with yhat, yhat_lower, yhat_upper. Evaluate on test set.
Phase 7: LightGBM Modelling
Train on lag-feature DataFrame. Use TimeSeriesSplit for cross-validation (no data leakage). Tune
num_leaves, learning_rate, n_estimators via Optuna.
AI & Data Science Student Projects · Batch 2: Machine Learning · Page 10
Master Project Roadmap — Batch 2: Machine Learning Projects

Phase 8: Ensemble & Evaluation

Simple ensemble: weighted average of SARIMA, Prophet, LightGBM. Compare individual and
ensemble MAPE. Plot actual vs. forecast chart with uncertainty bands.
Phase 9: Deployment
Streamlit app: select store + forecast horizon → display forecast chart + scenario comparison (base
case, optimistic, pessimistic). Export forecast to CSV.

8 Algorithms & Models

▸ SARIMA / SARIMAX (statsmodels)
▸ Facebook Prophet (additive model with changepoints)
▸ LightGBM with lag and rolling features
▸ Weighted ensemble forecasting
▸ Optuna hyperparameter optimisation
▸ TimeSeriesSplit cross-validation

9 Evaluation Metrics
MAE (Mean Absolute Error) RMSE (Root Mean Squared Error) MAPE (Mean Absolute
Percentage Error)

sMAPE (Symmetric MAPE) Coverage of Prediction Intervals Winkler Score (interval sharpness)

10 Expected Output

A sales forecasting system that generates accurate multi-horizon forecasts for any store,
produces confidence intervals, and compares 3 model families — with a Streamlit interface
where planners can explore forecast scenarios, download projections to Excel, and see
which stores are forecast to miss targets.

11 Optional Advanced Enhancements

▸ Add N-BEATS or N-HiTS neural forecasting models (Darts library)
▸ Build a hierarchical forecasting system: reconcile store-level forecasts to chain total
▸ Add real-time forecast refresh pipeline with automated retraining
▸ Implement forecast monitoring: alert when actual deviates > 2σ from forecast

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 11

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Scale to production with MLflow model registry and CI/CD pipeline

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 12

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 3 of 30 · Machine Learning

3. Credit Card Fraud Detection

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Card-not-present (CNP) fraud costs the global payments industry over $32 billion annually. The core
challenge is extreme class imbalance: only 0.17% of transactions are fraudulent, making standard
accuracy meaningless. This project builds a fraud detection pipeline that handles class imbalance,
achieves high recall (catching most fraud), and minimises false positives (blocking legitimate
customers).

2 Objectives
▸ Build a fraud detector with high recall and controlled false positive rate
▸ Handle extreme class imbalance using SMOTE, ADASYN, and cost-sensitive learning
▸ Compare Isolation Forest (unsupervised) vs. supervised classifiers
▸ Optimise the classification threshold using cost-benefit analysis
▸ Build a real-time fraud scoring API with sub-100ms response time

3 Real-World Applications
▸ Payment card issuers (Visa, Mastercard, Amex) real-time transaction scoring
▸ E-commerce platforms flagging suspicious checkout attempts
▸ Digital wallets and neobanks protecting user accounts
▸ Insurance companies detecting claim fraud patterns

4 Suggested Datasets
▸ Kaggle: 'Credit Card Fraud Detection' (284,807 transactions, 492 fraud, PCA features) —
canonical
▸ Kaggle: 'IEEE-CIS Fraud Detection' (590,540 transactions, 433 features)
▸ Kaggle: 'Synthetic Financial Datasets (PaySim)' (6.3M transactions)
▸ Kaggle: 'Fraudulent Transactions Prediction'

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost LightGBM
imbalanced-learn SHAP FastAPI Streamlit

6 System Architecture & Pipeline

📥 Transaction Data (Amount, Time, V1–V28 PCA features, Class)
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 13

Master Project Roadmap — Batch 2: Machine Learning Projects

🔧 Feature Scaling (StandardScaler on Amount, Time)

↓
⚖️Handle Imbalance: SMOTE, ADASYN, Class Weights
↓
🤖 Supervised: LogReg, RF, XGBoost, LightGBM
↓
🔍 Unsupervised: Isolation Forest, One-Class SVM
↓
📊 Threshold Optimisation (Cost Matrix)
↓
🚀 FastAPI Real-time Scoring Endpoint

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Define business cost: False Negative (missed fraud) = full transaction amount loss. False Positive
(blocking legit) = customer friction cost ≈ $5. Compute optimal threshold using cost-benefit matrix.
Phase 2: Dataset Collection
Credit Card Fraud dataset: 284,807 transactions over 2 days. 492 fraud (0.172%). V1–V28 are PCA-
transformed for privacy. Amount and Time are raw.
Phase 3: Data Cleaning & Scaling
StandardScaler on Amount and Time. No nulls in this dataset. Create log_amount = log(Amount+1).
Separate fraud and legit for analysis.
Phase 4: Imbalance Handling
Strategy 1: SMOTE on training set. Strategy 2: ADASYN (adaptive synthetic). Strategy 3:
class_weight='balanced'. Strategy 4: cost-sensitive XGBoost (scale_pos_weight). Compare all
strategies.
Phase 5: Supervised Models
Train on 80% split. Models: LogisticRegression, RandomForest, XGBoost, LightGBM. Evaluate on test
set using Precision-Recall AUC (more informative than ROC-AUC for imbalanced data).
Phase 6: Unsupervised Models
Train Isolation Forest on full dataset (contamination=0.002). One-Class SVM on legit transactions only.
Compute anomaly scores. Evaluate recall on fraud cases.
Phase 7: Threshold Optimisation
For best supervised model: sweep threshold from 0.1 to 0.9. Compute cost at each threshold =
FN_count × avg_fraud_amount + FP_count × fp_cost. Find minimum cost threshold.
Phase 8: SHAP Explanation
Compute SHAP values for XGBoost model. Waterfall plot for individual flagged transaction. Summary
plot showing top features globally.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 14

Master Project Roadmap — Batch 2: Machine Learning Projects

Phase 9: Deployment
FastAPI: POST /score-transaction → JSON input → fraud_probability + decision + top_3_risk_factors.
Response < 100ms. Streamlit: real-time fraud alert queue, daily fraud stats, confusion matrix.

8 Algorithms & Models

▸ Logistic Regression (L1 regularised)
▸ Random Forest with class weighting
▸ XGBoost with scale_pos_weight
▸ LightGBM
▸ Isolation Forest (anomaly detection)
▸ One-Class SVM
▸ SMOTE & ADASYN (synthetic oversampling)
▸ SHAP explanations

9 Evaluation Metrics
Precision-Recall AUC (primary) ROC-AUC F1 Score

Recall @ 80% Precision (business False Positive Rate at optimal Business Cost Score (FN ×
threshold) threshold fraud_amount + FP × fp_cost)

Inference Latency (ms)

10 Expected Output

A real-time fraud detection API that scores every incoming transaction within 100ms,
provides a fraud probability score and top risk factors, and achieves > 80% recall with <
5% false positive rate — enabling payment providers to block most fraud while preserving
customer experience.

11 Optional Advanced Enhancements

▸ Add graph neural network (GNN) for transaction network fraud detection
▸ Build a continuous learning pipeline that retrains on new labelled fraud cases weekly
▸ Add a velocity check layer: rule-based pre-filter before ML scoring
▸ Implement federated learning to train across banks without sharing raw data

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 15

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Scale to production with MLflow model registry and CI/CD pipeline

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 16

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 4 of 30 · Machine Learning

4. Product Recommendation System

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Recommendation engines drive 35% of Amazon's revenue and 75% of Netflix's watch time. This
project builds a product recommendation system using collaborative filtering (user-item matrix
factorisation), content-based filtering (item similarity), and a hybrid approach — providing personalised
product suggestions at scale.

2 Objectives
▸ Build user-based and item-based collaborative filtering models
▸ Implement matrix factorisation using SVD and ALS
▸ Build a content-based recommender using TF-IDF on product descriptions
▸ Combine CF and CB into a hybrid recommender
▸ Build a recommendation API with real-time user preference updates

3 Real-World Applications
▸ E-commerce platforms (Amazon-style 'Customers also bought')
▸ Grocery delivery apps suggesting frequently bought-together items
▸ Fashion retail recommending complementary clothing items
▸ Digital marketplaces surfacing relevant products to new and returning users

4 Suggested Datasets
▸ Kaggle: 'Amazon Product Reviews' (multiple categories, millions of ratings)
▸ Kaggle: 'MovieLens 1M / 25M' (movie ratings — standard CF benchmark)
▸ Kaggle: 'E-Commerce Product Recommendation Dataset'
▸ UCI: 'Online Retail Dataset' (purchase history for item-item CF)

5 Recommended Tech Stack

Python Pandas Scikit-learn Surprise library implicit (ALS)
TF-IDF (Scikit-learn) Streamlit FastAPI [Link]

6 System Architecture & Pipeline

📥 User-Item Interaction Matrix (ratings / purchase history)
↓
🔧 Build Sparse User-Item Matrix
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 17

Master Project Roadmap — Batch 2: Machine Learning Projects

🤖 Model A: User-Based CF (cosine similarity)

↓
🤖 Model B: Item-Based CF (cosine similarity)
↓
🤖 Model C: SVD Matrix Factorisation (Surprise)
↓
🤖 Model D: Content-Based (TF-IDF on product descriptions)
↓
🔀 Hybrid: Weighted average of CF + CB scores
↓
📊 Evaluate: Precision@K, Recall@K, NDCG
↓
🚀 Recommendation API

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Define recommendation types: Collaborative Filtering (behaviour-based), Content-Based (feature-
based), Knowledge-Based (rule-based), Hybrid. Define cold-start problem.
Phase 2: Dataset Collection
MovieLens 1M: 1M ratings, 6,040 users, 3,900 movies. Columns: UserID, MovieID, Rating, Timestamp.
Movie metadata: genres, title.
Phase 3: Data Preprocessing
Create user-item rating matrix (sparse). Compute sparsity = 1 - (non_zero / total). Filter users with < 5
ratings and items with < 10 ratings. Normalise ratings: subtract user mean.
Phase 4: User-Based CF
Compute cosine similarity between user vectors. For target user: find top-K similar users. Aggregate
their ratings on unseen items (weighted by similarity). Return top-N recommendations.
Phase 5: Item-Based CF
Compute cosine similarity between item vectors. For items user has rated: find similar items. Score =
weighted sum of user ratings × item similarities. Return top-N.
Phase 6: Matrix Factorisation (SVD)
Use Surprise library: SVD(n_factors=100, lr_all=0.005, reg_all=0.02). Train on 80% data. Predict
ratings for all user-item pairs. Evaluate with RMSE using cross_validate.
Phase 7: Content-Based Filtering
TF-IDF on movie genres + title. Compute cosine similarity between item TF-IDF vectors. For items user
liked: return most similar items not yet seen.
Phase 8: Hybrid Recommender
AI & Data Science Student Projects · Batch 2: Machine Learning · Page 18
Master Project Roadmap — Batch 2: Machine Learning Projects

Combined score = α × CF_score + (1-α) × CB_score. Tune α on validation set. Handle cold-start users
with CB-only; power users with CF-dominant hybrid.
Phase 9: Deployment
FastAPI: GET /recommend?user_id=123&n=10 → returns ranked product list with scores. Streamlit
demo: input user ID → display top 10 recommendations with item details.

8 Algorithms & Models

▸ User-Based Collaborative Filtering (cosine similarity)
▸ Item-Based Collaborative Filtering (Pearson correlation)
▸ SVD Matrix Factorisation (Surprise library)
▸ ALS (Alternating Least Squares, implicit library)
▸ TF-IDF Content-Based Filtering
▸ Hybrid weighted ensemble

9 Evaluation Metrics
RMSE (rating prediction accuracy) Precision@K Recall@K

F1@K NDCG@K (Normalised Coverage (% of catalogue

Discounted Cumulative Gain) recommended)

Novelty and Diversity scores

10 Expected Output

A personalised recommendation API that returns the top-N product recommendations for
any user, combining collaborative and content-based signals. A Streamlit demo shows
recommendations updating as user preferences change, with explanations of why each
item was suggested ('Because you liked X').

11 Optional Advanced Enhancements

▸ Implement Neural Collaborative Filtering (NCF) using PyTorch
▸ Add session-based recommendations using GRU4Rec for anonymous users
▸ Build a multi-armed bandit system for real-time exploration vs. exploitation
▸ Add diversity-aware re-ranking to prevent filter bubbles

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 19

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Scale to production with MLflow model registry and CI/CD pipeline

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 20

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 5 of 30 · Machine Learning

5. Customer Lifetime Value Prediction

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Customer Lifetime Value (CLV) tells a business how much net profit a customer will generate over their
entire relationship. Accurately predicting CLV enables smarter marketing spend allocation, customer
tier management, and acquisition ROI calculations. This project builds a CLV prediction system
combining BG/NBD probabilistic modelling with machine learning regression.

2 Objectives
▸ Compute historical CLV using RFM and purchase frequency models
▸ Build a predictive CLV model using BG/NBD and Gamma-Gamma models
▸ Segment customers into CLV tiers (Bronze/Silver/Gold/Platinum)
▸ Predict 12-month CLV for new customers using ML regression
▸ Build a CLV dashboard for marketing budget allocation

3 Real-World Applications
▸ E-commerce companies allocating marketing spend by customer value tier
▸ Banks calculating relationship profitability for product cross-sell
▸ Subscription businesses optimising acquisition channels by CLV
▸ Loyalty programme design based on CLV-tier benefits

4 Suggested Datasets
▸ UCI: 'Online Retail Dataset' (UK e-commerce, 541k transactions)
▸ Kaggle: 'E-Commerce Data' (real UK online retail)
▸ Kaggle: 'Customer Lifetime Value Prediction Dataset'
▸ Kaggle: 'Online Retail II Dataset' (2009-2011)

5 Recommended Tech Stack

Python Pandas lifetimes (BG/NBD Scikit-learn XGBoost
library)
Plotly Streamlit

6 System Architecture & Pipeline

📥 Transaction History (CustomerID, InvoiceDate, Amount)
↓
🔧 Compute RFM Features

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 21

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
📐 BG/NBD Model: Predict Purchase Frequency
↓
📐 Gamma-Gamma Model: Predict Monetary Value
↓
💰 Compute Predicted CLV = freq × monetary × margin
↓
CLV Tier Segmentation
↓
🤖 ML Regression: Predict 12-month CLV for new customers
↓
CLV Dashboard & Marketing Allocation Tool

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

CLV = Avg Purchase Value × Purchase Frequency × Customer Lifespan. Distinguish Historical CLV
(backward-looking) vs. Predictive CLV (forward-looking).
Phase 2: Dataset Collection
UCI Online Retail: 541,909 rows. Columns: InvoiceNo, StockCode, Description, Quantity, InvoiceDate,
UnitPrice, CustomerID, Country. Remove nulls and cancelled orders.
Phase 3: RFM Computation
Snapshot date = max(InvoiceDate) + 1. Recency (days since last purchase). Frequency (number of
repeat purchases). Monetary (total spend). Filter: customers with >1 purchase.
Phase 4: BG/NBD Modelling
Use lifetimes library: BetaGeoFitter. Fit on frequency, recency, T (observation period). Predict:
expected_purchases_in_next_90_days. Validate with holdout period.
Phase 5: Gamma-Gamma Modelling
GammaGammaFitter predicts expected average monetary value per transaction. Prerequisite:
monetary value must be correlated < 0.3 with frequency.
Phase 6: CLV Computation
clv = ggf.customer_lifetime_value(bgf, frequency, recency, T, monetary, time=12, discount_rate=0.01).
This gives 12-month expected revenue per customer.
Phase 7: CLV Tier Segmentation
Percentile-based: Platinum (top 5%), Gold (5-20%), Silver (20-50%), Bronze (bottom 50%). Map each
customer to tier.
Phase 8: ML Regression for New Customers

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 22

Master Project Roadmap — Batch 2: Machine Learning Projects

Features: first_purchase_amount, days_since_first_purchase, acquisition_channel. Target: 12-month

CLV. Model: XGBoost Regressor. Evaluate: MAE, RMSE.
Phase 9: Dashboard
CLV distribution by tier. Revenue concentration (Platinum customers = ? % of total revenue). New
customer CLV estimator widget. Marketing budget ROI calculator.

8 Algorithms & Models

▸ BG/NBD (Beta-Geometric/Negative Binomial Distribution) model
▸ Gamma-Gamma model for monetary value
▸ XGBoost Regression for new customer CLV prediction
▸ Percentile-based CLV tier classification
▸ RFM feature computation

9 Evaluation Metrics
MAE on CLV prediction RMSE on CLV prediction Calibration curve (predicted vs.
actual purchase frequency)

CLV Gini coefficient (revenue Tier accuracy (% of customers

concentration) correctly tiered in holdout)

10 Expected Output

A CLV prediction system that assigns every customer a 12-month revenue forecast,
segments them into value tiers, and provides a marketing allocation tool showing how
much to spend on acquisition channels based on the CLV of customers they deliver —
increasing marketing ROI by focusing spend on channels that bring highest-CLV
customers.

11 Optional Advanced Enhancements

▸ Add channel-level CLV (CLV by acquisition source)
▸ Build a CLV-adjusted bidding model for Google/Meta ad campaigns
▸ Implement deep learning CLV prediction using Pareto/NBD neural network
▸ Add a CLV erosion early warning (customers showing declining purchase frequency)

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 23

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI

▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 24

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 6 of 30 · Machine Learning

6. Fake News Detection

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Misinformation spreads 6× faster than accurate news on social media, causing real-world harm
including election interference, public health crises, and financial market manipulation. This project
builds a fake news classifier using NLP techniques, from classical TF-IDF models to fine-tuned
transformer models, with a browser extension prototype for real-time article scoring.

2 Objectives
▸ Build and compare TF-IDF + ML and BERT-based fake news classifiers
▸ Achieve high accuracy on headline and full-article classification
▸ Explain model predictions using LIME for individual articles
▸ Handle adversarial examples and out-of-domain generalisation
▸ Build a real-time article scoring API

3 Real-World Applications
▸ Social media platforms automatically flagging unverified content
▸ News aggregators warning readers about low-credibility sources
▸ Fact-checking organisations triaging articles for manual review
▸ Academic research on computational journalism

4 Suggested Datasets
▸ Kaggle: 'Fake and Real News Dataset' (44,898 articles, Kaggle gold standard)
▸ Kaggle: 'ISOT Fake News Dataset' (23,481 articles)
▸ Kaggle: 'LIAR Dataset' (12,791 statements with 6-class truthfulness labels)
▸ HuggingFace: 'GonzaloA/fake_news' dataset

5 Recommended Tech Stack

Python Pandas NLTK Scikit-learn TF-IDF
Transformers PyTorch LIME FastAPI Streamlit
(HuggingFace)

6 System Architecture & Pipeline

📥 Articles Dataset (title, text, label: real/fake)
↓
🧹 Clean: Remove URLs, HTML, punctuation, stopwords

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 25

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
📝 Feature Extraction: TF-IDF on title + text
↓
🤖 Model A: TF-IDF + Logistic Regression / SVM
↓
🤖 Model B: Fine-tune DistilBERT for classification
↓
📊 Evaluate & Compare
↓
🔍 LIME Explanation for Individual Predictions
↓
🚀 FastAPI Scoring API

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Binary classification: REAL vs. FAKE. Understand dataset biases: topic distribution, source distribution.
Define evaluation: accuracy + F1 (balanced classes).
Phase 2: Dataset Collection
Kaggle Fake & Real News: 21,417 fake + 21,417 real articles. Columns: title, text, subject, date, label.
Concatenate fake (label=0) and real (label=1) DataFrames.
Phase 3: Text Preprocessing
Lowercase. Remove URLs ([Link]). Remove punctuation. NLTK word_tokenize. Remove stopwords.
Lemmatise with WordNetLemmatizer. Combine title and text into content.
Phase 4: TF-IDF Baseline
TfidfVectorizer(max_features=50000, ngram_range=(1,2)). Train/test 80/20 split. Models:
LogisticRegression, PassiveAggressiveClassifier, MultinomialNB, LinearSVC. Best usually LinearSVC
≈ 99% (beware: same-source test leakage).
Phase 5: Cross-source Evaluation
Split by source: train on one subset, test on different sources. This reveals true generalisation.
Accuracy typically drops to 70-80% — the real challenge.
Phase 6: BERT Fine-tuning
Use DistilBERT from HuggingFace. Tokenise with DistilBertTokenizer (max_length=512). Fine-tune
with AutoModelForSequenceClassification. 3 epochs, lr=2e-5, batch_size=16. GPU recommended.
Phase 7: LIME Explanation
lime.lime_text.LimeTextExplainer on test articles. Show top positive (real) and negative (fake) words
contributing to prediction. Visualise in Streamlit.
Phase 8: Evaluation

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 26

Master Project Roadmap — Batch 2: Machine Learning Projects

Compare all models: accuracy, F1, confusion matrix. Cross-source generalisation matrix. Calibration
curve for probability outputs.
Phase 9: Deployment
FastAPI: POST /check-article with URL or text → returns: label, confidence, top contributing words.
Streamlit: paste URL or text → see real-time prediction.

8 Algorithms & Models

▸ TF-IDF vectorisation (unigram + bigram)
▸ Logistic Regression
▸ Linear SVM (LinearSVC)
▸ Passive Aggressive Classifier
▸ DistilBERT fine-tuning (HuggingFace Transformers)
▸ LIME (Local Interpretable Model-agnostic Explanations)

9 Evaluation Metrics
Accuracy F1 Score (macro) Precision / Recall by class

Cross-source Generalisation ROC-AUC Calibration Error

Accuracy

10 Expected Output

A fake news detection API that accepts any article URL or text and returns a credibility
score (0–100%), confidence level, and the key words driving the decision — enabling fact-
checkers, browser extensions, and social platforms to automatically flag potentially false
content at scale.

11 Optional Advanced Enhancements

▸ Build a source credibility database and integrate it as a feature
▸ Add claim verification using knowledge graph lookup (Wikidata)
▸ Train a multi-class model: True, Mostly True, Half True, Mostly False, False, Pants on Fire
▸ Build a browser extension that scores news articles in real time

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 27

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI

▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 28

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 7 of 30 · Machine Learning

7. Spam Email Classifier

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Spam accounts for 45% of all email traffic globally. This project builds a production-grade spam
classifier using NLP — from classical Naive Bayes (which Google originally used for Gmail spam) to
modern transformer models — with an end-to-end email processing pipeline that parses raw email
headers and body text.

2 Objectives
▸ Build a spam classifier on email text using Naive Bayes, SVM, and BERT
▸ Process raw email data including headers, HTML stripping, and encoding handling
▸ Achieve > 98% accuracy with < 0.1% false positive rate (legitimate email blocked)
▸ Implement continuous learning: incorporate user feedback to improve model
▸ Deploy as an email filtering microservice

3 Real-World Applications
▸ Email service providers (Gmail, Outlook) spam filtering
▸ Corporate IT teams filtering phishing and business email compromise (BEC)
▸ Marketing platforms ensuring legitimate bulk emails reach inboxes
▸ Anti-phishing systems for financial institutions

4 Suggested Datasets
▸ Kaggle: 'SMS Spam Collection Dataset' (5,574 messages, 13.4% spam)
▸ Kaggle: 'Enron Email Dataset' (500k+ business emails)
▸ Apache SpamAssassin Public Corpus (real spam + ham)
▸ TREC 2007 Spam Track Public Corpus

5 Recommended Tech Stack

Python Pandas NLTK Scikit-learn email (Python
stdlib)
BeautifulSoup Transformers FastAPI Streamlit

6 System Architecture & Pipeline

📥 Raw Emails (text/HTML) + Labels (spam/ham)
↓
🧹 Parse: Strip HTML, decode MIME, extract body text

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 29

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
📝 NLP: Clean, tokenise, remove stopwords, stem/lemmatise
↓
🔢 Vectorise: Count Vectoriser / TF-IDF / BERT embeddings
↓
🤖 Train: Naive Bayes, SVM, Logistic Regression, BERT
↓
📊 Evaluate with strict FPR constraint
↓
🔄 Continuous Learning Loop
↓
🚀 Email Filter API

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Binary: SPAM vs. HAM. Critical constraint: False Positive Rate < 0.1% (blocking a legitimate email is
worse than missing spam). Optimise for this constraint.
Phase 2: Dataset Collection
SMS Spam Collection: 5,572 messages, 4,825 ham (86.6%), 747 spam (13.4%). Columns: label,
message. Simple and clean, ideal for learning pipeline.
Phase 3: Email Parsing (Advanced)
For Enron dataset: use Python email library to parse .msg files. Strip HTML with BeautifulSoup. Extract
plain text. Handle Base64 encoded content. Clean signatures.
Phase 4: Feature Extraction
Count Vectoriser: bag of words. TF-IDF: account for term frequency. Word2Vec average embedding.
BERT [CLS] token embedding. Compare all representations.
Phase 5: Naive Bayes Classifier
MultinomialNB on Count Vectoriser features. BernoulliNB as alternative. Compute: accuracy, FPR,
FNR. This baseline achieves ~98% accuracy.
Phase 6: SVM and Logistic Regression
LinearSVC and LogisticRegression on TF-IDF features. Tune C parameter. These typically achieve >
99% accuracy.
Phase 7: BERT Fine-tuning
DistilBERT on GPU. Particularly useful for short SMS-style spam. Fine-tune 3 epochs. Compare with
SVM on short text.
Phase 8: Threshold Optimisation

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 30

Master Project Roadmap — Batch 2: Machine Learning Projects

Sweep threshold to find point where FPR < 0.1%. Report recall at this constrained threshold. This is the
real business metric.
Phase 9: Deployment
FastAPI: POST /classify-email with email text body → spam probability + decision + key spam signals.
Add feedback endpoint: POST /feedback → label + text for retraining queue.

8 Algorithms & Models

▸ Multinomial Naive Bayes
▸ BernoulliNB
▸ LinearSVC (one-vs-rest)
▸ Logistic Regression with L2
▸ DistilBERT fine-tuning
▸ TF-IDF with character n-grams (for obfuscated spam)

9 Evaluation Metrics
Accuracy False Positive Rate (critical: < False Negative Rate
0.1%)

F1 Score AUC-ROC Precision @ FPR=0.1% constraint

10 Expected Output

A spam classification microservice that processes incoming emails, assigns a spam

probability score, and filters with < 0.1% false positive rate — deployed as a FastAPI
endpoint that any email client or server can integrate with, complete with a feedback loop
for continuous improvement.

11 Optional Advanced Enhancements

▸ Add phishing detection layer (URL analysis, sender domain reputation)
▸ Build a multi-label classifier: spam, phishing, promotional, newsletter, primary
▸ Add header analysis (SPF/DKIM/DMARC validation features)
▸ Implement adversarial spam detection: catch deliberate obfuscation techniques

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 31

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI

▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 32

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 8 of 30 · Machine Learning

8. Product Review Sentiment Analysis

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
E-commerce platforms receive millions of product reviews daily, and manually reading them is
impossible. This project builds a multi-class sentiment analysis system that classifies reviews as
Positive, Neutral, or Negative, extracts aspect-level sentiments (price, quality, delivery, customer
service), and surfaces actionable product improvement insights from review text.

2 Objectives
▸ Build a 3-class sentiment classifier on product reviews
▸ Implement aspect-based sentiment analysis (ABSA) for product dimensions
▸ Extract most common complaint and praise topics using topic modelling
▸ Build a product reputation dashboard updated from review streams
▸ Fine-tune a transformer model for domain-specific accuracy

3 Real-World Applications
▸ E-commerce platforms surfacing product quality signals to buyers
▸ Brands monitoring their product reception on Amazon and Flipkart
▸ Product management teams identifying improvement priorities
▸ Market research teams tracking competitive product sentiment

4 Suggested Datasets
▸ Kaggle: 'Amazon Product Reviews' (multiple categories, 1M+ reviews)
▸ Kaggle: 'Flipkart Product Reviews'
▸ Kaggle: 'IMDB Movie Review Sentiment'
▸ HuggingFace: 'amazon_polarity' (3.6M reviews)

5 Recommended Tech Stack

Python Pandas NLTK TextBlob VADER
Scikit-learn Transformers Gensim (LDA) Plotly Streamlit
(BERT)

6 System Architecture & Pipeline

📥 Product Reviews (text, rating, product_id, user_id)
↓
🧹 Clean: HTML, URLs, special chars, emojis

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 33

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
📝 NLP Pre-processing
↓
🤖 Sentiment Classification (3-class)
↓
Aspect Extraction (price, quality, delivery)
↓
📊 Topic Modelling on Negative Reviews (LDA)
↓
Product Reputation Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

3-class: Positive (4-5 stars), Neutral (3 stars), Negative (1-2 stars). Aspect-level: identify sentiment for
specific product dimensions within a review.
Phase 2: Dataset Collection
Amazon Product Reviews (Electronics category): columns: reviewText, overall (rating), summary,
reviewerID, asin (product ID). Sample 100k reviews.
Phase 3: Data Cleaning
Remove HTML entities. Strip URLs. Handle emojis (replace or remove). Convert rating to sentiment
label: 1-2 → Negative, 3 → Neutral, 4-5 → Positive. Balance classes.
Phase 4: Baseline Lexicon Models
VADER: classify each review. TextBlob: polarity score. Compare to ground truth (rating-derived labels).
Typically 70-75% accuracy — good baseline.
Phase 5: TF-IDF + ML
TF-IDF (max_features=50000). LogisticRegression, RandomForest, LinearSVC. 5-fold CV. Best:
LinearSVC ≈ 88-90% accuracy.
Phase 6: BERT Fine-tuning
DistilBERT or RoBERTa (better for review sentiment). Tokenise reviews (max_length=256). Fine-tune 3
epochs. Expected: 92-95% accuracy.
Phase 7: Aspect-Based Sentiment
Define aspects: 'price', 'quality', 'battery', 'screen', 'delivery'. For each aspect: extract sentences
containing aspect keywords. Run sentiment on those sentences. Aggregate per product.
Phase 8: Topic Modelling on Negatives
Filter negative reviews. LDA on TF-IDF matrix (n_topics=10). Label topics manually. Build 'Top 10
Complaints' chart per product.
Phase 9: Dashboard

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 34

Master Project Roadmap — Batch 2: Machine Learning Projects

Product card: avg rating, sentiment distribution, aspect radars, top complaints (word cloud), review
trend. Brand view: compare products side by side.

8 Algorithms & Models

▸ VADER lexicon-based sentiment
▸ TextBlob polarity
▸ TF-IDF + LinearSVC
▸ RoBERTa fine-tuning
▸ Latent Dirichlet Allocation (LDA) for topic modelling
▸ Aspect-Based Sentiment Analysis (rule-based aspect extraction)

9 Evaluation Metrics
3-class Accuracy F1 Score (macro) Per-class Precision & Recall

Aspect Sentiment Accuracy (on LDA Topic Coherence Score (Cv) Dashboard Coverage (% of
manually labelled subset) products with sufficient reviews)

10 Expected Output

A product review intelligence platform where brand managers can see overall sentiment
distribution, aspect-level ratings (e.g., 'Battery: 3.2/5, Price: 4.1/5'), top complaint topics,
and sentiment trends over time — turning millions of unstructured reviews into structured
product insights in seconds.

11 Optional Advanced Enhancements

▸ Add multilingual support (Hindi, Spanish reviews) using mBERT or XLM-RoBERTa
▸ Build a review summarisation module using BART or T5
▸ Add fake review detection layer
▸ Implement opinion mining with dependency parsing for fine-grained aspects

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post
AI & Data Science Student Projects · Batch 2: Machine Learning · Page 35
Master Project Roadmap — Batch 2: Machine Learning Projects

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 36

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 9 of 30 · Machine Learning

9. Resume Screening System

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Recruiters spend 6-7 seconds scanning a resume. Large companies receive thousands of applications
per role, making manual screening impractical. This project builds an automated resume screening and
ranking system using NLP — matching resumes to job descriptions using semantic similarity — and
extracting structured candidate profiles from unstructured PDF/Word resumes.

2 Objectives
▸ Parse and extract structured data from PDF and Word resumes
▸ Compute semantic similarity between resumes and job descriptions
▸ Rank candidates by job-match score using TF-IDF and sentence transformers
▸ Identify skill gaps and missing qualifications
▸ Build a recruiter dashboard for candidate shortlisting

3 Real-World Applications
▸ Corporate HR teams automating initial screening of mass applications
▸ Recruitment agencies ranking candidates for multiple client roles
▸ Job portals personalising job recommendations to candidates
▸ Applicant Tracking Systems (ATS) supplementing rule-based filters with ML

4 Suggested Datasets
▸ Kaggle: 'Resume Dataset' (2,484 resumes, 25 categories)
▸ Kaggle: '[Link]'
▸ Kaggle: 'Job Description Dataset'
▸ Custom: collect 50 real job descriptions + 200 anonymised resumes

5 Recommended Tech Stack

Python spaCy pdfplumber / python-docx Sentence-
PyMuPDF Transformers
Scikit-learn Streamlit FastAPI

6 System Architecture & Pipeline

📄 Raw Resumes (PDF/DOCX) + Job Descriptions
↓
📑 Resume Parser (text extraction + structure identification)

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 37

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
🔍 Entity Extraction (Name, Skills, Education, Experience)
↓
📐 Embedding: TF-IDF / Sentence-Transformer
↓
📊 Similarity Scoring vs. Job Description
↓
🏆 Candidate Ranking + Gap Analysis
↓
Recruiter Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Two tasks: (1) Information Extraction — parse resume into structured fields. (2) Matching — score each
resume against a job description.
Phase 2: Dataset Collection
Kaggle Resume Dataset: 2,484 resumes across 25 job categories (HR, Engineer, Data Science, etc.).
Text-extracted resumes in CSV.
Phase 3: Resume Parsing
pdfplumber for PDF text extraction. python-docx for Word files. Use regex patterns to identify sections:
EDUCATION (pattern: '[Link]', 'B.E.', 'MBA'), EXPERIENCE (year ranges), SKILLS (keyword list
matching).
Phase 4: Entity Extraction with spaCy
Train a custom spaCy NER model to extract: PERSON, ORG (company), GPE (location), DATE
(employment dates), SKILL (from a curated skill taxonomy). Use spaCy's prodigy annotation or pre-
labelled dataset.
Phase 5: TF-IDF Matching
Combine all resume text. TF-IDF vectorise. Compute cosine similarity between job description vector
and each resume vector. Rank by similarity score.
Phase 6: Sentence Transformer Matching
Use 'all-MiniLM-L6-v2' from sentence-transformers. Encode JD and all resumes into dense vectors.
Compute cosine similarity. This captures semantic matches (e.g., 'Python' ↔ 'programming in Python').
Phase 7: Skill Gap Analysis
Extract required skills from JD. Extract candidate skills. Compute: matched_skills, missing_skills,
bonus_skills. Compute skill_coverage_score = matched / required.
Phase 8: Scoring
final_score = 0.5 × semantic_sim + 0.3 × skill_coverage + 0.2 × experience_years_score. Rank all
candidates by final_score.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 38

Master Project Roadmap — Batch 2: Machine Learning Projects

Phase 9: Dashboard
Upload JD → upload resumes → see ranked candidates table with scores, skill match radar, and top 3
reasons for ranking. One-click shortlist to interview.

8 Algorithms & Models

▸ TF-IDF cosine similarity for resume-JD matching
▸ Sentence-BERT (all-MiniLM-L6-v2) semantic similarity
▸ spaCy NER for entity extraction
▸ Regex-based section parsing
▸ Weighted multi-factor scoring
▸ K-Means clustering for candidate grouping by profile type

9 Evaluation Metrics
Precision@K (top-K contains Recall@K NDCG@K
relevant candidates)

Parsing Accuracy (manually Skill Extraction F1 Score Mean Reciprocal Rank (MRR)
validated on 50 resumes)

10 Expected Output

A recruiter-facing platform where hiring managers upload a job description and a batch of
resumes — and within seconds receive a ranked shortlist with skill match scores, gap
analysis, and experience summary for each candidate — reducing manual screening time
from hours to minutes.

11 Optional Advanced Enhancements

▸ Add LLM-powered resume parsing using GPT-4 structured output
▸ Build a bias detection layer: flag when gender/age/ethnicity features influence ranking
▸ Add video interview AI assessment module
▸ Build a job-to-candidate recommendation engine for job portals

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 39

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Add A/B testing framework to validate model improvements in production

▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 40

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 10 of 30 · Machine Learning

10. News Topic Classification

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
News publishers produce thousands of articles daily across hundreds of topics. Automatic topic
classification enables smart routing, personalised feeds, content tagging, and trend tracking. This
project builds a multi-class news topic classifier covering 20+ categories using classical NLP and
transformer models.

2 Objectives
▸ Build a 20+ class news topic classifier
▸ Compare TF-IDF + ML vs. BERT-based classification accuracy
▸ Handle class imbalance across news categories
▸ Enable hierarchical classification (broad category → sub-topic)
▸ Build a real-time news tagging pipeline

3 Real-World Applications
▸ News aggregators (Google News, Flipboard) auto-tagging articles
▸ Content management systems auto-routing articles to editors
▸ Financial trading systems classifying market-moving news by sector
▸ Media monitoring platforms tracking coverage by topic

4 Suggested Datasets
▸ Kaggle: 'News Category Dataset' (HuffPost, 200k+ articles, 42 categories)
▸ BBC News Dataset (2,225 articles, 5 categories — clean benchmark)
▸ HuggingFace: 'ag_news' (120k articles, 4 classes)
▸ HuggingFace: 'dbpedia_14' (560k articles, 14 classes)

5 Recommended Tech Stack

Python Pandas NLTK Scikit-learn Transformers
(BERT)
FastAPI Streamlit

6 System Architecture & Pipeline

📥 News Articles (headline + description + category)
↓
🧹 Text Cleaning & Preprocessing

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 41

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
📐 Feature Extraction: TF-IDF / BERT embeddings
↓
🤖 Multi-class Classification
↓
📊 Evaluate per-class Performance
↓
🚀 Real-time Tagging API

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Multi-class classification (not binary). 42 categories in HuffPost → consider merging similar categories
(e.g., 'ARTS' + 'ARTS & CULTURE' → 'Arts'). Reduce to 20 clean classes.
Phase 2: Dataset Collection
HuffPost News Category: 209,527 articles, 42 categories. Columns: category, headline, authors, link,
short_description, date.
Phase 3: Data Cleaning
Combine headline and short_description into text. Map 42 raw categories to 20 clean categories.
Balance classes: downsample overrepresented to 5,000 each.
Phase 4: TF-IDF Baseline
TF-IDF(max_features=100000, ngram_range=(1,2)). LogisticRegression(multi_class='multinomial').
LinearSVC. Expected: 75-82% accuracy.
Phase 5: BERT Classification
DistilBERT or BERT-base-uncased. AutoModelForSequenceClassification(num_labels=20). Fine-tune
3 epochs on headline+description. Expected: 88-93% accuracy.
Phase 6: Hierarchical Classification
Level 1: 5 broad categories (Politics, Business, Entertainment, Sports, Science). Level 2: sub-
categories within each broad class. Train a two-stage classifier.
Phase 7: Evaluation
Confusion matrix (20×20). Per-class F1 (identify hard-to-classify topics). Macro vs. micro-averaged F1.
Most confused class pairs.
Phase 8: API & Dashboard
FastAPI: POST /classify → input article text → returns top-3 categories with confidence. Streamlit:
news feed with auto-tagged articles. Filter news by topic.
Phase 9: Continuous Learning
Add feedback endpoint: users can correct labels. Store corrections in retraining queue. Retrain model
weekly.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 42

Master Project Roadmap — Batch 2: Machine Learning Projects

8 Algorithms & Models

▸ TF-IDF + Logistic Regression (multinomial)
▸ LinearSVC (one-vs-all)
▸ DistilBERT fine-tuning
▸ BERT + CNN head for faster inference
▸ Hierarchical multi-stage classification

9 Evaluation Metrics
Accuracy Macro F1 Score Per-class F1 Score

Confusion Matrix Top-3 Accuracy Inference Latency (ms per article)

10 Expected Output

A news classification API that tags any article with its topic category (and confidence
score) within 200ms — enabling news platforms to auto-route content, power personalised
feeds, and track editorial coverage patterns across topics in real time.

11 Optional Advanced Enhancements

▸ Add multi-label classification (an article can belong to multiple topics)
▸ Build a news clustering system for grouping related articles on the same event
▸ Add trend detection: identify emerging topics from classification frequency changes
▸ Train a zero-shot classifier to handle new topic categories without retraining

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 43

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 11 of 30 · Machine Learning

11. Face Mask Detection

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
During the COVID-19 pandemic, compliance monitoring for face mask usage became critical in public
spaces, offices, and healthcare facilities. This project builds a real-time face mask detection system
using computer vision — combining face detection with a binary classifier to determine mask presence
— deployable on edge devices and webcam streams.

2 Objectives
▸ Train a CNN classifier to distinguish masked vs. unmasked faces
▸ Integrate with a face detection pipeline (MTCNN or OpenCV Haar cascades)
▸ Achieve > 98% accuracy on diverse mask types (surgical, N95, cloth)
▸ Deploy real-time detection on webcam video stream
▸ Build a compliance monitoring dashboard

3 Real-World Applications
▸ Workplace safety compliance monitoring in factories and offices
▸ Hospital entrance screening for PPE compliance
▸ Public transport mask mandate enforcement cameras
▸ Retail entry control systems during health emergencies

4 Suggested Datasets
▸ Kaggle: 'Face Mask Detection Dataset' (853 images, 3 classes: with_mask, without_mask,
mask_weared_incorrect)
▸ Kaggle: 'Face Mask Detection' (7,553 annotated images)
▸ Kaggle: 'COVID Face Mask Detection Dataset'
▸ Custom: augment with SMOTE-equivalent image augmentation

5 Recommended Tech Stack

Python TensorFlow / Keras OpenCV MobileNetV2 MTCNN
NumPy Streamlit FastAPI

6 System Architecture & Pipeline

📷 Webcam / Image Input
↓
🔍 Face Detection (MTCNN / OpenCV Haar Cascade)

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 44

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
✂️Face Region Crop + Resize (224×224)
↓
🤖 CNN Classifier (MobileNetV2 fine-tuned)
↓
Prediction: Mask / No Mask / Incorrect Mask
↓
Bounding Box + Label Overlay
↓
📊 Real-time Compliance Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Two-stage pipeline: (1) Detect faces in image. (2) For each face, classify mask status. 3 classes:
with_mask, without_mask, mask_weared_incorrect.
Phase 2: Dataset Collection
Kaggle Face Mask Dataset: 853 images annotated with bounding boxes (XML format). Parse using
[Link]. Classes: with_mask, without_mask, mask_weared_incorrect.
Phase 3: Data Augmentation
ImageDataGenerator: rotation_range=20, zoom_range=0.15, shear_range=0.15, horizontal_flip=True,
brightness_range=[0.75, 1.25]. Augment to balance 3 classes to 3,000 images each.
Phase 4: Face Detection Integration
OpenCV: [Link]('haarcascade_frontalface_default.xml'). For each detected face ROI:
resize to 224×224. Feed to classifier.
Phase 5: CNN Architecture
Base: MobileNetV2 (ImageNet weights, include_top=False). Add: GlobalAveragePooling2D →
Dense(128, relu) → Dropout(0.5) → Dense(3, softmax). Fine-tune top 20 layers.
Phase 6: Training
Adam(lr=1e-4). Loss: categorical_crossentropy. Epochs: 20. EarlyStopping(patience=5).
ReduceLROnPlateau. Batch size: 32. Train/val/test: 70/15/15.
Phase 7: Evaluation
Classification report (precision, recall, F1 per class). Confusion matrix. Grad-CAM visualisation to verify
model is looking at mouth-nose region.
Phase 8: Real-time Inference
OpenCV VideoCapture(0) for webcam. For each frame: detect faces → crop → classify → draw
bounding box + label (green=masked, red=unmasked). FPS counter.
Phase 9: Dashboard

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 45

Master Project Roadmap — Batch 2: Machine Learning Projects

Streamlit: upload image → see detections. Real-time webcam tab. Compliance metrics: % compliant,
frame-by-frame trend.

8 Algorithms & Models

▸ MobileNetV2 (transfer learning, ImageNet)
▸ OpenCV Haar Cascade face detector
▸ MTCNN (Multi-task Cascaded CNN) face detector
▸ Grad-CAM (Gradient-weighted Class Activation Mapping)
▸ ImageDataGenerator augmentation

9 Evaluation Metrics
Classification Accuracy Per-class F1 Score Confusion Matrix

Inference FPS (frames per Detection Recall (% of faces Grad-CAM Region Relevance
second) detected)

10 Expected Output

A real-time face mask detection system that processes webcam or CCTV feeds, detects all
faces in each frame, classifies mask status in < 30ms per face, draws colour-coded
bounding boxes, and logs compliance rates over time — production-ready for deployment
in office buildings or public venues.

11 Optional Advanced Enhancements

▸ Upgrade to YOLOv8 for end-to-end face mask object detection (faster and more accurate)
▸ Add social distancing detection using pose estimation
▸ Deploy on Raspberry Pi for edge computing in camera hardware
▸ Build a multi-camera dashboard aggregating compliance rates from multiple feeds

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 46

Master Project Roadmap — Batch 2: Machine Learning Projects

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 47

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 12 of 30 · Machine Learning

12. Handwritten Digit Recognition

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Handwritten digit recognition is the 'Hello World' of computer vision and deep learning. Despite being a
well-studied problem, this project goes beyond simple MNIST training to build a robust, production-
quality digit recognition system with data augmentation, architecture comparison, interpretability, and a
real-time drawing canvas application.

2 Objectives
▸ Train CNN architectures on MNIST to achieve > 99% accuracy
▸ Compare LeNet-5, VGG-style, and ResNet-style architectures
▸ Handle the real-world domain gap (user-drawn vs. MNIST digits)
▸ Implement Grad-CAM for visual interpretability
▸ Deploy an interactive digit drawing canvas application

3 Real-World Applications
▸ Bank cheque amount reading automation
▸ Postal code recognition for mail sorting systems
▸ Educational tools for learning handwriting recognition
▸ Medical form and prescription digit reading

4 Suggested Datasets
▸ MNIST (70,000 images, 28×28, 10 classes) — built into TensorFlow/Keras
▸ Kaggle: 'Digit Recognizer' competition dataset
▸ EMNIST (extended MNIST: letters + digits, 814,255 characters)
▸ Custom: collect hand-drawn digits from multiple users to test domain gap

5 Recommended Tech Stack

Python TensorFlow / Keras PyTorch NumPy OpenCV
Matplotlib Streamlit (canvas
component)

6 System Architecture & Pipeline

✍️Handwritten Digit Image (28×28 greyscale)
↓
🔧 Preprocessing: Normalise, Threshold, Centre

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 48

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
🤖 CNN Classifier (multiple architectures)
↓
📊 Evaluate on MNIST Test Set
↓
🔍 Grad-CAM Visualisation
↓
🎨 Live Drawing Canvas Application

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

10-class classification (0–9). Key challenge: the domain gap between clean MNIST and real user
drawings. The system must handle stroke width variation, rotation, and noise.
Phase 2: Dataset Loading
[Link].load_data(). Shape: (60000, 28, 28). Reshape to (60000, 28, 28, 1). Normalise
to [0,1]. One-hot encode labels.
Phase 3: Data Augmentation
ImageDataGenerator: rotation_range=10, zoom_range=0.1, width_shift_range=0.1,
height_shift_range=0.1. This improves generalisation to user-drawn digits.
Phase 4: Architecture 1: LeNet-5
Conv(6,5×5,relu) → AvgPool → Conv(16,5×5,relu) → AvgPool → Flatten → Dense(120) → Dense(84)
→ Dense(10,softmax). Train as reference implementation.
Phase 5: Architecture 2: Modern CNN
Conv(32,3×3) → BN → Conv(32,3×3) → MaxPool → Dropout(0.25) → Conv(64,3×3) → BN →
Conv(64,3×3) → MaxPool → Dropout(0.25) → Flatten → Dense(512) → Dropout(0.5) → Dense(10).
Target: 99.4%+.
Phase 6: Architecture 3: ResNet (optional)
Build a mini ResNet-9 with skip connections. Compare training speed and accuracy.
Phase 7: Training Protocol
Adam(lr=1e-3). CosineAnnealingLR scheduler. EarlyStopping(patience=5). Batch size 128. 50 epochs.
Save best model checkpoint.
Phase 8: Evaluation
Test accuracy. Confusion matrix (find most confused digit pair: usually 4↔9, 3↔8). Per-digit accuracy.
Grad-CAM heatmaps on misclassified examples.
Phase 9: Interactive Canvas App
Streamlit with streamlit-drawable-canvas component. User draws a digit → canvas captures → resize
to 28×28 → preprocess (invert, threshold) → model predict → show digit + confidence bars.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 49

Master Project Roadmap — Batch 2: Machine Learning Projects

8 Algorithms & Models

▸ LeNet-5 (convolutional neural network)
▸ Modern CNN with Batch Normalisation and Dropout
▸ ResNet with skip connections
▸ Grad-CAM (Gradient-weighted Class Activation Mapping)
▸ ImageDataGenerator augmentation

9 Evaluation Metrics
Test Accuracy (%) Per-class Accuracy Confusion Matrix

Top-5 Accuracy Model Parameters Count Inference Time (ms)

Grad-CAM Localisation Quality

10 Expected Output

An interactive digit recognition app where users draw any digit on a canvas and the model
predicts it in real time with confidence scores. Extended to recognise multi-digit sequences
(e.g., reading a cheque amount) — demonstrating both high accuracy on MNIST and
reasonable generalisation to user-drawn input.

11 Optional Advanced Enhancements

▸ Extend to EMNIST for letter recognition
▸ Build a CTC-based sequence recogniser for multi-digit strings
▸ Deploy to TensorFlow Lite for mobile app integration
▸ Add adversarial example generation to test model robustness

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 50

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 13 of 30 · Machine Learning

13. Traffic Sign Recognition

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Traffic sign recognition is a safety-critical component of autonomous driving systems. This project
builds a robust multi-class traffic sign classifier using deep learning — handling real-world challenges
including varying lighting conditions, partial occlusion, motion blur, and class imbalance across 43 sign
types.

2 Objectives
▸ Train a CNN on GTSRB dataset to achieve > 99% accuracy on 43 traffic sign classes
▸ Handle class imbalance (some signs appear 2,000× more than others)
▸ Apply aggressive augmentation to simulate real driving conditions
▸ Implement model calibration for uncertainty-aware predictions
▸ Build a dashcam video analysis demo

3 Real-World Applications
▸ Autonomous vehicle perception systems (Tesla, Waymo)
▸ Advanced Driver Assistance Systems (ADAS)
▸ Traffic sign inventory management for road authorities
▸ Driver training simulators for sign recognition testing

4 Suggested Datasets
▸ GTSRB — German Traffic Sign Recognition Benchmark (51,839 images, 43 classes)
▸ Kaggle: 'GTSRB — German Traffic Sign Recognition Benchmark'
▸ Belgian Traffic Sign Dataset
▸ US Traffic Sign Dataset (LISA)

5 Recommended Tech Stack

Python TensorFlow / Keras PyTorch OpenCV NumPy
Matplotlib Streamlit

6 System Architecture & Pipeline

🚦 Traffic Sign Image Input
↓
🔧 Preprocess: Resize (32×32), Histogram Equalise, Normalise
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 51

Master Project Roadmap — Batch 2: Machine Learning Projects

📈 Augmentation: Rotation, Brightness, Zoom, Blur

↓
🤖 CNN (VGG-style / ResNet)
↓
⚖️Class Weighting for Imbalance
↓
📊 Evaluate on GTSRB Test Set
↓
📹 Video Stream Demo

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

43-class classification. Class imbalance: 'Speed Limit 50' has 2,250 training samples; 'Dangerous
Curve Left' has 210. Safety context: false positives on stop signs are critical.
Phase 2: Dataset Loading
GTSRB: 51,839 images in train (39,209) and test (12,630) sets. Images vary in size and lighting. Use
CSV to map image filenames to class labels.
Phase 3: Preprocessing Pipeline
Resize all images to 32×32 (or 64×64). Apply CLAHE (Contrast Limited Adaptive Histogram
Equalisation) using OpenCV to normalise lighting. Normalise pixel values to [0,1].
Phase 4: Data Augmentation
Random rotation: ±15°. Random brightness/contrast. Gaussian noise. Zoom: 0.8–1.2×. Slight shear.
Perspective transform (simulate viewing angle). This is critical for generalisation.
Phase 5: Class Weighting
Compute class_weight = {class: total_samples / (n_classes × class_count)}. Pass to
[Link](class_weight=cw). This penalises misclassification of rare signs more.
Phase 6: Model Architecture
VGG-style CNN: [Conv32→Conv32→MaxPool→Dropout] × 2 →
[Conv64→Conv64→MaxPool→Dropout] × 2 → Flatten → Dense512 → Dropout0.5 → Dense43. Or
use ResNet-18 pretrained on ImageNet, fine-tune last 2 blocks.
Phase 7: Training
Adam(1e-3) → ReduceLROnPlateau. Batch 64. 30 epochs. EarlyStopping. Data augmentation via
flow_from_dataframe.
Phase 8: Evaluation
Per-class accuracy (identify weakest classes). Confusion matrix. Grad-CAM on misclassified samples.
Model calibration: reliability diagram (confidence vs. actual accuracy).
Phase 9: Deployment

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 52

Master Project Roadmap — Batch 2: Machine Learning Projects

Streamlit: upload image → see predicted sign + confidence. Video stream: process dashcam footage
frame-by-frame, overlay sign predictions.

8 Algorithms & Models

▸ VGG-style CNN (deep convolutional network)
▸ ResNet-18 (transfer learning)
▸ CLAHE preprocessing
▸ Class-weighted loss function
▸ Grad-CAM
▸ Test-Time Augmentation (TTA) for inference reliability

9 Evaluation Metrics
Top-1 Accuracy Per-class Recall (critical for Confusion Matrix (43×43)
safety)

Expected Calibration Error (ECE) Inference FPS mAP if detection (not just
classification) is added

10 Expected Output

A traffic sign recognition system achieving > 99% accuracy on the GTSRB benchmark,
capable of processing dashcam video in real time (> 30 FPS on GPU), with calibrated
confidence scores and Grad-CAM visualisations showing what the model focuses on —
suitable as a component of an ADAS proof-of-concept.

11 Optional Advanced Enhancements

▸ Extend to traffic sign detection + recognition using YOLOv8
▸ Add domain adaptation for US/Indian traffic signs
▸ Implement uncertainty quantification using Monte Carlo Dropout
▸ Build a model robustness tester with adversarial examples (FGSM attack)

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 53

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 54

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 14 of 30 · Machine Learning

14. Object Detection using YOLO

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Object detection is one of the most impactful computer vision tasks, powering self-driving cars, security
cameras, retail automation, and medical imaging. This project trains and deploys a YOLOv8 object
detection model for a custom domain (security, retail, or industrial), covering annotation, training,
evaluation with mAP, and real-time video deployment.

2 Objectives
▸ Train YOLOv8 on a custom domain-specific dataset
▸ Understand and implement YOLO data format and annotation pipeline
▸ Evaluate using mAP@0.5 and mAP@0.5:0.95 metrics
▸ Deploy real-time inference on webcam and video files
▸ Compare YOLOv8 (n/s/m) model variants for speed-accuracy trade-off

3 Real-World Applications
▸ Retail shelf monitoring: detect out-of-stock or misplaced products
▸ Security camera systems: detect people, weapons, or suspicious objects
▸ Industrial quality control: detect defects on manufacturing lines
▸ Medical imaging: detect and localise lesions in X-rays

4 Suggested Datasets
▸ Roboflow: multiple annotated datasets in YOLO format (100+ free datasets)
▸ Kaggle: 'COCO 2017 Object Detection' (80 classes, 328k images)
▸ Kaggle: 'Open Images V7' (600 classes)
▸ Custom: annotate domain-specific images using Roboflow or LabelImg

5 Recommended Tech Stack

Python Ultralytics YOLOv8 OpenCV Roboflow NumPy
Streamlit FastAPI

6 System Architecture & Pipeline

📸 Images + Annotations (YOLO format: .txt bounding boxes)
↓
🔧 Dataset: train/val/test split (70/20/10)
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 55

Master Project Roadmap — Batch 2: Machine Learning Projects

📈 Augmentation (Mosaic, MixUp, Flip, HSV)

↓
🤖 YOLOv8 Training (yolov8n/s/m)
↓
📊 Evaluate: mAP@0.5, mAP@0.5:0.95
↓
📹 Real-time Video Inference
↓
🚀 FastAPI Detection Endpoint

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Object detection: simultaneously localise (bounding box) and classify (class label) multiple objects in an
image. YOLO (You Only Look Once) does this in a single forward pass.
Phase 2: Dataset Selection
Option A: Use a pre-annotated Roboflow dataset for your domain (e.g., 'Construction Site Safety' or
'Retail Products'). Option B: Collect 500+ images and annotate using LabelImg.
Phase 3: Annotation Format
YOLO format: one .txt per image. Each line: class_id cx cy w h (normalised 0-1). Prepare [Link]:
train/val/test paths, nc (num classes), names list.
Phase 4: Data Augmentation
Ultralytics applies: Mosaic, RandomFlip, HSV shifts, RandomPerspective, MixUp. Configure in
[Link] or pass via training arguments.
Phase 5: Training
from ultralytics import YOLO. model = YOLO('[Link]'). [Link](data='[Link]',
epochs=100, imgsz=640, batch=16). Use pretrained weights for transfer learning.
Phase 6: Evaluation
[Link]() → returns mAP50, mAP50-95, precision, recall, F1 per class. Plot PR curves. Confusion
matrix.
Phase 7: Speed-Accuracy Trade-off
Compare YOLOv8n (nano): fastest. YOLOv8s (small): balanced. YOLOv8m (medium): best accuracy.
Measure FPS on CPU and GPU.
Phase 8: Real-time Inference
results = model(source=0, stream=True). Loop through results. Draw bounding boxes using OpenCV.
Calculate FPS. Stream to Streamlit video display.
Phase 9: FastAPI Deployment
POST /detect: accepts image upload → returns JSON with detected objects (class, confidence, bbox
coordinates). Include visual: return base64-encoded annotated image.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 56

Master Project Roadmap — Batch 2: Machine Learning Projects

8 Algorithms & Models

▸ YOLOv8 (You Only Look Once, version 8)
▸ Mosaic and MixUp data augmentation
▸ Non-Maximum Suppression (NMS)
▸ CIoU loss function
▸ Transfer learning from COCO pretrained weights

9 Evaluation Metrics
mAP@0.5 (mean Average mAP@0.5:0.95 (COCO standard Per-class AP
Precision at IoU 0.5) metric)

Precision / Recall F1 Score FPS (inference speed on

CPU/GPU)

10 Expected Output

A domain-specific object detection system that detects and localises multiple object types
in real time at > 30 FPS on GPU, with a FastAPI endpoint for image-based detection and a
Streamlit demo showing live webcam object detection with labelled bounding boxes.

11 Optional Advanced Enhancements

▸ Add instance segmentation using YOLOv8-seg for pixel-level masks
▸ Implement object tracking across video frames (ByteTrack / DeepSORT)
▸ Deploy to ONNX format for edge device inference
▸ Add active learning loop: automatically flag low-confidence detections for re-annotation

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 57

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 15 of 30 · Machine Learning

15. Plant Disease Detection

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Crop diseases cause 20-40% of global food production losses annually. Early detection enables
targeted treatment, reducing chemical use and saving harvests. This project builds a plant disease
classification system using transfer learning on leaf images — deployable on a mobile device for use by
farmers in the field.

2 Objectives
▸ Train a CNN to classify 38 plant disease classes across 14 crop species
▸ Apply transfer learning with VGG16, ResNet50, and EfficientNet
▸ Handle real-world challenges: similar disease symptoms, lighting variation
▸ Build a mobile-friendly web app for in-field diagnosis
▸ Generate Grad-CAM visualisations showing diseased leaf regions

3 Real-World Applications
▸ Agricultural advisory apps for smallholder farmers
▸ Precision agriculture platforms for large-scale farm monitoring
▸ Agricultural extension services providing remote diagnosis
▸ Crop insurance underwriting based on disease risk assessment

4 Suggested Datasets
▸ PlantVillage Dataset (54,306 images, 38 classes, 14 crop species) — Kaggle/HuggingFace
▸ Kaggle: 'New Plant Diseases Dataset' (87,000 images)
▸ Kaggle: 'Plant Disease Recognition Dataset'
▸ CGIAR — AI for Agriculture datasets (real field images, not lab conditions)

5 Recommended Tech Stack

Python TensorFlow / Keras PyTorch EfficientNet OpenCV
Streamlit FastAPI

6 System Architecture & Pipeline

🌿 Leaf Image Input
↓
🔧 Resize (224×224), Normalise, Augment
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 58

Master Project Roadmap — Batch 2: Machine Learning Projects

🤖 Transfer Learning: VGG16 / ResNet50 / EfficientNetB3

↓
Fine-tune top layers
↓
📊 38-class Evaluation
↓
🔍 Grad-CAM Heatmap of Disease Region
↓
📱 Mobile Web App

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

38-class classification: 26 disease classes + 12 healthy classes across 14 crops. Examples: Tomato
Bacterial Spot, Corn Rust, Apple Scab, Grape Black Rot.
Phase 2: Dataset Loading
PlantVillage: 54,306 colour images in 38 subdirectories. ImageDataGenerator.flow_from_directory().
80/10/10 train/val/test split.
Phase 3: Augmentation
Random horizontal/vertical flip. Rotation ±30°. Brightness ±40%. Random zoom 0.8–1.2×. Shear.
These simulate real field photography conditions.
Phase 4: Transfer Learning: EfficientNetB3
base = EfficientNetB3(weights='imagenet', include_top=False). Add: GlobalAveragePooling →
Dense(256,relu) → Dropout(0.4) → Dense(38,softmax). Freeze base initially.
Phase 5: Two-stage Training
Stage 1: Train only top layers, 10 epochs, Adam(1e-3). Stage 2: Unfreeze last 30 layers, 20 epochs,
Adam(1e-5). This prevents catastrophic forgetting.
Phase 6: Evaluation
Top-1 and Top-3 accuracy. Per-class F1 (find hard classes). Confusion matrix. Compare VGG16 vs.
ResNet50 vs. EfficientNetB3 on same val set.
Phase 7: Grad-CAM
Target layer: last convolutional block. Overlay heatmap on original image. Verify model focuses on
lesion areas (not background or pot).
Phase 8: Real-World Testing
Test on self-photographed plant images (not from PlantVillage). Expected accuracy drop: 20-30%
(domain gap). Document and analyse failures.
Phase 9: Mobile App
Streamlit: drag-and-drop leaf photo → prediction + confidence + Grad-CAM. FastAPI: POST /diagnose
→ returns top-3 diseases + treatment recommendations.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 59

Master Project Roadmap — Batch 2: Machine Learning Projects

8 Algorithms & Models

▸ EfficientNetB3 (transfer learning)
▸ VGG16 (transfer learning)
▸ ResNet50 (transfer learning)
▸ Grad-CAM (disease region localisation)
▸ Test-Time Augmentation (TTA)
▸ Two-stage fine-tuning strategy

9 Evaluation Metrics
Top-1 Accuracy Top-3 Accuracy Per-class F1 (38-class)

Macro F1 Score Confusion Matrix Grad-CAM Relevance (qualitative)

Cross-domain accuracy (real field

photos)

10 Expected Output

A plant disease diagnosis app where a farmer can photograph a diseased leaf, upload it,
and instantly receive the disease name, confidence score, affected region heatmap, and
treatment recommendations — reducing diagnosis time from days (waiting for an
agronomist) to seconds.

11 Optional Advanced Enhancements

▸ Train on real-field images (CGIAR dataset) to close the domain gap
▸ Add disease severity scoring (0-5 scale based on lesion coverage)
▸ Build a disease progression tracker: photograph same plant over time
▸ Deploy TensorFlow Lite model for offline mobile use without internet

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 60

Master Project Roadmap — Batch 2: Machine Learning Projects

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 61

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 16 of 30 · Machine Learning

16. Diabetes Prediction System

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Type 2 diabetes affects 537 million adults globally, yet up to 50% are undiagnosed. Early prediction
using health screening data can enable timely intervention, lifestyle changes, and treatment. This
project builds a clinical decision support tool for diabetes risk prediction using machine learning on
patient biomarker data, with full explainability for clinical use.

2 Objectives
▸ Build a diabetes risk classifier from patient health metrics
▸ Compare logistic regression, random forest, XGBoost, and neural network models
▸ Apply SHAP for clinical explainability ('Why did the model predict diabetes?')
▸ Handle missing values and data quality issues in clinical data
▸ Build a clinical decision support interface for healthcare providers

3 Real-World Applications
▸ Primary care screening tools for diabetes risk assessment
▸ Population health management programmes in insurance companies
▸ Hospital EMR systems flagging high-risk patients for follow-up
▸ Digital health apps for personalised diabetes prevention coaching

4 Suggested Datasets
▸ Pima Indians Diabetes Dataset (768 patients, 8 features) — UCI/Kaggle
▸ Kaggle: 'Diabetes Health Indicators Dataset' (BRFSS 2015, 253,680 records)
▸ Kaggle: 'Early Stage Diabetes Risk Prediction Dataset'
▸ CDC Diabetes Health Indicators Dataset

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost SHAP
Matplotlib Streamlit FastAPI

6 System Architecture & Pipeline

📋 Patient Health Data (BMI, Glucose, BP, Insulin, Age, etc.)
↓
🧹 Handle Missing Values (0-value imputation in clinical data)
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 62

Master Project Roadmap — Batch 2: Machine Learning Projects

🔧 Feature Engineering + Scaling

↓
⚖️Handle Class Imbalance
↓
🤖 Train: Logistic Reg, RF, XGBoost, MLP
↓
📊 Evaluate with Clinical Metrics
↓
🔍 SHAP Explanation per Patient
↓
🏥 Clinical Decision Support Interface

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Binary: Diabetic (1) vs. Non-Diabetic (0). Key features: Glucose, BMI, Age, Insulin, BloodPressure,
SkinThickness, DiabetesPedigreeFunction, Pregnancies. Clinical requirement: high recall (catch most
diabetics).
Phase 2: Data Cleaning
Pima dataset: 0 values in Glucose, BloodPressure, SkinThickness, Insulin, BMI are physiologically
impossible → replace with NaN → impute with median grouped by outcome. This is clinically important.
Phase 3: EDA
Box plots: all features by Outcome. Correlation heatmap. Glucose distribution (bimodal for diabetics vs.
non-diabetics). Pairplot with colour by Outcome.
Phase 4: Feature Engineering
Glucose_BMI_interaction = Glucose × BMI. Age_groups bins. Insulin_resistance_index =
Glucose/Insulin (where Insulin>0). Normalise all features with StandardScaler.
Phase 5: Logistic Regression Baseline
LogisticRegression(C=0.1). Evaluate: Accuracy, Recall, Precision, F1, ROC-AUC. Coefficient plot for
feature importance.
Phase 6: Ensemble Models
RandomForestClassifier(n_estimators=200). XGBClassifier(n_estimators=200, max_depth=4).
Compare all. Tune threshold for highest recall with acceptable precision (≥ 70%).
Phase 7: Neural Network
MLPClassifier(hidden_layer_sizes=(64,32), activation='relu'). Compare to XGBoost.
Phase 8: SHAP Explainability
[Link](xgb_model). For each patient: waterfall plot showing which features push
prediction toward or away from diabetes. Force plot for single prediction.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 63

Master Project Roadmap — Batch 2: Machine Learning Projects

Phase 9: Clinical Interface

Streamlit: input sliders for all 8 features → prediction + probability + SHAP explanation. 'This patient
has 78% probability of diabetes. Primary drivers: Glucose=180 (+0.42), BMI=33 (+0.31).'

8 Algorithms & Models

▸ Logistic Regression (L2, L1 regularisation)
▸ Random Forest Classifier
▸ XGBoost Classifier
▸ MLPClassifier (feed-forward neural network)
▸ SHAP (TreeExplainer)
▸ SMOTE for class imbalance
▸ KNN Imputation for missing values

9 Evaluation Metrics
Recall (sensitivity) — primary for Specificity ROC-AUC
clinical use

Precision-Recall AUC F1 Score Brier Score (calibration)

Clinical Utility: Net Benefit at

decision threshold

10 Expected Output

A clinical decision support tool that a GP or nurse can use to enter a patient's routine
health measurements and receive an instant diabetes risk probability with a patient-specific
explanation of driving factors — enabling early intervention and reducing undiagnosed
diabetes rates.

11 Optional Advanced Enhancements

▸ Add retinal image analysis for diabetic retinopathy risk
▸ Build a longitudinal risk tracker: repeat risk assessment over multiple visits
▸ Integrate with EHR systems (HL7 FHIR API)
▸ Build a population risk stratification tool for insurance or public health bodies

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
AI & Data Science Student Projects · Batch 2: Machine Learning · Page 64
Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI

▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 65

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 17 of 30 · Machine Learning

17. Heart Disease Prediction

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Cardiovascular disease is the world's leading cause of death, responsible for 17.9 million deaths
annually. Many of these are preventable with early detection and risk management. This project builds
a heart disease risk prediction system using clinical features from the Cleveland Heart Disease dataset,
with emphasis on model interpretability and clinical decision support.

2 Objectives
▸ Build a binary heart disease classifier from clinical measurements
▸ Compare classical ML and neural network models
▸ Explain predictions at patient level using SHAP waterfall plots
▸ Perform feature selection to identify the most predictive clinical markers
▸ Build an interactive patient risk assessment tool

3 Real-World Applications
▸ Cardiologist decision support: flagging high-risk patients for further testing
▸ Telemedicine platforms providing remote cardiac risk assessment
▸ Life insurance underwriting for cardiovascular risk pricing
▸ Preventive healthcare programmes targeting high-risk populations

4 Suggested Datasets
▸ UCI: 'Heart Disease Dataset' (Cleveland, 303 records, 14 attributes) — canonical
▸ Kaggle: 'Heart Disease UCI'
▸ Kaggle: 'Heart Failure Prediction Dataset' (918 patients, 12 features)
▸ Kaggle: 'Cardiovascular Disease Dataset' (70,000 records)

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost SHAP
Seaborn Plotly Streamlit

6 System Architecture & Pipeline

📋 Patient Clinical Data (Age, Chest Pain, Cholesterol, etc.)
↓
🧹 Clean: Handle '?' values, encode categoricals
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 66

Master Project Roadmap — Batch 2: Machine Learning Projects

📊 EDA: Feature distributions by disease status

↓
🔍 Feature Selection (RFECV, Correlation)
↓
🤖 Train Multiple Classifiers
↓
📐 Cross-Validated Evaluation
↓
🔍 SHAP Analysis
↓
🏥 Patient Risk Tool

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Binary: Heart Disease (1) vs. No Disease (0). Key features: age, sex, cp (chest pain type), trestbps
(resting BP), chol (cholesterol), fbs (fasting blood sugar), restecg, thalach (max heart rate), exang,
oldpeak, slope, ca, thal.
Phase 2: Data Cleaning
Cleveland dataset: some values are '?'. Replace with NaN. Impute with median. Convert to correct
dtypes. Target: 'target' (0/1) or map num>0 to 1.
Phase 3: EDA
Age distribution by outcome. Chest pain type breakdown. Correlation heatmap. Thalach (max HR) is
highly predictive (lower in disease). Oldpeak distribution.
Phase 4: Feature Selection
Correlation filter: remove features with |corr| < 0.1 to target. RFECV with Logistic Regression: find
minimal feature set maintaining accuracy. Mutual information scores.
Phase 5: Model Training
LogisticRegression, KNeighborsClassifier, SVC, RandomForest, XGBoost, MLP. 10-fold stratified CV.
Report mean ± std ROC-AUC.
Phase 6: Hyperparameter Tuning
GridSearchCV on top 2 models. XGBoost: tune max_depth, learning_rate, n_estimators, subsample.
Phase 7: SHAP Analysis
Global: SHAP summary plot (beeswarm). Identify top 5 clinical risk factors. Patient-level: waterfall plot
for any test case. Interaction plots for age × thalach.
Phase 8: Model Calibration
CalibratedClassifierCV on best model. Reliability diagram. Platt scaling. Ensure predicted probabilities
are meaningful clinically.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 67

Master Project Roadmap — Batch 2: Machine Learning Projects

Phase 9: Clinical Interface

Streamlit input form: all 13 features. Output: risk probability gauge (0-100%), risk category
(Low/Medium/High), SHAP waterfall, recommended next steps.

8 Algorithms & Models

▸ Logistic Regression (clinical baseline)
▸ Random Forest
▸ XGBoost
▸ SVM (RBF kernel)
▸ K-Nearest Neighbours
▸ MLP Neural Network
▸ SHAP TreeExplainer
▸ RFECV Feature Selection

9 Evaluation Metrics
ROC-AUC (10-fold CV) Sensitivity (Recall) Specificity

F1 Score Brier Score Expected Calibration Error

95% Confidence Interval on all

metrics

10 Expected Output

A clinical heart disease risk assessment tool where any clinician can input a patient's
routine measurements (age, ECG, cholesterol, etc.) and receive a calibrated risk
probability with a patient-specific explanation of which factors are most driving the
prediction — ready for integration into a clinical workflow.

11 Optional Advanced Enhancements

▸ Add ECG signal analysis using 1D CNN for direct waveform-based risk scoring
▸ Build a risk factor modification simulator: 'If patient reduces cholesterol by 20%, risk drops by X
%'
▸ Integrate with wearable data (heart rate variability, activity levels)
▸ Add federated learning to train across hospitals without sharing patient data

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 68

Master Project Roadmap — Batch 2: Machine Learning Projects

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 69

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 18 of 30 · Machine Learning

18. Breast Cancer Detection

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Breast cancer is the most common cancer worldwide. Early detection dramatically improves survival
rates — from 99% (localised) to 27% (metastatic). This project builds a breast cancer classification
system using the Wisconsin Diagnostic Breast Cancer dataset, with full clinical explainability, and
optionally extends to histopathology image classification using deep learning.

2 Objectives
▸ Build a binary classifier: Malignant vs. Benign tumour from cell nucleus measurements
▸ Apply Recursive Feature Elimination to identify the most predictive features
▸ Achieve > 98% AUC with clinical-grade sensitivity
▸ Extend to histopathology image classification using CNN (optional)
▸ Build a pathologist decision support interface

3 Real-World Applications
▸ Radiologist second-opinion AI systems for mammography analysis
▸ Pathology labs automating biopsy slide classification
▸ Clinical trial patient stratification by cancer subtype
▸ Breast cancer screening programmes prioritising high-risk cases

4 Suggested Datasets
▸ UCI: 'Breast Cancer Wisconsin (Diagnostic)' (569 samples, 30 features) — canonical
▸ Kaggle: 'Breast Cancer Wisconsin' dataset
▸ Kaggle: 'Histopathologic Cancer Detection' (PCam, 220k histology patches)
▸ Kaggle: 'Breast Cancer Histopathological Image Classification' (BreaKHis)

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost SHAP
TensorFlow/Keras Streamlit
(for image
extension)

6 System Architecture & Pipeline

📋 Cell Nucleus Features (radius, texture, perimeter, area, etc.)
↓
🧹 Clean: Check distributions, remove correlated features

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 70

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
🔧 PCA visualisation (optional dimensionality reduction)
↓
🤖 Train: LR, SVM, RF, XGBoost
↓
📊 Clinical Evaluation (Sensitivity ≥ 95%)
↓
🔍 SHAP + Feature Importance
↓
🏥 Pathologist Support Interface

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Binary: Malignant (M→1) vs. Benign (B→0). 30 features: 10 cell nucleus measurements (radius,
texture, perimeter, area, smoothness, compactness, concavity, concave_points, symmetry,
fractal_dimension), each with mean, SE, and worst values.
Phase 2: Data Exploration
569 samples: 357 Benign (62.7%), 212 Malignant (37.3%). Class imbalance is mild. Plot feature
distributions by class. Correlation heatmap — many highly correlated features.
Phase 3: Feature Selection
Remove highly correlated features (|r| > 0.95). RFECV with SVM. Select top 10 features from 30. Verify
no significant accuracy drop.
Phase 4: Model Training
LR, SVM (RBF), RF, XGBoost. 10-fold stratified CV. Focus on: Sensitivity (recall for Malignant) ≥ 95%
constraint.
Phase 5: SVM Tuning
SVC(kernel='rbf'). GridSearchCV: C in [0.1, 1, 10, 100], gamma in ['scale', 'auto', 0.001, 0.01]. SVM
achieves highest AUC on this dataset (typically > 99%).
Phase 6: SHAP Analysis
SHAP KernelExplainer for SVM (slower but model-agnostic). Or use TreeExplainer for XGBoost. Global
summary + individual patient waterfall.
Phase 7: Threshold Optimisation
Sweep threshold to achieve Sensitivity ≥ 95%. Report specificity at this clinical threshold. This is the
medically required constraint.
Phase 8: Image Extension (Optional)
PCam histopathology dataset. CNN: EfficientNetB0 fine-tuned. Patch-level classification: tumour vs.
non-tumour. Aggregate patch predictions for slide-level diagnosis.
Phase 9: Interface

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 71

Master Project Roadmap — Batch 2: Machine Learning Projects

Streamlit: input 10 top features → risk probability + SHAP waterfall + recommendation ('Recommend
biopsy' or 'Continue screening').

8 Algorithms & Models

▸ SVM (RBF kernel) — best on this dataset
▸ Logistic Regression
▸ Random Forest
▸ XGBoost
▸ RFECV Feature Selection
▸ PCA for visualisation
▸ EfficientNetB0 for histopathology images

9 Evaluation Metrics
ROC-AUC Sensitivity (Recall for Malignant) Specificity
— clinical primary metric

Positive Predictive Value (PPV) Negative Predictive Value (NPV) F1 Score

Matthews Correlation Coefficient

10 Expected Output

A breast cancer risk classifier that achieves > 98% AUC and ≥ 95% sensitivity, with
patient-level SHAP explanations showing which cell measurements are most concerning
— deployable as a pathologist second-opinion tool to flag malignant cases for priority
review.

11 Optional Advanced Enhancements

▸ Add multiclass classification: benign, ductal carcinoma in situ (DCIS), invasive
▸ Extend to whole-slide image (WSI) analysis using attention-based MIL
▸ Add uncertainty quantification using Bayesian neural networks
▸ Build a treatment response predictor using genomic features

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 72

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Implement online learning for continuous model updates on new data

▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 73

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 19 of 30 · Machine Learning

19. Medical Image Classification

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Medical imaging is the fastest-growing application of AI in healthcare. This project builds a multi-domain
medical image classifier covering chest X-rays (pneumonia), skin lesions (melanoma), and retinal
images (diabetic retinopathy) — demonstrating transfer learning, multi-label classification, and the
unique challenges of medical AI including class imbalance, rare conditions, and regulatory
requirements.

2 Objectives
▸ Train CNNs for chest X-ray, skin lesion, and retinal image classification
▸ Apply DenseNet, EfficientNet, and ResNet for each domain
▸ Handle extreme class imbalance in medical datasets
▸ Implement Grad-CAM for clinical region-of-interest localisation
▸ Understand regulatory requirements for medical AI (FDA/CE marking context)

3 Real-World Applications
▸ Radiology AI: flagging pneumonia or lung nodules in X-rays
▸ Dermatology AI: screening skin lesions for malignancy
▸ Ophthalmology: automated diabetic retinopathy screening
▸ Pathology: cell/tissue classification in histology slides

4 Suggested Datasets
▸ Kaggle: 'Chest X-Ray Images (Pneumonia)' (5,856 images, 2 classes)
▸ Kaggle: 'ISIC 2019 Skin Lesion Analysis' (25,331 images, 9 classes)
▸ Kaggle: 'Diabetic Retinopathy Detection' (35,126 retinal images, 5 severity levels)
▸ NIH Chest X-ray 14 dataset (112,120 images, 14 diseases, multi-label)

5 Recommended Tech Stack

Python TensorFlow / Keras PyTorch DenseNet121 EfficientNetB4
OpenCV Streamlit

6 System Architecture & Pipeline

🩻 Medical Image Input (DICOM / JPEG)
↓
🔧 Domain-specific Preprocessing

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 74

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
📈 Aggressive Augmentation
↓
🤖 Transfer Learning (DenseNet/EfficientNet)
↓
⚖️Focal Loss for Class Imbalance
↓
📊 Clinical Evaluation
↓
🔍 Grad-CAM Heatmap
↓
🏥 Radiologist/Clinician Interface

7 Step-by-Step Implementation Roadmap

Phase 1: Task 1: Chest X-Ray

Binary: Normal vs. Pneumonia. 5,856 images. Class imbalance: 3,875 vs. 1,341. Use DenseNet121
(CheXNet paper). Focal Loss for imbalance. Target AUC > 0.95.
Phase 2: Task 2: Skin Lesion
9-class classification: Melanoma, Melanocytic Nevus, etc. Class imbalance is severe (melanoma: 4.5%
of dataset). Use EfficientNetB4. Focal Loss. ISIC 2019 standard evaluation.
Phase 3: Task 3: Diabetic Retinopathy
5-class ordinal: 0=No DR, 1=Mild, 2=Moderate, 3=Severe, 4=Proliferative. Ordinal regression loss.
ResNet50. Quadratic Weighted Kappa metric.
Phase 4: Preprocessing per Domain
X-ray: CLAHE enhancement, normalise by mean/std of ImageNet. Skin: circle crop, remove hair
artifacts. Retina: green channel emphasis, Ben Graham preprocessing (subtract local mean).
Phase 5: Augmentation
Heavy augmentation for all: rotation, brightness, contrast, horizontal flip, zoom. For skin lesion:
additionally simulate dermoscope artifacts.
Phase 6: Focal Loss
focal_loss = -α(1-p)^γ log(p). γ=2 (standard). Downweights easy examples, focuses on hard/rare cases.
Critical for class-imbalanced medical datasets.
Phase 7: Grad-CAM Analysis
For each model: visualise which regions activated the prediction. Verify: X-ray model → opacities
region; skin model → lesion area; retina model → microaneurysms region.
Phase 8: Evaluation

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 75

Master Project Roadmap — Batch 2: Machine Learning Projects

Chest X-ray: AUC. Skin: balanced accuracy + AUC per class. Retinal: Quadratic Weighted Kappa
(competition standard). Per-class sensitivity at 90% specificity.
Phase 9: Clinical Interface
Upload medical image → select domain → see prediction + confidence + Grad-CAM overlay +
recommendation. Include confidence threshold warning for uncertain cases.

8 Algorithms & Models

▸ DenseNet121 (CheXNet architecture for X-ray)
▸ EfficientNetB4 (skin lesion)
▸ ResNet50 (diabetic retinopathy)
▸ Focal Loss for class imbalance
▸ Grad-CAM
▸ Test-Time Augmentation (TTA)
▸ Label smoothing for calibration

9 Evaluation Metrics
ROC-AUC per class Sensitivity at 90% Specificity Quadratic Weighted Kappa
(clinical threshold) (retinopathy)

Balanced Accuracy mAP for multi-label tasks Expected Calibration Error

10 Expected Output

A multi-domain medical image classification platform covering 3 clinical domains — chest

X-ray, skin lesion, and retinal screening — each with Grad-CAM visualisations, calibrated
confidence scores, and clinical-grade sensitivity targets, demonstrating deep learning's
potential as a radiologist/clinician support tool.

11 Optional Advanced Enhancements

▸ Add DICOM format support for real hospital image integration
▸ Implement Monte Carlo Dropout for uncertainty quantification
▸ Build a multi-label model for NIH ChestX-ray14 (14 simultaneous conditions)
▸ Research FDA AI/ML Software as a Medical Device (SaMD) regulatory pathway

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 76

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Scale to production with MLflow model registry and CI/CD pipeline

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 77

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 20 of 30 · Machine Learning

20. Stock Price Prediction

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Stock price prediction sits at the intersection of time-series modelling, financial theory, and deep
learning. While price prediction is inherently uncertain (markets are partly efficient), this project builds a
systematic framework using technical indicators, LSTM networks, and attention mechanisms — with
emphasis on correct backtesting methodology and the Efficient Market Hypothesis as a theoretical
constraint.

2 Objectives
▸ Build LSTM and Transformer models for stock price sequence prediction
▸ Engineer technical indicators as model features (RSI, MACD, Bollinger Bands)
▸ Implement proper walk-forward validation (no data leakage)
▸ Evaluate using financial metrics: Sharpe Ratio, Maximum Drawdown
▸ Build a trading signal backtesting framework

3 Real-World Applications
▸ Algorithmic trading firms developing quantitative strategies
▸ Portfolio managers using ML signals as one input to investment decisions
▸ FinTech companies building robo-advisory products
▸ Financial research for academic study of market predictability

4 Suggested Datasets
▸ Yahoo Finance via yfinance library (free, any stock, any period)
▸ Kaggle: 'Huge Stock Market Dataset' (S&P500 historical prices)
▸ Quandl / Tiingo API for professional-grade financial data
▸ Kaggle: 'NIFTY-50 Stock Market Data' (Indian market)

5 Recommended Tech Stack

Python yfinance Pandas NumPy TensorFlow/Keras
(LSTM)
PyTorch pandas-ta Backtrader Plotly Streamlit

6 System Architecture & Pipeline

📡 OHLCV Data (yfinance)
↓
🔧 Technical Indicator Engineering

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 78

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
📐 Sequence Construction (sliding window)
↓
🤖 Model A: LSTM
↓
🤖 Model B: Bidirectional LSTM + Attention
↓
📊 Walk-Forward Validation
↓
🔄 Backtesting (Backtrader)
↓
📈 Trading Signal Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Two formulations: (1) Price regression: predict next day's closing price. (2) Direction classification:
predict Up/Down movement. Classification is more actionable for trading.
Phase 2: Data Collection
[Link]('AAPL', period='10y', interval='1d'). For robustness: use multiple assets. Get OHLCV +
adjust for dividends/splits.
Phase 3: Feature Engineering
Technical indicators via pandas-ta: SMA(20,50,200), EMA(12,26), RSI(14), MACD, Bollinger Bands,
ATR, OBV. Lag features: returns at lag 1,2,3,5,10. Rolling stats.
Phase 4: Sequence Construction
Sliding window: look_back=60 days. X = 60 days of features. y = next day return or direction. Normalise
each window independently (avoid lookahead bias).
Phase 5: LSTM Model
Input: (batch, 60, n_features). LSTM(128, return_sequences=True) → Dropout(0.2) → LSTM(64) →
Dropout(0.2) → Dense(1, sigmoid for direction / linear for price). Adam(1e-3). 100 epochs.
Phase 6: Walk-Forward Validation
Critical: never split data randomly. Use expanding window: train on years 1-7, validate year 8. Then
train 1-8, validate year 9. Retrain monthly. This prevents data leakage.
Phase 7: Attention Mechanism
Add temporal attention layer after LSTM: learns to weight recent vs. older timesteps differently.
Visualise attention weights to see which past days matter most.
Phase 8: Backtesting

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 79

Master Project Roadmap — Batch 2: Machine Learning Projects

Backtrader framework. Strategy: buy when model predicts Up, sell when Down. Metrics: Total Return,
Sharpe Ratio, Max Drawdown, vs. Buy-and-Hold benchmark.
Phase 9: Dashboard
Streamlit: select stock + date range. Plot actual vs. predicted prices. Show trading signals. Display
backtest performance metrics.

8 Algorithms & Models

▸ LSTM (Long Short-Term Memory)
▸ Bidirectional LSTM
▸ Temporal Attention Mechanism
▸ Transformer (optional: time-series Transformer)
▸ Technical indicator engineering
▸ Walk-forward validation

9 Evaluation Metrics
Directional Accuracy (%) RMSE (price prediction) Sharpe Ratio (backtested)

Maximum Drawdown (%) Cumulative Return vs. Benchmark Calmar Ratio

10 Expected Output

A stock price prediction and backtesting system that generates directional signals for any
stock, visualises predicted vs. actual prices, and runs a complete strategy backtest
showing realistic performance — with proper walk-forward validation to ensure results are
not artificially inflated by data leakage.

11 Optional Advanced Enhancements

▸ Add sentiment features from financial news (NLP pipeline integration)
▸ Build a multi-asset portfolio optimiser combining ML signals with Modern Portfolio Theory
▸ Implement Temporal Fusion Transformer (TFT) for state-of-the-art time-series performance
▸ Add risk management: position sizing, stop-loss rules in backtest

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 80

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Implement online learning for continuous model updates on new data

▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 81

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 21 of 30 · Machine Learning

21. Loan Approval Prediction

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Loan approval decisions affect millions of people's financial lives. This project builds an automated loan
approval prediction system that classifies applications as Approved or Rejected based on applicant
profile and financial data — with fairness analysis to ensure the model does not discriminate against
protected groups (gender, race, marital status).

2 Objectives
▸ Build a loan approval classifier from applicant and loan data
▸ Perform fairness analysis: detect and mitigate demographic bias in predictions
▸ Handle missing values common in financial application forms
▸ Build a loan officer decision support dashboard
▸ Comply with ECOA/Fair Lending regulatory requirements (conceptually)

3 Real-World Applications
▸ Retail banks and NBFCs automating loan origination
▸ Fintech lending platforms building credit scoring engines
▸ Microfinance institutions in emerging markets
▸ Peer-to-peer lending platforms for risk-based pricing

4 Suggested Datasets
▸ Kaggle: 'Loan Prediction Problem Dataset' (Dream Housing Finance, 614 records)
▸ Kaggle: 'Loan Status Prediction'
▸ Kaggle: 'Home Credit Default Risk'
▸ LendingClub Loan Dataset (Kaggle, 2.2M loans)

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost SHAP
Fairlearn Streamlit

6 System Architecture & Pipeline

📋 Loan Application Data (income, credit history, loan amount, etc.)
↓
🧹 Handle Missing Values (Multiple Imputation)
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 82

Master Project Roadmap — Batch 2: Machine Learning Projects

🔧 Feature Engineering (EMI, income ratio)

↓
🤖 Train Classifiers
↓
⚖️Fairness Analysis (demographic parity, equalised odds)
↓
🔍 SHAP Explainability
↓
🏦 Loan Officer Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Binary: Loan Approved (1) vs. Rejected (0). Key features: Gender, Married, Dependents, Education,
Self_Employed, ApplicantIncome, CoapplicantIncome, LoanAmount, Loan_Amount_Term,
Credit_History, Property_Area.
Phase 2: Missing Values
Credit_History: 50 missing → impute with mode (most people have credit history). LoanAmount: impute
with median by Education × Self_Employed group. Self_Employed: 32 missing → impute with mode.
Phase 3: Feature Engineering
Total_Income = ApplicantIncome + CoapplicantIncome. EMI = LoanAmount / Loan_Amount_Term.
Balance_Income = Total_Income - (EMI × 1000). Loan_Income_Ratio = LoanAmount / Total_Income.
Log-transform skewed features.
Phase 4: Encoding
Label encode: Gender, Married, Education, Self_Employed, Property_Area. One-hot for Property_Area
(3 categories).
Phase 5: Model Training
LR, RF, XGBoost, SVM. Stratified 5-fold CV. Credit_History alone achieves ~80% — ensure models
learn beyond this heuristic.
Phase 6: Fairness Analysis
Compute: Demographic Parity Difference (approval rate gap by gender). Equalised Odds Difference.
Use Fairlearn library. Identify if the model systematically disadvantages any group.
Phase 7: SHAP Explainability
For each application: SHAP waterfall. 'Application rejected because: Credit_History absent (-0.45),
High Loan_Income_Ratio (-0.23).' Essential for regulatory compliance.
Phase 8: Mitigation
If bias detected: Fairlearn Reductions approach (ExponentiatedGradient) or post-processing
(ThresholdOptimizer) to equalise approval rates across groups.
Phase 9: Dashboard

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 83

Master Project Roadmap — Batch 2: Machine Learning Projects

Loan officer interface: input form → prediction + probability + SHAP. Compliance report: show fairness
metrics across demographic groups monthly.

8 Algorithms & Models

▸ Logistic Regression
▸ Random Forest
▸ XGBoost
▸ SVM
▸ SHAP TreeExplainer
▸ Fairlearn ExponentiatedGradient (bias mitigation)
▸ Multiple Imputation

9 Evaluation Metrics
Accuracy ROC-AUC F1 Score

Demographic Parity Difference Equalised Odds Difference SHAP Fidelity Score

False Rejection Rate (regulatory

focus)

10 Expected Output

A loan approval prediction system that provides instant credit decisions with SHAP-based
explanations for each application — along with a fairness audit dashboard that flags if
approval rates differ significantly by gender or other protected attributes, helping lenders
meet fair lending regulatory requirements.

11 Optional Advanced Enhancements

▸ Add credit score simulation: 'These 3 changes would flip this to approved'
▸ Build a scorecard using Weight of Evidence (WoE) + Information Value (IV)
▸ Integrate with bureau APIs (CIBIL, Experian) for real credit data
▸ Add a continuous monitoring system for model drift and emerging bias

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 84

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Implement online learning for continuous model updates on new data

▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 85

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 22 of 30 · Machine Learning

22. Credit Risk Modelling

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Credit risk modelling is foundational to banking — determining the probability of default (PD), loss given
default (LGD), and exposure at default (EAD) for every borrower. This project builds a Basel-compliant
credit risk scorecard and a machine learning PD model, comparing the classic logistic regression
scorecard approach with gradient boosting.

2 Objectives
▸ Build a PD (Probability of Default) model using logistic regression scorecard
▸ Implement Weight of Evidence (WoE) and Information Value (IV) analysis
▸ Compare scorecard with XGBoost and validate using Gini coefficient and KS statistic
▸ Segment the portfolio into risk grades (AAA through D)
▸ Build a credit risk portfolio dashboard for risk managers

3 Real-World Applications
▸ Bank credit underwriting for retail, SME, and corporate loans
▸ Credit card risk scoring for setting credit limits
▸ Bond rating agencies modelling corporate default probability
▸ Basel III / IRB approach regulatory capital calculation

4 Suggested Datasets
▸ Kaggle: 'Lending Club Loan Data' (2.2M loans, 150 features)
▸ Kaggle: 'Home Credit Default Risk' (307,511 applications)
▸ UCI: 'Default of Credit Card Clients' (30,000 clients)
▸ Kaggle: 'Give Me Some Credit' (150,000 borrowers)

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost optbinning
SHAP Plotly Streamlit

6 System Architecture & Pipeline

📋 Loan Application + Performance Data
↓
🔧 WoE Transformation for all features
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 86

Master Project Roadmap — Batch 2: Machine Learning Projects

📊 Information Value Analysis (feature selection)

↓
🤖 Logistic Regression Scorecard
↓
🤖 XGBoost PD Model
↓
📐 Evaluate: Gini, KS, AUC
↓
Risk Grade Assignment
↓
🏦 Portfolio Risk Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Target: default (1) vs. no default (0). PD = probability of default within 12 months. Scorecard = logistic
regression with WoE-transformed features, output as numeric score.
Phase 2: Data Preparation
UCI Credit dataset: 30,000 records. Target: [Link]. Features: LIMIT_BAL, SEX,
EDUCATION, MARRIAGE, AGE, PAY_0 to PAY_6, BILL_AMT1-6, PAY_AMT1-6.
Phase 3: WoE Binning
optbinning library: OptimalBinning for each feature. Compute WoE = ln(Events_Rate /
Non_Events_Rate) per bin. IV = Σ(Events% - Non_Events%) × WoE. Select features with IV > 0.1.
Phase 4: Logistic Regression Scorecard
Train LR on WoE-transformed features. Convert to points scale: Score = Offset + Factor × log(odds).
Typical: 500-1000 range, higher = better credit.
Phase 5: XGBoost PD Model
XGBClassifier on original features. SHAP analysis. Compare AUC to scorecard.
Phase 6: Evaluation Metrics
Gini = 2 × AUC - 1 (credit standard: aim > 0.45). KS Statistic = max(TPR - FPR) across thresholds.
Population Stability Index (PSI) for drift.
Phase 7: Risk Grading
Map PD scores to risk grades: AAA (PD < 0.5%), AA (0.5-1%), A (1-2%), BBB (2-5%), BB (5-10%), B
(10-20%), CCC+ (> 20%). Compute Expected Loss per grade.
Phase 8: Model Validation
Holdout validation. Vintage analysis: compare default rates across cohorts. Backtesting: does predicted
PD match actual default rate?
Phase 9: Dashboard

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 87

Master Project Roadmap — Batch 2: Machine Learning Projects

Portfolio distribution by risk grade. PD score histogram. Migration matrix (grade changes YoY).
Expected Loss calculation. Concentration risk by grade.

8 Algorithms & Models

▸ Weight of Evidence (WoE) transformation
▸ Information Value (IV) feature selection
▸ Logistic Regression Scorecard
▸ XGBoost Classifier
▸ OptimalBinning (monotone WoE bins)
▸ SHAP
▸ Population Stability Index (PSI)

9 Evaluation Metrics
Gini Coefficient (2×AUC-1) KS Statistic ROC-AUC

Brier Score PSI (model stability) Expected Calibration Error

Scorecard Concordance Rate

10 Expected Output

A credit risk system with a points-based scorecard (like FICO) for interpretable credit
decisions and an XGBoost model for maximum discriminatory power — plus a portfolio risk
dashboard showing risk grade distribution, expected loss, and migration trends — meeting
Basel III internal ratings-based (IRB) modelling standards.

11 Optional Advanced Enhancements

▸ Build LGD (Loss Given Default) model using linear regression on recovery rates
▸ Add EAD modelling for revolving credit products
▸ Implement survival analysis for time-to-default modelling
▸ Build a stress testing module: what happens to portfolio PD if unemployment rises 2%?

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 88

Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Add A/B testing framework to validate model improvements in production

▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 89

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 23 of 30 · Machine Learning

23. Movie Recommendation System

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Netflix, Spotify, and YouTube attribute up to 80% of user engagement to their recommendation
engines. This project builds a comprehensive movie recommendation system covering content-based
filtering (movie metadata similarity), collaborative filtering (user rating patterns), and a deep learning
neural collaborative filtering (NCF) model — with a movie discovery application.

2 Objectives
▸ Build content-based recommender using TF-IDF on movie metadata
▸ Build collaborative filtering using matrix factorisation (SVD, ALS)
▸ Implement Neural Collaborative Filtering (NCF) using PyTorch
▸ Handle cold-start: recommend to new users with no history
▸ Build a movie discovery app with personalised recommendations

3 Real-World Applications
▸ Streaming platforms (Netflix, Hotstar) personalising content catalogues
▸ Movie ticket booking apps (BookMyShow) suggesting relevant films
▸ Editorial curation tools for film critics and reviewers
▸ Cinema chains promoting upcoming releases to relevant audiences

4 Suggested Datasets
▸ MovieLens 1M (1M ratings, 6,040 users, 3,952 movies) — standard benchmark
▸ MovieLens 25M (25M ratings, 162,541 users, 62,423 movies)
▸ Kaggle: 'TMDB 5000 Movie Dataset' (metadata: cast, crew, genres, plot)
▸ Kaggle: 'Netflix Prize Data'

5 Recommended Tech Stack

Python Pandas Surprise library implicit (ALS) PyTorch (NCF)
TF-IDF (Scikit-learn) Streamlit

6 System Architecture & Pipeline

📥 Movie Ratings + Metadata (genres, cast, plot)
↓
🤖 Model A: Content-Based (TF-IDF on metadata)
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 90

Master Project Roadmap — Batch 2: Machine Learning Projects

🤖 Model B: SVD Matrix Factorisation (Surprise)

↓
🤖 Model C: ALS (implicit feedback)
↓
🤖 Model D: Neural CF (Embedding + MLP)
↓
🔀 Hybrid: CF + CB combination
↓
📊 Evaluate: Precision@K, NDCG@K
↓
🎬 Movie Discovery App

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Explicit feedback: star ratings. Implicit feedback: watches, clicks. Content-Based: similar movie
characteristics. CF: users with similar taste liked this.
Phase 2: Data Preparation
MovieLens 1M. Users file: UserID, Gender, Age, Occupation, Zip. Movies: MovieID, Title, Genres.
Ratings: UserID, MovieID, Rating, Timestamp.
Phase 3: Content-Based Filtering
Build movie_soup: combine genres + extracted director + top 3 cast (from TMDB). TF-IDF on
movie_soup. Cosine similarity matrix (n_movies × n_movies). Recommend most similar movies to input
title.
Phase 4: SVD (Collaborative Filtering)
Surprise SVD(n_factors=50). Train/test split by timestamp (last 20% = test). Tune: n_factors, lr_all,
reg_all via RandomizedSearchCV. RMSE on test set.
Phase 5: ALS (Implicit)
implicit library ALS(factors=50, iterations=30, regularization=0.01). Convert ratings to confidence matrix
(Cui = 1 + α × rating). Suited for implicit data.
Phase 6: Neural CF (PyTorch)
Two pathways: GMF (dot product of user/item embeddings) + MLP (concatenation through dense
layers). Final: sigmoid(GMF_output + MLP_output). Binary cross-entropy loss on implicit feedback.
Phase 7: Hybrid
For users with ≥ 20 ratings: CF-dominant (0.7 CF + 0.3 CB). For new users: CB-only. Cold-start: ask for
favourite genres → use CB.
Phase 8: Evaluation
Precision@10, Recall@10, NDCG@10. Compute on held-out last ratings per user. Novelty: avg
popularity rank of recommended items. Coverage: % catalogue recommended.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 91

Master Project Roadmap — Batch 2: Machine Learning Projects

Phase 9: Movie App

Streamlit: enter username or select user → see Top 10 recommendations with movie posters (via
TMDB API). Also: 'Because you watched X...' explanations. Genre filter.

8 Algorithms & Models

▸ TF-IDF Content-Based Filtering
▸ SVD Matrix Factorisation (Surprise)
▸ ALS (Alternating Least Squares, implicit)
▸ Neural Collaborative Filtering (GMF + MLP, PyTorch)
▸ Hybrid weighted ensemble
▸ BPR (Bayesian Personalised Ranking) for ranking optimisation

9 Evaluation Metrics
RMSE (rating prediction) Precision@K Recall@K

NDCG@K Hit Rate@K Coverage

Novelty Diversity (ILD — Intra-List

Diversity)

10 Expected Output

A personalised movie recommendation app where users receive top-10 tailored movie
suggestions based on their viewing history — with explanations, genre filters, and a
'Discover' mode for finding hidden gems outside their usual preferences. New users are
handled via genre-based cold-start.

11 Optional Advanced Enhancements

▸ Add sequence-aware recommendations using SASRec (Self-Attentive Sequential
Recommendation)
▸ Build a social recommendation layer: 'Your friends also liked...'
▸ Integrate real-time TMDB API for movie metadata, trailers, and posters
▸ Add contextual recommendations: suggest horror on Friday night vs. family films on Sunday
morning

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
AI & Data Science Student Projects · Batch 2: Machine Learning · Page 92
Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI

▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 93

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 24 of 30 · Machine Learning

24. Music Recommendation System

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Spotify attributes 30% of streams to its recommendation algorithms. Music recommendations require
understanding audio features (tempo, energy, danceability), listening context (time, activity), and social
signals. This project builds a music recommendation system using audio features from the Spotify API,
user listening history, and collaborative filtering.

2 Objectives
▸ Build a music recommender using Spotify audio features (BPM, energy, danceability)
▸ Implement K-Means clustering for music mood/genre grouping
▸ Build collaborative filtering on user playlist co-occurrence
▸ Create a 'radio station' mode: continuous music stream based on seed track
▸ Build a playlist generation tool for any mood or activity

3 Real-World Applications
▸ Music streaming platforms (Spotify, Apple Music, JioSaavn) for discovery
▸ Workout apps generating energy-matched playlists
▸ Driving or focus apps creating context-appropriate music streams
▸ Music events and festivals for personalised set recommendations

4 Suggested Datasets
▸ Kaggle: 'Spotify 1.2M Songs' (audio features: energy, tempo, danceability, valence, etc.)
▸ Kaggle: 'Spotify Dataset 1921-2020' (600k tracks)
▸ Kaggle: 'Spotify Tracks Dataset' (114k tracks, 20+ features)
▸ Spotify API (via spotipy library) for live data

5 Recommended Tech Stack

Python Pandas Scikit-learn spotipy (Spotify K-Means
API)
Surprise / implicit Plotly Streamlit

6 System Architecture & Pipeline

🎵 Track Audio Features (energy, tempo, danceability, valence, etc.)
↓
📊 EDA + Feature Distributions

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 94

Master Project Roadmap — Batch 2: Machine Learning Projects

↓
🔧 Feature Scaling (StandardScaler)
↓
🔍 K-Means Clustering (Mood Segments)
↓
🤖 Content-Based: cosine similarity on audio features
↓
🤖 CF: Playlist co-occurrence matrix
↓
📻 Radio Mode + Playlist Generator
↓
Music Discovery App

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Audio features: danceability [0,1], energy [0,1], loudness (dB), speechiness, acousticness,
instrumentalness, liveness, valence (positive/negative mood), tempo (BPM), duration.
Phase 2: Dataset Loading
Spotify 1.2M dataset. Key columns: track_name, artists, year, popularity, danceability, energy, key,
loudness, mode, speechiness, acousticness, instrumentalness, liveness, valence, tempo.
Phase 3: EDA
Distribution of each audio feature. Correlation heatmap. Popularity distribution (heavy tail). Decade
trends: how music energy/valence changed over time.
Phase 4: K-Means Clustering
Scale all audio features. Find optimal k using Elbow method + Silhouette score. Typical: k=6-8 clusters.
Label clusters: 'Energetic', 'Calm', 'Happy', 'Melancholic', 'Danceable', 'Focus'.
Phase 5: Content-Based Recommender
For a seed track: get its audio feature vector. Compute cosine similarity to all other tracks. Return top-N
most similar tracks.
Phase 6: Radio Mode
Iterative recommendation: start with seed → recommend similar → use that recommendation as next
seed with 20% random noise → creates a continuous stream. Add diversity constraint.
Phase 7: Playlist Generator
Input: mood (Happy, Workout, Sleep, Focus) + duration → select from matching cluster → rank by
popularity → return playlist. Genre filter option.
Phase 8: CF on Playlists

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 95

Master Project Roadmap — Batch 2: Machine Learning Projects

User-playlist matrix: if multiple users include the same tracks, those tracks are 'collaboratively filtered'.
Use implicit ALS on this co-occurrence data.
Phase 9: Spotify Integration
spotipy library: authenticate → fetch user's saved tracks → get audio features → personalise
recommendations based on their actual music taste.

8 Algorithms & Models

▸ K-Means clustering on audio features
▸ Cosine similarity for content-based recommendation
▸ ALS (implicit feedback) on playlist co-occurrence
▸ StandardScaler for feature normalisation
▸ PCA + t-SNE for music cluster visualisation

9 Evaluation Metrics
Silhouette Score (cluster quality) Intra-cluster similarity (audio Precision@K (user rating on
feature cohesion) recommendations)

Novelty score (vs. user's existing Diversity (audio feature spread in

library) recommendations)

10 Expected Output

A music discovery app where users can input a seed song or select a mood, and receive a
personalised playlist of similar tracks — with an interactive scatter plot showing how all
songs cluster by audio features, and a 'Radio Mode' that streams continuously similar
music like Spotify's artist radio.

11 Optional Advanced Enhancements

▸ Add lyrics analysis using NLP for theme-based recommendations
▸ Integrate with Spotify API for real-time personalisation and playlist export
▸ Build a mood detection model from user facial expression for automatic mood matching
▸ Add a trend-aware component: boost recent songs to surface new discoveries

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
AI & Data Science Student Projects · Batch 2: Machine Learning · Page 96
Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Implement online learning for continuous model updates on new data

▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 97

Master Project Roadmap — Batch 2: Machine Learning Projects

Project 25 of 30 · Machine Learning

25. House Price Prediction

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Real estate is the world's largest asset class. Accurate house price prediction helps buyers make
informed decisions, sellers set optimal prices, banks determine mortgage amounts, and assessors
compute property taxes. This project builds a house price regression model using the Ames Housing
dataset — often called the 'advanced MNIST' of regression problems due to its rich feature set.

2 Objectives
▸ Build a house price regression model with > 90% R² on test set
▸ Apply extensive feature engineering on 79 housing variables
▸ Compare linear models, tree ensembles, and neural networks
▸ Implement SHAP to explain individual property valuations
▸ Build an interactive property valuation tool

3 Real-World Applications
▸ Real estate portals (Zillow, 99acres) providing automated valuations (AVMs)
▸ Mortgage lenders setting loan-to-value ratios
▸ Property tax assessment systems for municipalities
▸ Real estate investors screening acquisition opportunities

4 Suggested Datasets
▸ Kaggle: 'House Prices — Advanced Regression Techniques' (Ames Housing, 79 features)
▸ Kaggle: 'Bengaluru House Price Data'
▸ Kaggle: 'Boston Housing Dataset' (simpler, 13 features)
▸ King County (Seattle) House Sales Dataset (Kaggle)

5 Recommended Tech Stack

Python Pandas NumPy Scikit-learn XGBoost
LightGBM CatBoost SHAP Streamlit

6 System Architecture & Pipeline

📋 79 Housing Features (size, quality, location, condition, etc.)
↓
🧹 Handle 19 Features with Nulls
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 98

Master Project Roadmap — Batch 2: Machine Learning Projects

🔧 Feature Engineering + Log Transform Target

↓
🔢 Encode Ordinal + Nominal Features
↓
🤖 Train: Ridge, Lasso, ElasticNet, RF, XGB, LGB, CB
↓
🔀 Stacking Ensemble
↓
📊 RMSE on Log(SalePrice)
↓
🏠 Property Valuation Tool

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Regression target: SalePrice. 79 explanatory features. Log-transform SalePrice: log(SalePrice) makes
distribution more normal and reduces impact of extreme prices.
Phase 2: Missing Values (Critical)
19 features have nulls. Many are NOT missing at random: LotFrontage (no frontage), PoolQC,
MiscFeature, Alley, Fence are 'NA' meaning 'None' — fill with 'None' string. Numeric NA (GarageYrBlt):
fill with 0 or median.
Phase 3: Feature Engineering
TotalSF = TotalBsmtSF + 1stFlrSF + 2ndFlrSF. TotalBath = FullBath + 0.5×HalfBath + BsmtFullBath +
0.5×BsmtHalfBath. HouseAge = YrSold - YearBuilt. IsRemodelled = (YearRemodAdd != YearBuilt).
HasPool = (PoolArea>0).
Phase 4: Encoding
Ordinal features (ExterQual, KitchenQual: Poor→Excellent): map to 0-4 numeric. Nominal
(Neighborhood): one-hot encode. Label encode high-cardinality categoricals.
Phase 5: Linear Models
Ridge(alpha=10), Lasso(alpha=0.001), ElasticNet. Log transform target. StandardScaler. These provide
regularised baselines.
Phase 6: Ensemble Models
XGBRegressor, LGBMRegressor, CatBoostRegressor. All handle mixed data types well. Use KFold(5)
cross-validation. Target metric: RMSE on log(SalePrice).
Phase 7: Stacking Ensemble
Level 0: XGB, LGB, CatBoost, Ridge. Level 1: Ridge meta-learner on OOF predictions. Stacking
typically improves RMSLE by 3-5%.
Phase 8: SHAP Analysis

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 99

Master Project Roadmap — Batch 2: Machine Learning Projects

For a specific house: waterfall plot showing which features add or subtract from predicted price.
'OverallQual=9 adds +$45k, Neighborhood=NridgHt adds +$32k, No garage removes -$12k.'
Phase 9: Valuation Tool
Streamlit: input form for key features (sqft, bedrooms, location, quality). Output: estimated price +
confidence interval + SHAP breakdown + comparable recent sales.

8 Algorithms & Models

▸ Ridge, Lasso, ElasticNet Regression
▸ Random Forest Regressor
▸ XGBoost Regressor
▸ LightGBM Regressor
▸ CatBoost Regressor
▸ Stacking Ensemble (meta-learning)
▸ SHAP TreeExplainer

9 Evaluation Metrics
RMSLE (Root Mean Squared Log RMSE MAE
Error) — competition standard

R² Score MAPE (%) SHAP Mean Absolute Attribution

10 Expected Output

A property valuation tool that estimates any home's price within 10% accuracy (RMSLE <
0.12), with a SHAP breakdown explaining which features drive the estimate — useful for
buyers comparing properties, sellers setting asking prices, and analysts understanding real
estate value drivers.

11 Optional Advanced Enhancements

▸ Add geographic features using lat/lon (distance to schools, parks, metro)
▸ Integrate with real estate APIs for live comparable sales
▸ Build a 'what would increase my home's value most?' optimization tool
▸ Add temporal features for price trend forecasting by neighbourhood

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 100
Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Scale to production with MLflow model registry and CI/CD pipeline

▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 101
Master Project Roadmap — Batch 2: Machine Learning Projects

Project 26 of 30 · Machine Learning

26. Energy Consumption Forecasting

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Electric grids must match supply to demand in real time — over-supply wastes energy, under-supply
causes blackouts. Accurate energy demand forecasting enables smarter generation planning,
renewable energy integration, and demand response pricing. This project builds a building-level and
grid-level energy consumption forecaster using sensor data and external weather features.

2 Objectives
▸ Forecast hourly energy consumption for buildings or grid zones
▸ Incorporate weather features (temperature, humidity, wind) as external regressors
▸ Compare Prophet, LSTM, and gradient boosting approaches
▸ Build a multi-horizon forecast (1h, 24h, 7-day ahead)
▸ Create an energy operations dashboard with forecast visualisation

3 Real-World Applications
▸ Utility companies planning grid load dispatch
▸ Commercial building energy managers optimising HVAC scheduling
▸ Renewable energy developers planning solar/wind generation
▸ Demand response programme operators

4 Suggested Datasets
▸ Kaggle: 'Hourly Energy Consumption' (AEP, 10 regions, 2004-2018)
▸ Kaggle: 'Smart Meters in London' (5,566 households, 30-minute intervals)
▸ Kaggle: 'ASHRAE — Energy Prediction' (competition dataset)
▸ UCI: 'Individual Household Electric Power Consumption'

5 Recommended Tech Stack

Python Pandas Prophet TensorFlow/Keras LightGBM
(LSTM)
statsmodels Plotly Streamlit

6 System Architecture & Pipeline

📡 Smart Meter / Grid Sensor Data (hourly kWh)
↓
Merge Weather Data (temperature, humidity)

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 102
Master Project Roadmap — Batch 2: Machine Learning Projects

↓
🔧 Feature Engineering (time features, lag, rolling)
↓
📊 Seasonal Decomposition
↓
🤖 Prophet + LSTM + LightGBM
↓
📐 Walk-Forward Validation
↓
📈 Multi-horizon Forecast Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Time series regression. Target: energy consumption (kWh) per hour. Seasonality at three levels: daily
(peak morning/evening), weekly (lower weekend), annual (summer AC peak).
Phase 2: Data Loading
AEP hourly dataset: 121,273 hourly readings from 2004-2018. Columns: Datetime, AEP_MW. Parse
datetime. Set as index.
Phase 3: Feature Engineering
Time: hour, day_of_week, month, is_weekend, is_holiday. Lag features: lag_24 (same hour yesterday),
lag_168 (same hour last week). Rolling: rolling_mean_24, rolling_std_24. Weather (if available): temp,
humidity.
Phase 4: EDA
Plot full time series. Weekly pattern (bar chart by day). Hourly pattern (heatmap: hour × day_of_week).
Seasonal decomposition. Identify anomalies (holidays, outages).
Phase 5: Prophet
Add regressors (temperature if available). Add custom seasonalities. Changepoint detection. Generate
7-day forecast with uncertainty bands.
Phase 6: LSTM
Sliding window: 72 hours input → 24 hours ahead forecast. MultiStep LSTM. Train on 80% data. Walk-
forward validate on 20%.
Phase 7: LightGBM
Tabular format with all engineered features. TimeSeriesSplit(5) CV. Tune: num_leaves, learning_rate,
n_estimators. Fast and accurate for this type of data.
Phase 8: Evaluation
MAE, RMSE, MAPE per model per horizon. Plot actual vs. forecast. Prophet wins on interpretability;
LightGBM typically wins on accuracy.
Phase 9: Dashboard

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 103
Master Project Roadmap — Batch 2: Machine Learning Projects

Real-time energy dashboard: current consumption vs. forecast, daily/weekly trend, anomaly alerts
(consumption > 2σ above forecast).

8 Algorithms & Models

▸ Facebook Prophet (with external regressors)
▸ LSTM (multi-step ahead)
▸ LightGBM with time-series features
▸ SARIMA for statistical baseline
▸ STL decomposition
▸ Walk-forward TimeSeriesSplit validation

9 Evaluation Metrics
MAE (kWh) RMSE (kWh) MAPE (%)

sMAPE (%) Coverage of Prediction Intervals WAPE (Weighted Absolute

Percentage Error)

10 Expected Output

An energy forecasting platform that provides accurate 1-hour, 24-hour, and 7-day
consumption forecasts for any building or grid zone — with uncertainty bands, anomaly
detection alerts, and a clear visualisation of forecast vs. actual — enabling energy
managers to optimise operations, reduce peak demand charges, and plan renewable
energy dispatch.

11 Optional Advanced Enhancements

▸ Add solar generation forecasting to complement demand forecasting
▸ Build a demand response trigger system (alert + action when demand > threshold)
▸ Implement multi-site aggregation: forecast at building → zone → grid level
▸ Add carbon intensity integration: optimise energy use for lowest emissions

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 104
Master Project Roadmap — Batch 2: Machine Learning Projects

▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 105
Master Project Roadmap — Batch 2: Machine Learning Projects

Project 27 of 30 · Machine Learning

27. Dynamic Pricing Prediction

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Dynamic pricing — adjusting prices in real time based on demand, competition, and context — is used
by airlines, hotels, ride-sharing, and e-commerce to maximise revenue. This project builds a dynamic
pricing model that predicts the optimal price for a product or service based on demand elasticity,
competitive landscape, and temporal factors.

2 Objectives
▸ Model price elasticity of demand using regression analysis
▸ Build a demand prediction model as a function of price and context
▸ Implement a revenue-maximising pricing optimisation algorithm
▸ Simulate pricing strategies using historical data backtesting
▸ Build a pricing recommendation dashboard for category managers

3 Real-World Applications
▸ Airline revenue management (fare class optimisation)
▸ Hotel pricing based on occupancy and booking window
▸ Ride-sharing surge pricing (Uber/Ola dynamic pricing engine)
▸ E-commerce repricing against competitor prices

4 Suggested Datasets
▸ Kaggle: 'Airbnb Listings & Reviews' (NYC pricing data)
▸ Kaggle: 'Hotel Booking Demand Dataset'
▸ Kaggle: 'Uber Surge Pricing Dataset'
▸ Kaggle: 'E-Commerce Pricing Dataset'

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost [Link]
Plotly Streamlit

6 System Architecture & Pipeline

📋 Historical Pricing + Demand Data
↓
📊 EDA: Price vs Demand Relationship
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 106
Master Project Roadmap — Batch 2: Machine Learning Projects

📐 Elasticity Estimation (log-log regression)

↓
🔧 Feature Engineering (seasonality, competition, events)
↓
🤖 Demand Prediction Model
↓
🔀 Revenue Optimisation (argmax over price grid)
↓
Pricing Recommendation Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Dynamic pricing goal: maximise revenue = price × demand. Price elasticity = % change in demand / %
change in price. Most products: elasticity < -1 (elastic demand).
Phase 2: Dataset
Airbnb NYC listings: 48,895 listings. Columns: price, neighbourhood, room_type, minimum_nights,
number_of_reviews, reviews_per_month, availability_365, latitude, longitude.
Phase 3: Elasticity Analysis
Log-log regression: log(demand) ~ log(price) + controls. Coefficient of log(price) = elasticity. Compute
per neighbourhood and room type.
Phase 4: Demand Model
Features: price, neighbourhood, room_type, day_of_week, month, is_weekend, days_until_event,
competitor_avg_price (computed). Target: bookings or reviews_per_month. XGBoost Regressor.
Phase 5: Revenue Optimisation
For each listing: sweep price from $50 to $500 in $5 steps. Predict demand at each price using demand
model. Revenue = price × demand. Return price with max revenue. [Link].minimize_scalar as
alternative.
Phase 6: Competitive Pricing
Compute: median price of similar listings (same neighbourhood + room_type) as competitive_price. If
demand is elastic: set below competition. If inelastic: set at premium.
Phase 7: Seasonality
Month × room_type × neighbourhood interaction features. Holiday and event flags. Seasonal price
multipliers from additive decomposition.
Phase 8: Backtesting
Apply pricing model to last 3 months of historical data. Compare model-recommended prices to actual
prices. Estimate revenue uplift.
Phase 9: Dashboard

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 107
Master Project Roadmap — Batch 2: Machine Learning Projects

Pricing recommendation for each listing: current price, recommended price, expected demand change,
expected revenue change. Category-level elasticity charts.

8 Algorithms & Models

▸ Log-log regression for price elasticity estimation
▸ XGBoost demand prediction model
▸ Grid search revenue optimisation
▸ [Link] for continuous price optimisation
▸ Competitive pricing rule engine

9 Evaluation Metrics
Price Elasticity Coefficient Demand Model RMSE Simulated Revenue Uplift %

Pricing Recommendation Revenue per Available Unit Occupancy Rate Change

Acceptance Rate (RevPAU)

10 Expected Output

A dynamic pricing recommendation system that suggests the optimal price for each
product/listing based on demand elasticity, competitive context, and temporal factors —
with a backtest showing estimated revenue uplift compared to a static pricing strategy.

11 Optional Advanced Enhancements

▸ Add a multi-armed bandit for online price experimentation
▸ Build a real-time competitive scraping pipeline for price intelligence
▸ Implement reinforcement learning (RL) for continuous pricing optimisation
▸ Add customer willingness-to-pay segmentation for personalised pricing

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 108
Master Project Roadmap — Batch 2: Machine Learning Projects

Project 28 of 30 · Machine Learning

28. Retail Demand Forecasting

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Getting retail demand forecasting right is the difference between stockouts that lose sales and
overstock that destroys margin. This project builds a production-grade retail demand forecasting
system that generates SKU-level 30-day forecasts incorporating seasonality, promotions, and inter-
product correlation — using the M5 Forecasting competition framework as a guide.

2 Objectives
▸ Build SKU-level daily demand forecasts for a multi-store retail chain
▸ Incorporate promotional events, holidays, and price changes as features
▸ Compare hierarchical forecasting (bottom-up vs. top-down vs. optimal reconciliation)
▸ Generate probabilistic forecasts with uncertainty quantification
▸ Build an inventory planning tool using forecast output

3 Real-World Applications
▸ Retail chain inventory managers setting reorder points
▸ E-commerce fulfilment centres allocating warehouse picking resources
▸ FMCG manufacturers planning production runs based on demand signals
▸ 3PL logistics companies optimising vehicle routing based on demand forecasts

4 Suggested Datasets
▸ Kaggle: 'M5 Forecasting — Accuracy' (Walmart, 30,490 series, 42,840 products, 3,049 days)
▸ Kaggle: 'Store Sales — Time Series Forecasting' (Favorita, Ecuador)
▸ Kaggle: 'Retail Product Demand Dataset'
▸ Kaggle: 'Superstore Sales Dataset'

5 Recommended Tech Stack

Python Pandas LightGBM Prophet statsmodels
scikit-learn Darts Plotly Streamlit

6 System Architecture & Pipeline

📥 Daily Sales by SKU × Store
↓
🔧 Lag Features, Rolling Stats, Promotional Flags
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 109
Master Project Roadmap — Batch 2: Machine Learning Projects

🤖 LightGBM with Multi-Target Training

↓
🤖 Prophet per SKU for Seasonality
↓
🔀 Hierarchical Reconciliation (OptReconcile)
↓
📊 Probabilistic Forecast (Quantile Regression)
↓
🏪 Inventory Planning Output

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

M5: forecast 28 days ahead for 30,490 time series. Grouped hierarchically: Product → Category →
Department → Store → State. Metric: WRMSSE (Weighted Root Mean Squared Scaled Error).
Phase 2: Data Preparation
M5: [Link] (dates, events, SNAP days), sell_prices.csv (weekly prices), sales_train.csv (daily
quantities). Melt wide format to long format. Merge calendar and prices.
Phase 3: Feature Engineering
Lag: d_1, d_2, d_3, d_7, d_14, d_28, d_35. Rolling: mean and std at 7, 14, 28, 56 days. Price:
current_price, price_change, price_discount_pct. Event: is_event, event_name_1. SNAP day flag.
Phase 4: LightGBM Training
Single global model across all series (multi-task learning). Each row: (item_id, store_id, date, features)
→ next-day demand. LGB handles this efficiently. Use custom WRMSSE loss or MSE.
Phase 5: Quantile Regression
Train LGB with objective='quantile', alpha=0.1, 0.5, 0.9. This gives 10th, 50th, 90th percentile forecasts
→ uncertainty bands.
Phase 6: Hierarchical Reconciliation
Bottom-up: sum SKU forecasts to category → department → store → chain. Top-down: disaggregate
chain forecast to SKU level. Optimal: MinT reconciliation using covariance structure.
Phase 7: Validation
Walk-forward: train on days 1-1850, validate on 1851-1912. Compute RMSSE per series. Report mean
across all 30,490 series.
Phase 8: Inventory Planning
For each SKU: safety stock = z × std(demand) × √(lead_time). Reorder point =
forecast_demand_during_leadtime + safety_stock. Reorder quantity = EOQ.
Phase 9: Dashboard
SKU selector → 30-day forecast chart with uncertainty bands. Inventory recommendation (reorder now
if stock < ROP). Promotional impact toggle.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 110
Master Project Roadmap — Batch 2: Machine Learning Projects

8 Algorithms & Models

▸ LightGBM (global multi-task forecasting)
▸ Facebook Prophet (per-series with events)
▸ Quantile Regression for uncertainty
▸ Hierarchical MinT reconciliation
▸ Optimal Economic Order Quantity (EOQ)
▸ Walk-forward TimeSeriesSplit

9 Evaluation Metrics
WRMSSE (M5 competition metric) RMSE per SKU MAPE

Coverage of Quantile Prediction Bias (over vs. under-forecasting) Inventory Cost Saving (simulated)
Intervals

10 Expected Output

A retail demand forecasting platform that generates 30-day probabilistic forecasts for every
SKU in every store — with hierarchically reconciled aggregate forecasts, inventory
replenishment recommendations (reorder quantities and timing), and promotional uplift
modelling — reducing stockouts and overstock simultaneously.

11 Optional Advanced Enhancements

▸ Add N-BEATS or TFT (Temporal Fusion Transformer) for neural forecasting
▸ Build a causal inference model for promotional effect estimation
▸ Add sell-through optimisation: markdown timing recommendation
▸ Implement continuous forecast monitoring with drift detection

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 111
Master Project Roadmap — Batch 2: Machine Learning Projects

Project 29 of 30 · Machine Learning

29. Customer Purchase Prediction

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Predicting which customers will make a purchase in the next 30 days enables hyper-targeted
marketing, optimal campaign timing, and personalised promotions. This project builds a customer
purchase propensity model combining RFM features, behavioural signals, and product affinity patterns.

2 Objectives
▸ Build a binary purchase propensity model for next 30-day purchase prediction
▸ Engineer RFM + behavioural features from transaction history
▸ Segment customers by propensity score for targeted campaigns
▸ Optimise campaign ROI by sending offers only to high-propensity customers
▸ Build an ML-powered marketing campaign prioritisation tool

3 Real-World Applications
▸ E-commerce marketing teams targeting next purchase campaigns
▸ Retail loyalty apps personalising push notification timing
▸ Insurance companies predicting policy renewal intention
▸ Subscription businesses predicting upgrade or cross-sell readiness

4 Suggested Datasets
▸ UCI: 'Online Retail Dataset' (541k transactions, 4,372 customers)
▸ Kaggle: 'E-Commerce Data' (UK online retail)
▸ Kaggle: 'Customer Transaction Dataset'
▸ Kaggle: 'Retail Transaction Data'

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost SHAP
Streamlit FastAPI

6 System Architecture & Pipeline

📥 Historical Transaction Data (CustomerID, Date, Amount, Product)
↓
🔧 Observation Window + Label Construction
↓

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 112
Master Project Roadmap — Batch 2: Machine Learning Projects

📐 RFM + Behavioural Feature Engineering

↓
⚖️Class Imbalance Handling
↓
🤖 Train Binary Classifier
↓
🎯 Propensity Score Ranking
↓
📧 Marketing Campaign Prioritisation

7 Step-by-Step Implementation Roadmap

Phase 1: Label Construction

Observation window: use first 9 months of data as features. Label window: months 10-12. Label = 1 if
customer made any purchase in months 10-12. This is the proper temporal split.
Phase 2: RFM Features
Recency: days since last purchase in observation period. Frequency: number of purchases. Monetary:
total spend. Plus: avg order value, return rate, days between purchases.
Phase 3: Behavioural Features
Product category breadth (how many different categories). Evening/Weekend shopping flag.
Promotional purchases % (orders with discounts). Seasonal purchases. YoY purchase growth.
Phase 4: Feature Importance Pre-screen
Random Forest quick fit → plot feature importances → drop bottom 20% features with importance <
0.01. Reduces noise.
Phase 5: Model Training
XGBoost binary classifier. SMOTE if needed (typically mild imbalance: 60% won't purchase). 5-fold
stratified CV. Tune: max_depth (3-6), learning_rate (0.01-0.1), n_estimators (100-500).
Phase 6: Threshold Optimisation
Set threshold based on campaign budget: if budget allows contacting 1,000 customers, take top 1,000
by propensity score. Compute precision (conversion rate) at that cut-off.
Phase 7: SHAP Explanation
Campaign personalisation: high propensity customer whose top driver is 'high frequency' → don't offer
discount (they'd buy anyway). One whose top driver is 'declining recency' → send win-back offer.
Phase 8: Uplift Modelling (Extension)
Compare: treatment group (received offer) vs. control group (no offer). Compute incremental lift =
purchase rate in treatment - purchase rate in control. This measures true campaign impact.
Phase 9: Dashboard
Ranked customer list by propensity score. Expected conversion rate at each cut-off threshold. SHAP-
based personalisation recommendations per customer segment.

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 113
Master Project Roadmap — Batch 2: Machine Learning Projects

8 Algorithms & Models

▸ XGBoost Binary Classifier
▸ Random Forest
▸ Logistic Regression (baseline)
▸ SHAP TreeExplainer
▸ SMOTE
▸ Uplift Modelling (Two-model approach: treatment vs. control)

9 Evaluation Metrics
ROC-AUC Precision@K (at campaign budget Recall@K
cut-off)

Lift Curve (vs. random targeting) Cumulative Gain Chart Incremental Revenue per
Campaign (business metric)

10 Expected Output

A customer purchase propensity scoring system that ranks all customers by their likelihood
to purchase in the next 30 days, with SHAP-based personalisation signals for each
customer — enabling marketing teams to send the right offer to the right customer at the
right time, dramatically improving campaign ROI.

11 Optional Advanced Enhancements

▸ Add product affinity features: which products is each customer most likely to buy next?
▸ Build a full uplift model (Causal ML) for true incrementality measurement
▸ Integrate with email marketing platforms via API for automated campaign dispatch
▸ Add a multi-touch attribution model to understand which touchpoints drive conversions

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 114
Master Project Roadmap — Batch 2: Machine Learning Projects

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 115
Master Project Roadmap — Batch 2: Machine Learning Projects

Project 30 of 30 · Machine Learning

30. Employee Attrition Prediction

🤖 MACHINE LEARNING · Master Roadmap

1 Project Overview
Replacing an employee costs 50-200% of their annual salary in recruitment, training, and productivity
loss. Predicting which employees are likely to leave enables HR teams to intervene with targeted
retention programmes before attrition occurs. This project builds an employee attrition predictor using
IBM HR Analytics data, with workforce analytics insights for HR leadership.

2 Objectives
▸ Build a binary attrition predictor from employee profile and engagement data
▸ Identify the top drivers of attrition using SHAP analysis
▸ Segment employees into retention risk tiers
▸ Build a people analytics dashboard for HR leadership
▸ Perform fairness analysis: ensure model does not discriminate by age, gender

3 Real-World Applications
▸ HR analytics teams in large enterprises reducing voluntary turnover
▸ HRIS platforms (SAP SuccessFactors, Workday) providing attrition risk scores
▸ Consulting firms building people analytics practices
▸ Startups monitoring early-warning signals in small teams

4 Suggested Datasets
▸ Kaggle: 'IBM HR Analytics Employee Attrition & Performance' (1,470 employees, 35 features)
— canonical
▸ Kaggle: 'Employee Attrition Dataset'
▸ Kaggle: 'HR Analytics: Job Change of Data Scientists'
▸ Kaggle: 'HR Employee Attrition and Performance'

5 Recommended Tech Stack

Python Pandas Scikit-learn XGBoost SHAP
Fairlearn Plotly Streamlit

6 System Architecture & Pipeline

📋 Employee Data (demographics, job, satisfaction, performance)
↓
🧹 Encode categoricals, handle constants

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 116
Master Project Roadmap — Batch 2: Machine Learning Projects

↓
📊 EDA: Attrition rate by segment
↓
⚖️Handle Class Imbalance (16% attrition)
↓
🤖 Train Classifiers
↓
🔍 SHAP Feature Importance
↓
⚖️Fairness Check (age, gender)
↓
👥 People Analytics Dashboard

7 Step-by-Step Implementation Roadmap

Phase 1: Problem Understanding

Binary: Attrition=Yes (1) vs. No (0). 16.1% attrition rate. 35 features: Age, BusinessTravel, Department,
DistanceFromHome, Education, EnvironmentSatisfaction, Gender, JobInvolvement, JobLevel, JobRole,
JobSatisfaction, MaritalStatus, MonthlyIncome, NumCompaniesWorked, OverTime, PercentSalaryHike,
PerformanceRating, RelationshipSatisfaction, StockOptionLevel, TotalWorkingYears,
TrainingTimesLastYear, WorkLifeBalance, YearsAtCompany, YearsInCurrentRole,
YearsSinceLastPromotion, YearsWithCurrManager.
Phase 2: Data Cleaning
Remove constants: EmployeeCount (all 1), Over18 (all Y), StandardHours (all 80). Convert Attrition and
OverTime to binary. One-hot encode nominal categoricals.
Phase 3: EDA
Attrition rate by: Department, JobRole, OverTime, MaritalStatus, JobSatisfaction. Box plots:
MonthlyIncome and Age by Attrition. Distance distribution.
Phase 4: Feature Engineering
SatisfactionAvg = mean(JobSatisfaction, EnvironmentSatisfaction, RelationshipSatisfaction,
WorkLifeBalance). IncomePerYear = MonthlyIncome / TotalWorkingYears. PromotionLag =
YearsSinceLastPromotion. Stagnation = YearsInCurrentRole / TotalWorkingYears.
Phase 5: Model Training
LogisticRegression, RF, XGBoost. class_weight='balanced' or SMOTE. 5-fold CV. Focus on recall
(don't miss employees about to leave).
Phase 6: SHAP Analysis
Global: top 10 attrition drivers (typically: OverTime, MonthlyIncome, JobSatisfaction, Age,
WorkLifeBalance). Segment analysis: SHAP by Department.
Phase 7: Fairness

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 117
Master Project Roadmap — Batch 2: Machine Learning Projects

Demographic parity by Gender and AgeGroup. If model predicts higher attrition for a demographic
group due to proxy features, flag and mitigate.
Phase 8: Risk Tiers
Tier 1 (Critical Risk: P > 70%): immediate manager conversation. Tier 2 (High Risk: 50-70%):
engagement programme. Tier 3 (Medium: 30-50%): monitor. Tier 4 (Low: < 30%): no action.
Phase 9: Dashboard
HR leader view: attrition risk heatmap by Department × JobRole, Top 50 at-risk employees table, Driver
analysis, Fairness summary, Retention programme ROI calculator.

8 Algorithms & Models

▸ Logistic Regression (baseline + interpretability)
▸ Random Forest Classifier
▸ XGBoost Classifier
▸ SHAP TreeExplainer
▸ SMOTE for class imbalance
▸ Fairlearn for bias detection

9 Evaluation Metrics
ROC-AUC Recall (sensitivity for at-risk F1 Score
employees)

Precision@Risk Tier Demographic Parity Difference SHAP Attribution Stability

10 Expected Output

A people analytics platform where HR leaders can see a risk heatmap of attrition across
departments and roles, identify the top 50 employees most likely to leave within 3 months,
understand the primary reasons (overwork, low pay, lack of promotion), and measure the
financial impact of retention interventions — turning reactive HR into proactive workforce
management.

11 Optional Advanced Enhancements

▸ Add a flight risk score integrated with real HRIS data (Workday API)
▸ Build a retention intervention simulator: ROI of salary increase vs. promotion vs. training
▸ Add a sentiment analysis layer on employee survey data
▸ Build a succession risk layer: flag critical roles where high performers are at risk

12 Deliverables
▸ GitHub repository with full source code and README
▸ Jupyter Notebook with complete EDA and model training
▸ Project report (PDF, 10-15 pages)
▸ Presentation slides (10-12 slides)
▸ Working demo application (Streamlit / FastAPI)
▸ Model card documenting performance, limitations, and fairness analysis

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 118
Master Project Roadmap — Batch 2: Machine Learning Projects

13 Future Improvements
▸ Scale to production with MLflow model registry and CI/CD pipeline
▸ Deploy as containerised microservice on AWS SageMaker / GCP Vertex AI
▸ Implement online learning for continuous model updates on new data
▸ Add A/B testing framework to validate model improvements in production
▸ Publish results as an academic paper or technical blog post

AI & Data Science Student Projects · Batch 2: Machine Learning · Page 119