DATA ENGINEERING
┌────────────────────────────┐
│ DATA SOURCES │
│ (Yahoo Finance / NSE / │
│ Kaggle Stock Data) │
└──────────────┬─────────────┘
┌────────────────────────────────────────────────┐
│ 1. DATA INGESTION / COLLECTION │
│ - Download stock prices (Open, High, Low, │
│ Close, Volume) │
│ - API Calls / CSV Import │
└───────────────────────┬────────────────────────┘
┌────────────────────────────────────────────────┐
│ 2. DATA CLEANING │
│ - Remove missing values (NaN) │
│ - Fix incorrect data types │
│ - Sort by date │
│ - Remove duplicates │
└───────────────────────┬────────────────────────┘
┌────────────────────────────────────────────────┐
│ 3. DATA TRANSFORMATION │
│ - Convert to MultiIndex (Price, Ticker) │
│ - Add serial numbers / indexing │
│ - Feature scaling (if needed) │
└───────────────────────┬────────────────────────┘
┌────────────────────────────────────────────────┐
│ 4. FEATURE ENGINEERING (MOST │
│ IMPORTANT FOR OUR PROJECT) │
│ Add all technical indicators: │
│ - SMA │
│ - EMA │
│ - MACD (MACD, Signal, Histogram) │
│ - RSI │
│ - Bollinger Bands │
│ - ATR │
│ - Stochastic %D │
│ - Volume Indicators (OBV, MFI, VWAP) │
│ These features help the ML model decide │
│ BUY / SELL / HOLD. │
└───────────────────────┬────────────────────────┘
┌────────────────────────────────────────────────┐
│ 5. DATA STORAGE (Optional) │
│ - Save cleaned dataset to CSV/Database │
│ - Use it for ML model training │
└───────────────────────┬────────────────────────┘
┌────────────────────────────────────────────────┐
│ 6. ML MODEL / SIGNAL ENGINE │
│ - Train Machine Learning Model with │
│ indicators + price data │
│ - Predict BUY / SELL signals │
7. VISUALIZATION LAYER
- Graphs
- Indicator charts
- Buy/Sell signals visualization
RAW DATA
(Yahoo/Kaggle)
DATA INGESTION
(Download & Load)
DATA CLEANING
(Missing values, date format)
FEATURE ENGINEERING
(SMA, EMA, MACD, RSI etc.)
DATA TRANSFORMATION
(Normalization, scaling, indexing)
FINAL DATASET
(Ready for ML Model)
ML TRAINING & SIGNALS
(Predict Buy/Sell/Neutral)
**“Data Engineering is everything we do before Machine Learning.
It includes collecting data, cleaning it, organizing it, and creating new features (indicators) so
the ML model can understand the market better.”**
📌 Key Points to Explain
1️⃣ Data Collection
Download stock data (OHLCV).
Use Yahoo finance or Kaggle.
This is the starting point.
2️⃣ Data Cleaning
Remove missing values.
Fix date format.
Remove duplicates.
Without cleaning, ML model gives wrong results.
3️⃣ Data Transformation
Convert into structured format (MultiIndex).
Make data ready for indicators.
Ensure every column is correct.
4️⃣ Feature Engineering (Most important part in your project)
This is where indicators like SMA, EMA, MACD, RSI are added.
These indicators help the model understand trend, momentum, volatility.
5️⃣ Feature Storage
Save the final dataset.
6️⃣ Pass the engineered dataset to the ML Model
Model uses indicators to learn BUY / SELL decision.
7️⃣ Final Output
Signals like BUY, SELL, HOLD
Graphs and charts.