Data Science Process
TS. Lương Văn Thiện
Business AI Lab, Khoa KHDL&TTNT, Trường CN, ĐH Kinh tế Quốc dân
thienlv@[Link]
1
Content
1. Data collection
2. Understand problem and data
3. Data pre-processing
4. Predictive modelling and Evaluation
5. Deployment and Monitoring
2
Step 1. Data collection
• Web scrapping
• Kaggle
• Github
• Alibaba
• Vietnam
• API calling
• Financial data
• Data for data science and machine learning
3
Popular data sources
• Key word searching:
• Kaggle: [Link]
• Alibaba cloud: [Link]
• Github: [Link]
• [Link]
4
Step 2. Understand problem and dataset
• Data point/sample (stand for?)
• Input/output – Features/Labels
• Problem: classification vs regression
• Data size/ data type
• Range, distribution, outliers
• Missing values
• Correlation (features, labels)
• Visualization
5
House price prediction
• Scenario: You are a Data Scientist at a real estate
company in Melbourne, Australia. The company needs
an automated house valuation system to help customers
accurately estimate the selling price of their property.
• Input [X]: Number of rooms, area, distance to the city
center, property type, number of floors, ...
• Output [y]: Price (AUD)
𝑦 = 𝑤0 + 𝑤1 ∗ 𝑥1 + ⋯ + 𝑤𝑛 ∗ 𝑥𝑛
Melbourne, Australia
[Link]
6
Visualize label - Price
7
Step 3. Data Preprocessing
• Data imputation (missing values, null/nan)
• Outlier Detection & Treatment: Statistical methods
• Categorical Encoding: Label encoding, One-hot encoding, Dropping columns
• Feature Scaling: Standardization, Normalization
• Feature Selection: Eliminate unnecessary features
• Train/Test Split: Be careful with Data Leak
8
Handle missing values – Data Imputation
• Next or Previous Value
• K Nearest Neighbors
• Maximum or Minimum Value
• Missing Value Prediction
• Most Frequent Value
• Average or Linear Interpolation
• (Rounded) Mean or Moving Average or
Median Value
• Fixed Value
[Link]
9
Data Normalization
10
Step 4. Predictive Modelling and Evaluation
• Regression models
• Linear regression, Lasso, Ridge, ElasticNet
• Classification models
• KNN, Decision Tree, Random Forest, SVM,
Boosting, Logistic Regression
• Performance metrics
• MSE, MAE, R2-squared, MAPE
• Acc, F1-score, AUC, precision, recall
• NLP metrics: BLEU, BERT score
• Optimization (avoid underfitting &
overfitting)
• Hyperparameter tuning
• Regularization, Ensembling
• Data augmentation
• Cross-validation
• Feature selection/extraction
11
Pros and cons of K-Nearest Neighbors (KNN)
Pros
• Simple, intuitive, no training phase
• Non-parametric, works for classification & regression
• Flexible with irregular decision boundaries
Cons
• Slow at prediction, memory-heavy
• Sensitive to noise, irrelevant features, and feature scaling
• Struggles in high dimensions and with imbalanced data
• Requires careful choice of K and distance metric
12
Performance metrics for Regression
13
Advanced predictive models
Machine learning:
• Support Vector Regression
• Random Forest Regressor
• Gradient Boosting (XGBoost, LightGBM, CatBoost)
• k-Nearest Neighbors (kNN) Regression
Deep Learning:
• DNN: MLP
• RNN-based: LSTM, GRU, Seq2Seq models
• CNN-based: Temporal Convolutional Networks (TCN), 1D CNN for time-series
• Attention/Transformer-based: Transformer, Informer, Autoformer, TimesNet
LLM-based (Large Language Model):
• Chronos (Amazon, 2023)
• TimeGPT (Nixtla, 2023)
• UniTime / UniTS
• Time-LLM
14
Time-LLM
15
Step 5. Deployment and Monitoring
• Deployment
• Batch/real-time/hybrid
• UI/UX dashboard: web, mobile
• Monitoring
• Error tracking: Evaluation metrics (MSE, Acc, F1-score, …)
• Anomaly detection: alert when the difference between
“forecast” and “actual” is too high
• Data drift, model drift
• Retrain if needed
16
Web app UI
• Tools: Flask API, Streamlit
17
Sum up
• 5 steps
• Be careful with data leak
• Preprocessing is time-consuming
18