Machine Learning Algorithms
A Complete Reference Guide for Data Scientists
1. Supervised Learning
Supervised learning uses labeled training data to build predictive models. The algorithm learns
a mapping from inputs to outputs based on example input-output pairs.
1.1 Regression Algorithms
• Linear Regression — predicts continuous values using a linear relationship between
features and target variable
• Ridge & Lasso Regression — regularized variants that reduce overfitting by penalizing
large coefficients
• Decision Tree Regressor — splits data into regions based on feature thresholds;
interpretable but prone to overfitting
• Random Forest Regressor — ensemble of decision trees; reduces variance through
bagging
• Gradient Boosting (XGBoost, LightGBM) — sequential boosting; state-of-the-art for
tabular data
1.2 Classification Algorithms
• Logistic Regression — binary/multiclass; outputs probability using sigmoid or softmax
• Support Vector Machine (SVM) — finds optimal hyperplane maximizing margin between
classes
• k-Nearest Neighbors (kNN) — classifies based on majority vote among k closest training
samples
• Naive Bayes — probabilistic classifier based on Bayes theorem; fast and works well with
text
• Neural Networks — multi-layer perceptrons for complex non-linear classification tasks
2. Unsupervised Learning
Unsupervised learning finds hidden patterns in data without labeled responses. Useful for
exploratory analysis and feature learning.
2.1 Clustering
Algorithm Type Key Parameter Best Use Case
K-Means Centroid-based K (clusters) Spherical, balanced
clusters
DBSCAN Density-based eps, min_samples Arbitrary shapes, noise
detection
Hierarchical Tree-based linkage method When num clusters
unknown
Gaussian Mixture Probabilistic n_components Soft cluster
assignments
3. Model Evaluation Metrics
3.1 Regression Metrics
Metric Formula Interpretation
MAE mean(|y - y_hat|) Average absolute error; robust
to outliers
MSE mean((y - y_hat)^2) Penalizes large errors more
heavily
RMSE sqrt(MSE) Same unit as target; most
commonly reported
R-Squared 1 - SS_res/SS_tot Proportion of variance explained
(0 to 1)
3.2 Classification Metrics
• Accuracy — correct predictions / total predictions; misleading for imbalanced datasets
• Precision — TP / (TP + FP); how many predicted positives are truly positive
• Recall (Sensitivity) — TP / (TP + FN); how many actual positives are captured
• F1 Score — harmonic mean of precision and recall; balanced metric
• ROC-AUC — area under ROC curve; measures discriminative power across thresholds
4. Feature Engineering Best Practices
• Normalize/standardize numerical features before training distance-based models
• Encode categorical variables using one-hot encoding or target encoding
• Handle missing values with median/mode imputation or model-based imputation
• Create interaction features for domain-specific relationships
• Use PCA or t-SNE for dimensionality reduction in high-dimensional datasets
• Remove highly correlated features to reduce multicollinearity
5. Hyperparameter Tuning
Systematic hyperparameter search is critical for model performance. Three common strategies:
Method Speed Coverage Recommended For
Grid Search Slow Exhaustive Small parameter spaces
Random Search Fast Stochastic Large parameter spaces
Bayesian Optimization Medium Intelligent Expensive models (neural networks)
© 2024 | Machine Learning Reference Guide | For Educational Use