Model comparison in predictive analytics involves evaluating different models using consistent
metrics, data splits, and statistical significance to select the best one for a given task. This
process includes comparing performance based on error rates (like RMSE), classification
accuracy, and prediction intervals, and requires using the same datasets and training data to
ensure a fair and unbiased comparison.
Key principles for model comparison
Use the same data: Train and evaluate all models on the exact same datasets and data splits to
ensure a fair comparison.
Use appropriate metrics: Select performance metrics that are relevant to the model's purpose.
o For regression (predicting a number), use metrics like Root Mean Square Error (RMSE).
o For classification (predicting a category), use metrics like classification rate.
Account for statistical significance: A model may appear better simply due to random chance
in the data. Use statistical methods to determine if the observed difference in performance is
statistically significant.
Validate with external data: For a robust comparison, test models on external datasets that
were not used during training to assess their real-world performance.
Consider model interpretability: The "best" model isn't always the most accurate one.
Consider the model's complexity and interpretability. For example, a decision tree is more
interpretable than a neural network and may be preferred for this reason, even if its accuracy is
slightly lower, says this SAS article.
Example: Comparing two models
Imagine you are building two models to predict customer churn:
1. A Logistic Regression model that is simple and easy to understand.
2. A Random Forest model that is more complex but potentially more accurate.
You would compare them by:
Splitting your customer data into training and testing sets.
Training both models on the training set.
Using the test set to evaluate how well each model predicts churn.
Calculating the classification rate for both models.
Using a statistical test to see if the difference in classification rates is significant.
Considering if the extra accuracy of the Random Forest is worth the added complexity for your
specific business needs.
Feature Supervised Learning Unsupervised Learning
Data Labeled data (input-output pairs) Unlabeled data
Goal Predict a specific output based on input Discover hidden patterns and
structures
Guidance Explicit, with correct answers provided Implicit, with the model finding
during training patterns on its own
Examples Classification (e.g., spam detection), Clustering (e.g., customer
Regression (e.g., house price prediction) segmentation), Anomaly detection
Complexit Simpler, often more accurate for specific Computationally more complex
y tasks
Evaluation Can be evaluated against known correct More challenging to evaluate as
answers there is no "correct" answer
A data mining methodology involves a series of steps to extract valuable information from large
datasets, including data preprocessing (cleaning, integration, reduction), data mining (applying
techniques like classification, clustering, or regression), and pattern evaluation and knowledge
representation (interpreting and presenting results). Common techniques include classification
for categorization, clustering for grouping, regression for prediction, and association rule mining
for finding relationships
Data preprocessing
Data Cleaning: Removing errors, inconsistencies, and missing values from the data.
Data Integration: Combining data from multiple sources into a single dataset.
Data Reduction: Reducing the volume of data to be analyzed while retaining important
information, such as through dimensionality reduction.
Data Transformation: Converting data into a format suitable for mining algorithms, often by
normalizing or aggregating it.
Data mining
Classification:
Assigning data points to predefined categories based on labeled examples to build a predictive
model.
Clustering:
Grouping similar data points together without prior knowledge of the categories.
Regression:
Predicting a continuous numerical value based on a relationship with other variables.
Association Rule Mining:
Discovering relationships or co-occurrences between items in a dataset (e.g., "customers who
buy X also tend to buy Y").
Anomaly Detection (Outlier Detection):
Identifying data points that significantly deviate from the expected or normal behavior.
Sequential Pattern Mining:
Discovering frequent subsequences in a dataset, such as user clickstream data.
Pattern evaluation and knowledge representation
Pattern Evaluation:
Assessing the identified patterns to determine their usefulness and significance.
Knowledge Representation:
Presenting the discovered patterns in an understandable format, often using visualization tools, to
support decision-making.
Knowledge Refinement:
Refining the discovered knowledge based on feedback to improve accuracy.
Deployment:
Using the insights gained to make informed decisions or build applications.