3.
6 Model Justification and Tuning Rationale
XGBoost Classifier
XGBoost (Extreme Gradient Boosting) was selected due to its widespread success in
handling structured, tabular datasets, which aligns well with our manufacturing defect
dataset. One of its primary advantages is its robustness to skewed class distributions—a
frequent issue in industrial defect detection where defective samples are rare. Its
ensemble nature, built on gradient boosting trees, allows it to capture complex feature
interactions while remaining computationally efficient. Moreover, it offers built-in
regularization which helps reduce overfitting and provides interpretable feature
importance rankings based on split gains.
In our implementation, we configured the model with `eval_metric='logloss'` to optimize
probabilistic accuracy, which is important when assessing confidence in predictions. We
also set `random_state=42` to ensure reproducibility. Although we did not tune
hyperparameters such as learning rate or maximum tree depth, the model still delivered
strong baseline performance. This tuning strategy was driven by practicality and evidence
from prior literature suggesting that even default configurations of XGBoost yield
competitive results.
Random Forest Classifier
The Random Forest algorithm was chosen for its balance of simplicity, performance, and
interpretability. As an ensemble of decision trees, it offers a stable and resilient learning
method that reduces variance through bagging and bootstrap aggregation. This makes it
especially resistant to overfitting, which is critical when dealing with a moderate-sized
dataset with potential noise. In the context of manufacturing, the ability to extract Gini-
based feature importance is highly beneficial, as it provides transparency into which
operational variables most affect defect predictions.
Our tuning involved increasing the number of trees (`n_estimators=100`) to ensure robust
averaging across different subsets of the data. The `random_state` was again fixed at 42
to allow reproducibility across model runs. Other parameters such as maximum depth and
splitting criteria were left at their defaults due to already satisfactory performance and
low variance across folds. The goal was to maintain interpretability and training
efficiency while still achieving competitive results.
Logistic Regression
Logistic Regression served as our baseline linear classifier, providing a benchmark for
evaluating the improvements introduced by more complex models. Its greatest strength
lies in its simplicity and interpretability, making it particularly useful in settings where
understanding the influence of each input feature is as important as the prediction itself.
The model estimates the linear relationship between features and the log-odds of the
target class, allowing stakeholders in manufacturing to trace how variables like
production volume or material cost affect defect likelihood.
We increased the `max_iter` parameter to 1000 to avoid convergence issues, particularly
due to the presence of multiple engineered features and potentially correlated variables.
The `random_state` parameter was also fixed. While more extensive tuning—such as
regularization strength (`C`) or solver selection—could have marginally improved
performance, we prioritized keeping the model interpretable and easy to audit.
Multi-Layer Perceptron (MLP) Neural Network
The Multi-Layer Perceptron (MLP) was selected to explore the power of deep learning in
learning non-linear relationships in our data. As a feedforward artificial neural network,
the MLP can model complex feature interactions that may not be captured by linear
models or decision trees. This is particularly relevant in manufacturing processes where
defect causes may arise from subtle combinations of multiple operational metrics. The
inclusion of MLP helped broaden the modeling spectrum and ensured that our evaluation
covered both traditional and neural methods.
For tuning, we used one hidden layer with 100 neurons (`hidden_layer_sizes=(100,)`) to
provide sufficient capacity for non-linear representation without incurring overfitting or
excessive training time. We set `max_iter=1000` to allow the model ample time to
converge, especially given the SMOTE-balanced data. The choice of a simple
architecture reflects a strategic balance between complexity and interpretability, aiming
to maintain performance while avoiding the opacity of deeper or more complex networks.
TabPFN Classifier
TabPFN (Tabular Probabilistic Forward Network) was incorporated into our model suite
as an innovative, transformer-based approach capable of performing probabilistic
inference on tabular data in a single forward pass. Unlike traditional ML models,
TabPFN leverages a pre-trained transformer model to produce posterior class
probabilities, eliminating the need for manual hyperparameter tuning. This is particularly
advantageous in operational contexts like manufacturing, where rapid deployment and
uncertainty quantification are both valuable.
Due to the high computational demand, training was limited to 1000 samples, and
execution was conducted on CPU to match system constraints. Despite this, the model
yielded strong probabilistic predictions with minimal tuning effort. TabPFN represents a
promising direction for future work, especially in domains requiring interpretable
probabilistic reasoning and uncertainty estimation.