1

Selecting Machine Learning Algorithms using

Regression Models
Tri Doan∗ , Jugal Kalita†
Department of Computer Science, University of Colorado, Colorado Springs
Email: ∗ tdoan@[Link], † jkalita@[Link],

1 I NTRODUCTION teristics.
Learning from data is of interest to many dis- Example approaches include STATLOG [?],
parate fields such as banking, bioinformatics, and METAL [?] that use machine learning
business, computer vision and education. The on acquired knowledge of how various ma-
field of data mining uses a large collection chine learning algorithms perform on various
of machine learning algorithms whose goal is datasets. To construct meta-knowledge, statis-
to extract useful information from collected tical summaries and non-statistical information
data. For any given dataset, a common ques- (such as number of instances and number of
tion is which learning algorithm is best suited classes) are used. Metrics from information the-
for it. Performing experiments with several ory (such as entropy and mutual information)
algorithms using the data, or getting advice are used for nominal data. An approach called
from machine learning experts, can help assess landmarking [?] captures a set of informative
which algorithms might be the best candidates, features computed from the selected learners
but this is not always practical. (e.g., C5.0tree) on selected datasets. Going fur-
Using the idea of meta-learning, we solve ther, [?] introduce the use of model-based prop-
the problem of selecting a machine learning al- erties in meta-learning. These features include
gorithm for a particular dataset by supervised characteristics of induced decision trees (e.g.,
learning. In this work, we represent the effi- nodes per attribute, nodes per instance, or
cient way to deal with a non-standard format in average gain-ratio difference) using the same
real-world dataset to obtain training data. Our datasets [?].
work is further address the problem of well- We have identified several issues with past
known algorithm selection recently. approaches to select machine learning algo-
The remainder of the paper is organized as rithms. For example, statistical features that
follows. Section 2 presents related work. Sec- have been used in past studies [?], [?], [?]
tion 3 motivates the solution, followed by our include the mean of summary statistics (e.g.,
approach in Section 4. Section 5 describes our means of standard deviations, skewness, and
experiments. Discussion is in Section 6. Finally, kurtosis across all features) of a dataset. We be-
Section 7 summarizes the paper and provides lieve that averaging values of summary statis-
directions for future study. tics dilutes statistical meaning. For instance,
a left-skew on one attribute might cancel out
the right-skew on another attribute in the final
2 R ELATED W ORK mean of skewnesses. The mean of skewness
In the past, researchers have attempted to pre- or kurtosis values across all features loses its
dict algorithm behavior based on data charac- discriminating character when the distribution
 2

is non-uniform [?].
Using many or all features may also be a
source of problem for both real datasets and
meta-knowledge derived from them. A smaller
optimized number of meta-features may im-
prove the quality of training data to produce a
better predictive meta-model. Researchers deal
with a high number of dimensions in two
ways: feature selection and feature extraction. Fig. 1. Proposed model for algorithm selection
Feature selection retains the most discriminat-
ing features [?]. Feature extraction, on the other
hand, generates a new set of features as predic- fixed number of features before performing
tors in the form of composite attributes [?]. training. Our approach uses dimension reduc-
As a breakthrough of among studies of al- tion to generate a fixed number of features.
gorithm selection, Auto-Weka [?] allows one to Transformed datasets can be generated with
find the best algorithm with corresponding op- a feature generation approach which takes all
timal parameters among algorithm of interest. features from original dataset into account.
While its result provides optimal parameters In addition, the runtime for the transformed
for each algorithm, it suffers a computational dataset is likely to be lower than for the orig-
problem that our work desire to address. Two inal datasets because computational time for
main reasons include non heuristic to select a large number of features often results in
algorithms to perform the hyper-parameter higher run time of an algorithm. As each fea-
search for a given dataset and each execution ture has its own predictive power, a feature
starting from scratch. First reason results in selection approach may not have provided an
wasting time for searching hyper-parameter of adequate solution to generate the same fixed
algorithms that are actual low performance in a number of features. The reason is that if we
specific data domain. Second reason lies on the want the features to guarantee high predictive
approach such that Auto-Weka is built ignores power, feature selection is likely to select dif-
knowledge gained from past experience. ferent numbers of features for different datasets
In addition, there is very little work on al- whereas our meta-learning training set requires
gorithm selection using regression, especially a a fixed number of features.
regression tree model. Our work study the use With transformed datasets obtained, we use
of the family of tree model and demonstrate supervised learning to solve the problem of
the advantage of tree model. predicting performance of specific algorithms
on specific datasets. We apply both linear re-
gression and non-linear regression. Finally, the
3 M ETA - LEARNING IN STEPS outcomes of the regression methods are used to
•Create a training set from original datasets generate a ranked list of algorithms by perfor-
and evaluate performance of algorithm on mance for a specific dataset. Intuitively, using
these transformed datasets. regression is suitable as data mining practition-
• Select a best regression model and pre- ers may be interested in comparing how var-
dict algorithm performance on unknown ious algorithms perform on a dataset first by
dataset. performing regression to predict performance
• Generate a ranked list of machine learning
of individual algorithms on the dataset.
algorithms.
To create meta-data, each original dataset is
used to generate one meta-instance in the train- 4 A PPROACH
ing set. Since real-world datasets come with Figure ?? illustrates our proposed approach. At
different numbers of features, we transform first, the original datasets are preprocessed to
each original dataset into a dataset with a generate training examples. To do this, each
 3

TABLE 1 process is described in following sub-sections.

Meta-features used
Feature Description 4.1 Dimensionality Reduction with variable
Algorithm Data mining algorithm’s name feature issue
ClassInt Ratio of no of classes to instances
AttrClass Ratio of no of features to no of classes Dimensionality reduction uses different tech-
Median* Middle value in a dataset niques to generate a new small set of features
Mean* Average value in a dataset
Std* Standard deviation
from an original dataset. A commonly well-
Skewness* Measure of asymmetry of the prob. dis. known technique for dimension reduction is
Entro-Class Class entropy for target attribute Principal Components Analysis (PCA). We use
TotalCorr Amount of information shared among vari- PCA due to its computational efficiency, say
ables
Performance Measure of performance of each algorithm compared to the Single Value Decomposition
* 4 statistics computed for each of the four meta-features method [?]. To work with PCA, a nominal
feature needs to be converted into numeric.
TABLE 2 We take a straightforward approach: A nom-
Data mining algorithms in this study inal attribute that takes m distinct values, is
converted to m new binary features. As non-
Learner Description linear correlation cannot be detected by PCA,
J48 Non-commercial decision tree C4.5 we adapt the approach by [?] to include mutual
Decision Stump A decision stump
PART A version of C4.5 using decision list
information by discretizing all numeric values.
Decision Table Simple decision table
JRip Propositional rule learner
OneR Uses minimum-error attribute TABLE 3
ZeroR 0-R classifier Average Error rate with number of features
IBk K-NN classifier
KStar Entropy-based
No of Features 2 3 4 5 6
LWL Locally Weighed Learning
Avg Error Rate 0.3873 0.3859 0.3701 0.3705 0.3701
Naive Bayes Naive Bayes classifier
AdaBoost M1 Boosting a nominal class classifier
Bagging Reduces variance
Stacking Combines the output from others One important question is how many fea-
Logit Boost Additive logistic regression classifier tures to generate in the transformed dataset
Random Committe Ensemble of randomizable base classifiers for a typical classification problem. To justify
Random Forest Forest of random trees
Vote Uses majority vote to label new instance a certain number of features, we perform ex-
Logistic Logistic regression with a ridge estimator periments by reducing the dimension of the
Multilayer Percep- Neural network with backpropagation datasets into 2, 3, 4, 5 and 6 features. For each
tron
Simple Logistic Linear logistic regression dataset, we implement classification for each
SMO Sequential minimal optimization of the datasets after reducing dimensionality
SVM Support Vector Machine using 23 different algorithms and record the
accuracy measurement in each classification
problem. Accuracy metric, which is correlated
dataset is transformed into a corresponding with the SAR metric used later in the paper
intermediate dataset with a fixed number of (explained in Section 4.3), is used in this pre-
features. This allows us to generate meta- liminary experiment to justify the choice of
features for the training set. Weka is used the number of features for simplicity because
to obtain actual classification performance of it is generated directly from most classifier
all classifiers using the transformed datasets. algorithms. The error rate (determined by 1-
These values represent the labels in training accuracy) of each transformed dataset is used
examples corresponding to generated features to compute the averages of error rate for each
for each dataset. Next, we construct our regres- number of features (see Table ??). The number
sion model using the set of training examples. of features with the lowest average error rate is
Finally, we use this model to produce predicted selected as the number of features used for di-
performance and the ranked list. The whole mensionality reduction. Table ?? indicates that
 4

the more the number of features, the lower the to the number of target classes (the AttrClass
average error rates generated in classification attribute).
tasks. The feasible choices are 4, 5 and 6. It
is reasonable not to use higher numbers (e.g., 4.3 Measurement metrics
from 7 onward) as dimensionality reduction In reality, different evaluation metrics can
techniques cannot generate more features than present conflicting results when assessing algo-
the number of features in original datasets. rithms’ performance, especially when multiple
Choosing 5 or 6 features for transformed data algorithms based on different data mining ap-
is feasible but if we do so, we will exclude proaches are involved. This problem has been
original datasets with fewer than 5 or 6 fea- discussed extensively in the study by [?], [?].
tures. We choose 4 as the number of features As a result, to evaluate an algorithm’s perfor-
to produce transformed datasets to generate mance, we propose to use a combined metric
meta-data (training set) for our study. that takes advantage of three commonly used
metrics in classification, This metric, SAR is de-
4.2 Meta-features fined as SAR = [Accuracy+AUC+(1-RMSE)]/3.
The assessment of regression models is mea-
Table ?? describes the meta-features of the sured with Root Mean Square Error (RMSE)
training set. The first feature indicates the metric.
learner under consideration whereas the next To generate a list of algorithms for classifying
two features describe the relation between the a particular dataset, we use a minimum per-
number of classes to the number of attributes formance threshold (0.6) using the SAR metric.
and the number of instances, respectively. The With a threshold parameter value bigger than
set of four summary statistics, viz., Median, 0.5, the final list of algorithms includes only
Mean, Standard deviation and Skewness, is those with high performance measured by the
computed for each of the new mixed features combined metric. Finally, we generate a ranked
of the compressed datasets. It results in 4*4 =16 list of algorithms by predicted performance
meta-features based on statistical summaries. indicating how a particular algorithm may be-
These four statistics provide good predictive have given an unknown dataset.
power while keeping the number of features
in the meta-data relatively low. The last at- 4.4 Data source
tribute represents the learner’s actual perfor-
mance, which is an experimentally derived We use two types of datasets, real and syn-
value. This attribute (Performance) indicates the thetic. From the UCI repository [?], we select
performance of each algorithm on a particular 73 datasets from a variety of domains. The
dataset based on the SAR metric (Squared er- collection of data includes many datasets that
ror, Accuracy, ROC Area), described later. have been used in similar work on algorithm
Among these information theoretic metrics, selection [?], [?], [?].
class entropy (the EntroClass attribute) indicates As real world datasets often come with noise
how much information is needed to specify and imprecision due to error in measuring
one target class whereas total correlation (the devices and human error, drawing precise con-
TotalCorr attribute) measures the amount of in- clusions from mined results suffers from the
formation shared among the attributes. To off- fact that any assumption regarding data dis-
set the possible loss of critical information that tribution cannot be guaranteed. The artificial
might be caused by inappropriate bin bound- datasets also counter the limitation that se-
aries in discretization, we include an additional lected real datasets cover only a small number
feature: the ratio of the number of classes to the of data domains.
dataset’s size (the ClassInt attribute) to measure
the trade-off between the number of instances 4.5 Regression Models
and the number of target classes. We also We use Regression Tree models and compare
calculate the ratio of the number of features with other regression models. This feature
 5

gives the Regression Tree the ability to ap- data points to support a regression. SVR
ply linear models to non-linear data. They are uses a loss function with penalty for high
Classification and Regression Trees (CART) [?], performance in the presence of outliers.
Conditional Tree [?], Model Tree [?], Rule Based • K-Nearest Neighbor Regression (KNN re-
System [?], Bagging CART [?], Random Forest gression) locates K nearest neighbors in the
[?], Gradient Boost Regression [?] and Cubist predictor space to predict new value using
[?]. the summary statistic
• The PLS model tries to obtain a linear
• The CART tree splits attributes to achieve
minimize a loss function. Each final split combination of independent variables that
determines a sub-region of data space that can maximize the covariance needed to
indicate a linear relationship. separate groups.
• The Ridge uses regularization to penalize
• The Conditional Decision Tree (CDT) ap-
plies statistical tests to select split points of by shrinking the coefficients toward 0 to
attributesto avoid selection bias with splits. counter against highly correlated predic-
• The Model Tree represents each leaf as a tors in a dataset.
• LARS is based on the LASSO model [?]. It
linear regression model. Model tree aims
at use reduction of error rate at each node calculate its move in the least angle direc-
when constructing a tree. tion for the next step among the currently
• The Rule based system simplifies a Deci- most correlated covariates.
• The Elastic Net is a generalized combi-
sion Tree by removing parts of rules hav-
ing low predictive power to avoid overfit- nation of two penalized regression tech-
ting. niques: L1 (LASSO) and L2 (Ridge) intro-
• Bagging CART uses bootstrapping with duced by [?] to exploit both advantages.
• MARS initially use surrogates features
aggregation regression to reduce the vari-
ance of prediction. Each model can be built with only one or two predictors at a time
independently. that indicate clearly linear relationship to
• Random Forest uses a random selection of produce the best fit given the initial set.
features to split each node with bootstrap Our approach differs from [?] and [?], which
samples when building trees. either use only artificial datasets (the former)
• Gradient Boost Regression extend the Ad- or associate a set of rules with learner’s perfor-
aBoost [?] using gradient boosting. It adds mance (the latter).
new models to learn misclassification er-
rors in order to reduce bias. 4.6 Evaluation approach
• The Cubist tries to reduce the condition or
We use two statistical significance tests to as-
a rule without increasing error rate. Cubist
can adjust the model prediction using a sess the result of our approach: Spearman’s
training set to improve its [Link] correlation test [?] to measure how close
the two ranks are, other is Friedman test to
The remaining models include Neural Net- validate true ranking which is robust in case
work [?], Support Vector Regression [?], K- of normal distribution is not guarantee.
Nearest Neighbor [?], Partial Least Squared
or PLS [?], Ridge Regression [?], Least Angle
Regression or LARS [?], Elastic Net [?] and 5 E XPERIMENTS
Multi Variate Adaptive Regression Splines or Experiments are conducted on a system with
MARS [?]. Intel Core i5, CPU 2.4Ghz and 6GB RAM. Weka
• Neural Network connects its predictors to API is used for all classification tasks, Python
response through its hidden units. Each scripts for creating synthetic datasets and R
unit receives information from previous scripts for remain works.
layers and generates output to next layer. We use the stratified k-fold method [?] for
• Support Vector Regression (SVR) searches tackling real world unbalanced datasets We
 6

TABLE 4 6 D ISCUSSION
Datasets used as examples to demonstrate the
final ranking list Given training data obtained, we demonstrate
the experiment to select the best regression
Name Nominal Numeric Class Instances model that we use to generate predicted per-
Real data formances. The RMSE values in Table ?? show
credit-g 13 7 2 1000
Kr-vs-kp 37 0 2 3196 that all the regression models are competitive
Abalone 1 7 28 4117 in prediction of algorithm performances. While
Waveform 0 40 3 5000 the RMSEs are less than 1 for all models, Cubist
Shuttle 9 0 7 58000
Synthetic data stands out as the best model in this study. Es-
art-02 0 19 5 1038 pecially, the largest RMSE produced by Model
art-07 0 23 3 4430 Tree (Gradient Boost model) is still good com-
art-12 0 12 10 1276
art-17 0 21 10 3266 pared to the smallest RMSE produced by the
art-24 0 18 2 3552 best model (MARS) under the category of other
linear and non-linear regression models. In re-
ality, MARS is considered to be among the most
appealing models in regression. LARS, Elastic
estimate the predicted performance using the Net, PLS, Neural Net perform well following
regression models for all algorithms (refer to the MARS model. Among the tree models,
Table ??). Finally, predicted and observed val- Model Tree and Gradient Boost are considered
ues of performance of algorithms based on the comparable to the others.
test set are ordered and the following statistical Table ?? shows a monotonically increasing
tests are performed. relationship between predicted ranks and ob-
served ranks (indicated by ρ >0 Spearman’s
Using Spearman’s rank correlation test re- rank coefficient). This positive rank correlation
port in Table ??, we see that all Spearman coeffi- implies the validity of generated ranks where
cients fall into four groups (refer Table ??): very the predicted high rank is consistent with the
strong (such as Neural Net, MARS, Bagging observed rank. These results support our rank
CART, Cubist, Conditional Decision Tree, Ran- list based on predicted performance. Figure ??
dom Forest), strong (SVR, CART, Model Tree, and Figure ?? illustrate the predicted perfor-
Rule based System) and moderate (KNN, PLS, mance with error ranges for all 23 algorithms
Ridge Regresion, LARS and Elastic Net) and using examples generated by the Cubist model
weak (Gradient Boost). However, all p-values on few extra real world and synthetic datasets,
are significantly small, which imply strong ev- respectively. When the lowest error is higher
idence against the null hypothesis or that we than the threshold (dotted line), we are confi-
reject the null hypothesis. On the other hand, dent that the corresponding algorithm is capa-
Friedman rank sum test comes out as: χ2 = ble of working well for the dataset and we in-
186.2428, df = 8, p-value = 2.2e-18. Based on clude it in the ranked list. The predicted values
the result, we reject the null hypothesis that with error ranges imply further improvement
all algorithms are the same at the 0.005 level. may be possible using hyperparameter optimal
All codes in the study are provided in https: search if tuning option is available.
//[Link]/uccs-tdoan/algorithmSelection
TABLE 5
Performances below and above the threshold Spearman Rank Coefficient
(.6) with the horizontal line indicates perfor-
mance worse or better than random guess, Coefficient very weak weak moderate strong very strong
respectively (see Figures ?? and ??). These dia- range .001-.19 .2-.39 .4-.59 .6-.79 .8-1
grams illustrate how algorithms are expected to
perform in case of unknown datasets (in these The top 5 rank lists indicate that no single
figures, we have 5 real world and 5 synthetic learner dominates on all datasets (Table ??).
datasets). The well-known SVM learner appears in half
 7

TABLE 6
1.00

Abalone
0.75
0.50
●
● ● ●
●
●
● ● ●
● ● ● ●
● ●
● ● ● ●
● ● ●
●
RMSEs by multiple Regression Models
0.25
1.00

g−credit
0.75 ● ● ● ● ● ● ● ● ● ● ●
●
● ● ●
●
● ● ●
●
Tree Models RMSE Other Models RMSE
● ● ●
0.50
0.25 CART 0.9291 Neuron Net 0.9699
Performance

Conditional D.T. 0.9166 SVR 0.9714

KR−vs−KP
1.00 ● ● ● ● ● ● ● ● ●
0.75 ●
0.50
● ● ● ● ● ● ● ● ● ● ● ● ● Model Tree 0.9308 KNN 0.9692
0.25
Rule Based
1.00 ● ● ● ● ● ● ● ● ● ● 0.9166 PLS 0.9669
● ● ● ● ●
System

shuttle
● ● ●
0.75 ● ●
0.50
0.25 ● ● ● Bagging CART 0.9251 Ridge 0.5731
Random Forest 0.9216 LARS 0.9668

wave form
1.00 ● ● ● ● ● ● ● ● ● ● ● ●
● ●
0.75 ● ● ●
0.50
● ● ● ● ● ●
Gradient Boost 0.9439 Elastic Net 0.9668
0.25
Cubist 0.9025 MARS 0.9626

Rand..Committe
Deci..Stump
MLPercTron

NaïveBayve
RandForest

LogitBoost
Deci..Table
AdaBoost

Logistic.1
Bagging

Logistic
TABLE 7
ZeroR
OneR

Stack
Kstar

SMO
JRIP
SVM

LWL

Vote
J4.8

Part
Ibk

Spearman ’s ranking correlation

Spearman Spearman
Tree Models Other Models
rank coef. rank coef.
CART 0.7721 Neuron Network 0.8104
Fig. 2. Performance of algorithms on realworld Conditional D.T. 0.8856 SVR 0.7246
examples. Model Tree 0.6438 KNN 0.4849
Rule Based Sys. 0.6438 PLS 0.5619
Bagging CART 0.8374 Ridge 0.5731
Random Forest 0.8977 LARS 0.5790
Gradient Boost 0.3470 Elastic Net 0.5790
Fig. 3. Performance of algorithms on synthetic Cubist 0.8807 MARS 0.8385
datasets.

7 C ONCLUSION AND F UTURE W ORK

Given an unknown dataset, our proposed ap-
proach are able give a robust ranked list of
of all top 5 rank lists but is recommended as learning algorithms with default parameter set-
first choice only one time across the real world ting. Furthermore the ranked list can be use to
and synthetic datasets. Some learners such as select only feasible algorithms for Auto-Weka
J48 (decision tree) and Part often occur in the execution. Our future work is to extend this
top 5 lists and thus can be use as baseline study in case of a large scale data, particular
when algorithms are compared or new learn- for big data challenge.
ing algorithms are developed. Several learners
such as KStar, LWL, Logistic with long training
time also make their way into the ranking lists.
In Figure ??, we observe that only half the
algorithms are “suitable” or the abalone dataset
whereas the shuttle dataset has a wide range of TABLE 8
“suitable” algorithms. Half the “suitable” algo- Ranking of algorithm performance on datasets
rithms have high performance on the second
dataset, implying choices for practitioners. Rank First Second Third Fourth Fifth
abalone SVM AdaBoost J48 IBk Logistic
We note that our rank results obtained with waveform [Link] Logistic MultiL J48 JRip
shuttle Part Bagging IBk Kstar RForest
the combined SAR metric are more roust than g-credit SLogistic Logistic Bagging LWL LBoost
those obtained only with the accuracy metric KRvsKP Part Bagging RandC. Logistic SLogistic
[?]. For instance, [?] rank MultiLayer Percep- art-02 MultiL SVM IBk Logistic simpleL
art-07 MultiL IBk RandomC Logistic RForest
tron in first place but we rank it seventh due art-12 SVM IBk MultiL RandomC SMO
to low AUC of this algorithm’s performance on art-17 IBk MultiL SVM SMO RandC
abalone. art-24 MultiL SVM IBk Logistic RandC