Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
Data Mining with Weka (Lab3)
Selecting important data (Feature Selection)
Some things matter, some don’t
Features Types
1) Relevant Features:
Those that we need to perform well (improve performance - increase
accuracy)
2) Irrelevant Features: (noisy features)
Those are simply unnecessary (hurt performance – reduce accuracy)
3) Redundant Features:
Those that become irrelevant in the presence of others or those that only
show their worth in the context of others
Feature Selection
Feature selection is a method of reducing the number of inputs supplied to a
machine learning model such that only the relevant inputs are used. The technique is
applied to improve the accuracy of the machine learning model. Feature selection
is also called variable selection or attribute selection.
Benefits of Feature Selection 1) Reduces overfitting: less redundant data
means less opportunity to make decision based on noise
2) Improves accuracy: fewer misleading data means modeling accuracy
improves.
3) Reduced Training Time: less data means training process is more efficient.
Feature Selection Algorithms
There are three general classes of feature selection algorithms: filter methods,
wrapper methods and embedded methods.
1) Filter feature selection methods: apply a statistical measure to assign a
scoring to each feature. The features are ranked by the score and either
selected to be kept or removed from the dataset. The methods are often
Page 1 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
univariate and consider the feature independently, or with regard to the
dependent variable.
Some examples of some filter methods include the Chi squared test,
information gain and correlation coefficient scores.
2) Wrapper feature selection methods: consider the selection of a set of
features as a search problem, where different combinations are prepared,
evaluated and compared to other combinations. A predictive model is used to
evaluate a combination of features and assign a score based on model
accuracy.
The search process may be methodical such as a best-first search, it may stochastic
such as a random hill-climbing algorithm, or it may use heuristics, like forward and
backward passes to add and remove features.
An example if a wrapper method is the recursive feature elimination algorithm.
3) Embedded feature selection methods: learn which features best contribute
to the accuracy of the model while the model is being created. The most
common type of embedded feature selection methods are regularization
methods.
Regularization methods are also called penalization methods that introduce
additional constraints into the optimization of a predictive algorithm (such as a
regression algorithm) that bias the model toward lower complexity (fewer
coefficients).
Examples of regularization algorithms are the LASSO, Elastic Net and Ridge
Regression.
Practice: Predict the Onset of Diabetes
The dataset used for this example is the Pima Indians onset of diabetes dataset. It
is a classification problem where each instance represents medical details for one
patient and the task is to predict whether the patient will have an onset of diabetes
within the next five years.
Page 2 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
You can access this dataset in your Weka installation, under the data/
directory in the file called [Link].
Feature Selection in Weka
Many feature selection techniques are supported in Weka.
1. Open the Weka GUI Chooser.
2. Click the “Explorer” button to launch the Explorer.
3. Open the Pima Indians dataset.
4. Click the “Select attributes” tab to access the feature selection methods.
Feature selection is divided into two parts:
• Attribute Evaluator: The attribute evaluator is the technique by which each
attribute in your dataset (also called a column or feature) is evaluated in the
context of the output variable (e.g., the class).
• Search Method: The search method is the technique by which to try or
navigate different combinations of attributes in the dataset in order to arrive
on a short list of chosen features.
Page 3 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
Each section has multiple techniques from which to choose.
Some Attribute Evaluator techniques require the use of specific Search Methods. For
example, the CorrelationAttributeEval technique used in the next section can only
be used with a Ranker Search Method, that evaluates each attribute and lists the
results in a rank order. When selecting different Attribute Evaluators, the interface
may ask you to change the Search Method to something compatible with the chosen
technique.
Both the Attribute Evaluator and Search Method techniques can be configured. Once
chosen, click on the name of the technique to get access to its configuration details.
Now that we know how to access feature selection techniques in Weka, let’s take a
look at how to use some popular methods on our chosen standard dataset.
Page 4 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
Correlation Based Feature Selection
A popular technique for selecting the most relevant attributes in your dataset is to
use correlation.
You can calculate the correlation between each attribute and the output variable and
select only those attributes that have a moderate-to-high positive or negative
correlation (close to -1 or 1) and drop those attributes with a low correlation (value
close to zero).
Weka supports correlation-based feature selection with the
CorrelationAttributeEval technique that requires use of a Ranker search method.
Running this on our Pima Indians dataset suggests that one attribute (plas) has the
highest correlation with the output class. It also suggests a host of attributes with
some modest correlation (mass, age, preg). If we use 0.2 as our cut-off for relevant
attributes, then the remaining attributes could possibly be removed (pedi, insu, skin
and pres).
Page 5 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
Information Gain Based Feature Selection
Another popular feature selection technique is to calculate the information gain.
You can calculate the information gain (also called entropy) for each attribute for
the output variable. Entry values vary from 0 (no information) to 1 (maximum
information). Those attributes that contribute more information will have a higher
information gain value and can be selected, whereas those that do not add much
information will have a lower score and can be removed.
Weka supports feature selection via information gain using the
InfoGainAttributeEval Attribute Evaluator. The Ranker Search Method must be
used.
Running this technique on our Pima Indians we can see that one attribute
contributes more information than all of the others (plas). If we use an arbitrary cutoff
of 0.05, then we would also select the mass, age and insu attributes and drop the
rest from our dataset.
Page 6 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
Learner Based Feature Selection
A popular feature selection technique is to use a generic but powerful learning
algorithm and evaluate the performance of the algorithm on the dataset with different
subsets of attributes selected.
The subset that results in the best performance is taken as the selected subset. The
algorithm used to evaluate the subsets does not have to be the algorithm that you
intend to use to model your problem, but it should be generally quick to train and
powerful, like a decision tree method.
In Weka this type of feature selection is supported by the WrapperSubsetEval
technique and must use a GreedyStepwise or BestFirst Search Method. The
latter, BestFirst, is preferred if you can spare the compute time.
1. First select the “WrapperSubsetEval” technique.
2. Click on the name “WrapperSubsetEval” to open the configuration for the method.
3. Click the “Choose” button for the “classifier” and change it to J48 under “trees”.
4. Click “OK” to accept the configuration.
5. Change the “Search Method” to “BestFirst”.
Page 7 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
6. Click the “Start” button to evaluate the features.
Running this feature selection technique on the Pima Indians dataset selects 4 of the
8 input variables: plas, pres, mass and age.
Select Attributes in Weka
Looking back over the three techniques, we can see some overlap in the selected
features (e.g. plas), but also differences.
It is a good idea to evaluate a number of different “views” of your machine learning
dataset. A view of your dataset is nothing more than a subset of features selected by
a given feature selection technique. It is a copy of your dataset that you can easily
make in Weka.
For example, taking the results from the last feature selection technique, let’s
say we wanted to create a view of the Pima Indians dataset with only the
following attributes: plas, pres, mass and age:
1. Click the “Preprocess” tab.
2. In the “Attributes” selection Tick all but the plas, pres, mass, age and class
attributes.
Page 8 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
3. Click the “Remove” button.
4. Click the “Save” button and enter a filename.
What Feature Selection Techniques to Use
You cannot know which views of your data will produce the most accurate
models.
Therefore, it is a good idea to try a number of different feature selection
techniques on your data and in turn create many different views of your data.
Select a good generic technique, like a decision tree, and build a model for each
view of your data.
Compare the results to get an idea of which view of your data results in the
best performance. This will give you an idea of the view or more specifically
features that best expose the structure of your problem to learning algorithms in
general.
Page 9 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
10 of 11
Alamein University Data Mining
Faculty of Computer science Course Code: AIE323
& Engineering
Page
11 of 11