Accepted Manuscript

Rough Set Model Based Feature Selection for Mixed-Type Data with
Feature Space Decomposition

Kyung-Jun Kim , Chi-Hyuck Jun

PII: S0957-4174(18)30149-0
DOI: 10.1016/[Link].2018.03.010
Reference: ESWA 11857

To appear in: Expert Systems With Applications

Received date: 13 November 2017

Revised date: 25 February 2018
Accepted date: 7 March 2018

Please cite this article as: Kyung-Jun Kim , Chi-Hyuck Jun , Rough Set Model Based Feature Selection
for Mixed-Type Data with Feature Space Decomposition, Expert Systems With Applications (2018),
doi: 10.1016/[Link].2018.03.010

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service
to our customers we are providing this early version of the manuscript. The manuscript will undergo
copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please
note that during the production process errors may be discovered which could affect the content, and
all legal disclaimers that apply to the journal pertain.
 ACCEPTED MANUSCRIPT

Highlights
 Interpretability of the feature selection for mixed-type data is increased.

 Any transforming procedure is not needed on categorical and numerical features.

 FSMSD selects features that are not biased by any data-type.

 FSMSD and benchmark methods are compared with 15 mixed-type data.

T
IP
CR
US
AN
M
ED
PT
CE
AC
 ACCEPTED MANUSCRIPT

Rough Set Model Based Feature Selection for

Mixed-type Data with Feature Space Decomposition

Kyung-Jun Kim, Chi-Hyuck Jun*

Department of Industrial and Management Engineering,

T
Pohang University of Science and Technology (POSTECH),

IP
77 Cheongam-Ro, Nam-Gu, Pohang, Gyeongbuk, Republic of Korea, 37673

CR
US
E-mail: k_kim@[Link] (Kyung-Jun Kim), chjun@[Link] (Chi-Hyuck Jun)
AN
* Corresponding author. Tel.: +82-54-279-2197; fax: +82-54-279-2870.
M
ED
PT
CE
AC

Conflict of Interest: The authors declare that they have no conflict of interest.
 ACCEPTED MANUSCRIPT

Title: Interpretable Feature Selection Based on Rough Set Model for

Mixed-type Data Classification

(No. ESWA-D-17-04723)

New Title: Rough Set Model Based Feature Selection for Mixed-Type Data with
Feature Space Decomposition

(Before the point-to-point responses, we have changed a New Title as above to avoid
confusion about the meaning of “Interpretable” in this manuscript. We intended the meaning

T
of “Interpretable” is that it makes full use of each data-type (neighborhood rough set for

IP
numerical and feature space decomposition for categorical) and so does not lose its
information and physical meaning.)

CR
Rough Set Model Based Feature Selection for

US
Mixed-Type Data with Feature Space Decomposition
AN

Abstract
M

Feature selection plays an important role in the classification problems associated with expert and
intelligent systems. The central idea behind feature selection is to identify important input features in
order to reduce the dimensionality of the input space while maintaining or improving classification
ED

performance. Traditional feature selection approaches were designed to handle either categorical or
numerical features, but not the mix of both that often arises in real datasets. In this paper, we propose
a novel feature selection algorithm for classifying mixed-type data, based on a rough set model, called
feature selection for mixed-type data with feature space decomposition (FSMSD). This can handle
PT

both categorical and numerical features by utilizing rough set theory with a heterogeneous Euclidean-
overlap metric, and can be applied to mixed-type data. It also uses feature space decomposition to
preserve the properties of multi-valued categorical features, thereby reducing information loss and
CE

preserving the features’ physical meaning. The proposed algorithm was compared with four
benchmark methods using real mixed-type datasets and biomedical datasets, and its performance was
promising, indicating that it will be helpful to users of expert and intelligent systems.
AC

Keywords: Feature selection; Mixed-type data; Classification; Rough set model; Feature space
decomposition.

1. Introduction

Expert and intelligent systems use a variety of machine learning techniques to obtain and
understand the information inherent in the data. In classification problems, one of the main types of
task handled by expert and intelligent systems, high-dimensional input data can cause several issues,
such as increased prediction error and computational complexity, decreased interpretability, and data
 ACCEPTED MANUSCRIPT

sparsity. Moreover, mixed-type data (also called “heterogeneous” data) can further increase prediction
error or decrease interpretability. These issues occur frequently in areas such as business, medical,
manufacturing, bioinformatics, and finance research, and the key to dealing with them is to use
feature selection to select meaningful features. However, traditional feature selection approaches, such
as correlation coefficients, mutual information, distance metrics, and rough set theory, were only
designed to handle categorical or numerical features. They use discretization to transform any
unwanted numerical features into categorical ones. Similarly, unwanted categorical features are
transformed into numerical ones by treating each category as a number. In the process, both
information and the features’ physical meaning may be lost. Finding ways to avoid this and improve
feature interpretation is a very important topic in many research areas, but little work has been done

T
on feature selection approaches for mixed-type data.

IP
Feature selection methods can be divided into four categories: filter, wrapper, hybrid, and
embedded methods (Saeys et al., 2007). Filter methods do not involve learning, instead selecting the

CR
best feature subset directly based on the data’s intrinsic properties, and are usually employed for high-
dimensional data (Guyon and Elisseeff, 2003). The most widely used filter methods include Pearson
correlation coefficient (PCC) (Press et al., 1996), Relief-F (Kononenko, 1994), information gain (IG)

US
(Liu et al., 2002), minimum redundancy maximum relevance (mRMR) (Peng et al., 2005), and
correlation based feature selection (CFS) (Hall, 1999).

Wrapper methods select the subset of features that yields the best classification performance for a
AN
specific classifier (Blum and Langley, 1997). Typical wrapper methods include forward selection,
backward elimination, and stepwise regression (Efroymson, 1960; Kittler, 1978). Hybrid methods
combine the advantages of both filter and wrapper based methods (Hsu et al., 2011) by using a filter
M

method to extract the candidate features from the original feature set and then refining them using a
wrapper method.

For embedded methods, the feature selection process is part of classifier learning itself (Guyon and
ED

Elisseeff, 2003). This category includes decision trees, such as C4.5 (Quinlan, 1993) and classification
and regression trees (CART) (Breiman et al., 1984). Because both wrapper and embedded methods
use a classifier, their performance is highly dependent on the classifier used. Moreover, the classifier
PT

training required during the exhaustive search for feature subsets is very computationally costly.
However, these methods can nonetheless yield excellent performance for specific classifiers.
CE

For these reasons, filter methods are a reasonable choice for high-dimensional data because they are
relatively straightforward and quick. In addition, the resulting feature subsets are more general
because they do not depend on the classifier.
AC

Feature selection for mixed-type data has, to the best of our knowledge, received little attention and
the previous research has focused primarily on single-type data involving only categorical or
numerical features. Nevertheless, a few studies have investigated feature selection techniques that can
be employed to filter mixed-type data. Rough set based and mutual information based feature
selection are frequently used for mixed-type data. The original rough set model (Pawlak, 1991),
however, was designed around equivalence relations and classes (Hu et al., 2008) based on the values
of categorical features, so it is of limited use for numerical features. To analyze numerical features,
neighborhood (Lin, 2001; Hu et al., 2008) and fuzzy (Dubois and Prade, 1990) rough sets have been
proposed. Mutual information based feature selection is widely used for categorical features, and so
discretization techniques are used.
 ACCEPTED MANUSCRIPT

Recently, researchers in both fields have investigated filter methods. For example, they have been
applied to rough set models by Hu et al. (2010), Zhao and Qin (2014), and Chen and Yang (2014), and
to mutual information models by Hu et al., (2011), Hoque et al. (2014), Bennasar et al. (2015), and
Shishkin et al. (2016). However, mutual information models require numerical features to be
transformed into categorical ones, incurring information loss. Since, in contrast, neighborhood rough
set models do not require such transformations, and thus effectively preserve information, we focus on
rough set based feature selection in this paper.

We now discuss the use of filter methods with rough set models in more detail, as they are our topic
of interest. Hu et al. (2010) proposed the NDEM feature selection method, which uses a neighborhood

T
rough set model with a distance function to compute feature space decision boundaries for mixed-type
data. Zhao and Qin (2014) introduced an extended rough set model based on a neighborhood-

IP
tolerance relation that can handle missing feature values, while Chen and Yang (2014) selected
features for decision systems using both classical and fuzzy rough set models.

CR
We have chosen the previous NDEM (Hu et al., 2010) method as our benchmark filter based
feature selection method in order to investigate the effect of feature space decomposition. Previous
work has primarily focused on how to utilize rough set theory and extend it to numerical features for

US
mixed-type data, but has given little attention to ways of preserving the properties of categorical
features. Here, preserving their properties means avoiding feature transformations that change their
information or physical meaning. In this paper, we propose to solve this problem using a feature
AN
selection method with feature space decomposition, which utilizes neighborhood rough sets with
specific distance metrics and applies feature space decomposition for categorical features. Our
experimental results show that this algorithm could be helpful to users of expert and intelligent
M

systems as it provides both interpretable features and high performance.

The remainder of this paper is organized as follows. Section 2 briefly reviews some basic rough set
theory concepts and introduces the NDEM method. Section 3 introduces the proposed algorithm,
ED

called feature selection for mixed-type data with feature space decomposition (FSMSD). Section 4
presents and discusses our experimental results. Finally, Section 5 presents our conclusions and ideas
for future work.
PT
CE

2. Preliminaries

This section briefly introduces basic rough set model concepts and terminology, as well as methods
related to our proposed feature selection method.
AC

2.1. Rough Set Model

In Pawlak’s rough set model (Pawlak, 1991), objects with the same values for features in the subset
𝐵 are taken to form the equivalence class ,𝑜𝑖 -𝐵 , where 𝑜𝑖 is the 𝑖 th object. These equivalence
classes are also referred to as elemental information granules or elemental concepts. Elemental
granule families *,𝑜𝑖 -𝐵 , 𝑜𝑖 ∈ 𝑈+, where 𝑈 denotes the total sample set, make up concept systems that
can describe arbitrary sample space subsets. Two types of elemental granule union are associated with
any given sample subset 𝑂 (𝑂 ⊂ 𝑈), namely the lower (𝐵𝑂) and upper (𝐵𝑂) approximations of 𝑂:

𝐵𝑂 = *,𝑜𝑖 -𝐵 |,𝑜𝑖 -𝐵 ⊆ 𝑂+, (1)

 ACCEPTED MANUSCRIPT

𝐵𝑂 = *,𝑜𝑖 -𝐵 |,𝑜𝑖 -𝐵 ∩ 𝑂 ≠ ∅+. (2)

The lower and upper approximations give the maximal union of the elemental granules containing
𝑂 and the minimal union of the elemental granules containing 𝑂. The difference between these
approximations is called the approximation boundary of 𝑂, which can be expressed as follows:

𝐵𝑁(𝑂) = 𝐵𝑂 − 𝐵𝑂. (3)

Rough set based feature selection aims to determine the minimal subset of the selected features that
contains the maximal number of elemental granules, i.e., the maximal number of consistent granules.

T
We can use Pawlak’s original rough set model for categorical features without modification, but

IP
we cannot use it directly for numerical features, because it was built around equivalence relations and
classes based on the categorical features’ values. To apply it to numerical features, we require the

CR
following Definitions 1 and 2 (Pawlak, 1991).

Definition 1. Given a set of finite and nonempty objects, 𝑈 = *𝑜1 , 𝑜2 , … , 𝑜𝑛 +, and a numerical feature,
𝑎, describing the objects, the 𝛿-neighborhood of an arbitrary object 𝑜𝑖 ∈ 𝑈 can be defined as follows:

US
𝛿𝑎 (𝑜𝑖 ) = {𝑜𝑗 |∆(𝑜𝑖 , 𝑜𝑗 ) ≤ 𝛿, 𝑜𝑗 ∈ 𝑈},

where δ ≥ 0 and ∆(𝑜𝑖 , 𝑜𝑗 ) denotes a distance metric, e.g., Euclidean distance.

(4)
AN
Definition 2. Given an arbitrary subset 𝑈 and a family of 𝛿-neighborhood information granules
𝛿𝑎 (𝑜𝑖 ), 𝑖 = 1, 2, … , 𝑛, the lower and upper approximations of 𝑂 with respect to the 𝛿𝑎 can be
M

defined as follows:

𝛿𝑎 𝑂 = *𝑜𝑖 |𝛿𝑎 (𝑜𝑖 ) ⊆ 𝑂, 𝑜𝑖 ∈ 𝑈+, (5)

ED

𝛿𝑎 𝑂 = *𝑜𝑖 |𝛿𝑎 (𝑜𝑖 ) ∩ 𝑂 ≠ ∅, 𝑜𝑖 ∈ 𝑈+. (6)

Again, the difference between these approximations is called the boundary of 𝑂:

PT

𝐵𝑁𝑎 (𝑂) = 𝛿𝑎 𝑂 − 𝛿𝑎 𝑂. (7)

The neighborhood information system (NIS) of a feature set 𝐹 is denoted by 𝑁𝐼𝑆 = < 𝑈, 𝐹 >,
CE

where 𝑈 is the sample set (called the universe). More specifically, an NIS is also called a
neighborhood decision table (NDT), denoted by 𝑁𝐷𝑇 = < 𝑈, 𝐹, 𝐷 >, if there is a decision D to be
made. Decisions are also known as class labels in classification problems. Given an NDT 𝑁𝐷𝑇 = <
AC

𝑈, 𝐹, 𝐷 >, 𝑂1 , 𝑂2 , … , 𝑂𝐶 are the subsets of objects with class labels between 1 and 𝐶, and 𝛿𝐵 (𝑜𝑖 )
are the neighborhood information granules that include 𝑜𝑖 generated by mixed features 𝐵 ⊆ 𝐹. The
lower approximation of the decision 𝐷 with respect to 𝐵 can be expressed as

𝛿𝐵 𝐷 = {𝛿𝐵 𝑂1 , 𝛿𝐵 𝑂2 , … , 𝛿𝐵 𝑂𝐶 }. (8)

The dependence of 𝐷 on 𝐵 is defined as the fraction of consistent objects in its lower

approximation, 𝑃𝑂𝑆𝐵 (𝐷), given by

𝛾𝐵 (𝐷) = |𝑃𝑂𝑆𝐵 (𝐷)|/|𝑈|. (9)

 ACCEPTED MANUSCRIPT

This dependence function reflects the descriptive ability of the feature set 𝐵, i.e., it indicates the
significance of B’s features for approximating the decision 𝐷. Consequently, the significance of the
feature 𝑠 relative to 𝐵 and 𝐷 can be expressed as follows:

𝑆𝐼𝐺(𝑠, 𝐵, 𝐷) = 𝛾𝐵∪𝑠 (𝐷) − 𝛾𝐵 (𝐷). (10)

2.2. Neighborhood Decision Error Minimization (NDEM)

Here, we give an overview of the NDEM algorithm, focusing on the aspects that we have improved.
For further details, see Hu et al. (2010). We first introduce NDEM and then discuss its limitations.

T
Based on the rough set model introduced above, NDEM starts by introducing neighborhood rough

IP
sets (NRSs), which can handle both categorical and numerical features. Because these cannot be
measured simultaneously using a traditional distance metric, such as Euclidean distance, an improved
measure is needed for mixed-type data to compare feature performance. For NDEM, one frequently-

CR
used distance metric ∆(𝑜𝑖 , 𝑜𝑗 ) is the HEOM (Wilson and Martinez, 1997), which is defined as

𝐻𝐸𝑂𝑀(𝑝, 𝑞) = √∑𝑚 2
𝑎=1 𝑑𝑎 (𝑝, 𝑞) , (11)

US
where 𝑚 is the number of features and 𝑑𝑎 (𝑝, 𝑞) is the distance between objects 𝑝 and 𝑞 in terms
of the feature 𝑎. This is defined as
AN
1, if the value of 𝑝 𝑜𝑟 𝑞 is unknown,
𝑑𝑎 (𝑝𝑎 , 𝑞𝑎 ) = {𝑜𝑣𝑒𝑟𝑙𝑎𝑝𝑎 (𝑝, 𝑞), if 𝑎 is a categorical feature, (12)
|𝑥 − 𝑦|/(𝑚𝑎𝑥𝑎 − 𝑚𝑖𝑛𝑎 ), if 𝑎 is a numerical feature.
M

Here,
0, if 𝑝 = 𝑞,
𝑜𝑣𝑒𝑟𝑙𝑎𝑝𝑎 (𝑝, 𝑞) = { (13)
1, otherwise.
ED

By using equation (12) as a scaling factor, HEOM can handle the ranges of both numerical and
categorical features.
PT

With different metric functions, NRSs can be used to analyze categorical features, numerical
features, or a mixture of the two (Wang, 2006). With NRSs, equations (5) and (6) can be rewritten as
follows:
CE

𝑁𝑅𝑆𝑂 = *𝑜𝑖 |𝛿(𝑜𝑖 ) ⊆ 𝑂, 𝑜𝑖 ∈ 𝑈+, (14)

AC

𝑁𝑅𝑆𝑂 = *𝑜𝑖 |𝛿(𝑜𝑖 ) ∩ 𝑂 ≠ ∅, 𝑜𝑖 ∈ 𝑈+. (15)

Hu et al. (2010) then defined neighborhood decisions (NDs) as follows:

𝑁𝐷(𝑜𝑖 ) = 𝑐, if 𝑃(𝑐|𝛿(𝑜𝑖 )) = max𝑗 𝑃(𝑐|𝛿(𝑜𝑖 )). (16)

Here, 𝑃(𝑐|𝛿(𝑜𝑖 )) = 𝑛𝑐 /𝐾, where 𝐾 is the number of samples in the neighborhood, 𝑛𝑐 is the
number of samples with decision 𝑐, and the 𝑐 are class labels, 𝑐 = 1, 2, … , 𝐶. The 𝑁𝐷(𝑜𝑖 ) is the
class assigned to 𝑜𝑖 , based on the classification probability in its neighborhood. Then, the NDER,
which is used as the loss function, can be defined as
 ACCEPTED MANUSCRIPT

1 1
𝑁𝐷𝐸𝑅 = 𝑛 ∑𝑛𝑖=1 𝜆(𝑟𝑐(𝑜𝑖 )|𝑁𝐷(𝑜𝑖 )) = 𝑛 ∑𝑛𝑖=1 Λ, (17)

where 𝑟𝑐(𝑜𝑖 ) is the real class of 𝑜𝑖 , and Λ is 0 when 𝑟𝑐(𝑜𝑖 ) = 𝑁𝐷(𝑜𝑖 ), and 1 otherwise. For
consistency and convenience, we focus instead on the neighborhood recognition rate (NRR), defined
as 1 − 𝑁𝐷𝐸𝑅. Then, equation (10) can be expressed as

𝑆𝐼𝐺(𝑠, 𝐵, 𝐷) = 𝑁𝑅𝑅𝐵∪𝑠 (𝐷) − 𝑁𝑅𝑅𝐵 (𝐷). (18)

Using equation (18), the NDEM algorithm then proceeds with sequentially forward selection (SFS),
which is the most widely-used feature subset selection search strategy. SFS starts an empty set of

T
selected features and decides whether to select or discard a candidate feature at each iteration. In the
case of NDEM, if 𝑆𝐼𝐺(𝑠, 𝐵, 𝐷) is greater than 0 then feature s is selected and assigned to the selected

IP
feature set, otherwise it is eliminated.

CR
One of NDEM’s limitations is that the HEOM sets the distance between two objects to 0 if they
have the same value for the given categorical feature and 1 otherwise. This is a very strict metric for
the properties of categorical features and loses a significant amount of information. In this paper, we
mainly focus on this problem and alleviate it by decomposing the feature space so as to maximize the

US
properties of categorical features. The proposed algorithm is discussed in detail in Section 3.

2.3. Effective Range Based Gene Selection (ERGS)

AN
This section briefly reviews a filter based approach to ranking numerical features called the ERGS
method (Chandra and Gupta, 2011). Suppose that a dataset includes 𝑛 observarions of 𝑑 input
features, 𝑋 = *𝑥1 , 𝑥2 , … , 𝑥𝑑 +, and a response variable, 𝑦 ∈ *1, … , 𝐶+, representing a class label
M

between 1 and 𝐶. Let 𝜇𝑖𝑐 and 𝜎𝑖𝑐 denote the mean and standard deviation of the feature i for class
𝑐, respectively. The effective range of the 𝑖 th feature 𝑥𝑖 for the 𝑐th class, 𝑅𝑖𝑐 , is defined as the range
between the lower bound 𝑟𝑖𝑐− and the upper bound 𝑟𝑖𝑐+ :
ED

𝑅𝑖𝑐 = [𝑟𝑖𝑐− , 𝑟𝑖𝑐+ ], (19)

where
PT

𝑟𝑖𝑐− = 𝜇𝑖𝑐 − (1 − 𝑝𝑐 )𝛾𝜎𝑖𝑐 , (20a)

CE

𝑟𝑖𝑐+ = 𝜇𝑖𝑐 + (1 − 𝑝𝑐 )𝛾𝜎𝑖𝑐 . (20b)

Here, 𝑝𝑐 denotes the prior probability of class c and 𝛾 is determined by the Chebyshev inequality:
AC

𝑃(|𝐴 − 𝜇𝑖𝑐 | ≥ 𝛾𝜎𝑖𝑐 ) ≤ 1/𝛾 2 . (21)

The effective ranges for each feature are then sorted into ascending order by class.

The overlapping area (OA) represents the overlaps among the classes for each feature. Features
with smaller overlapping areas are considered more relevant, and so given more weight. The OA
among the classes for feature i is defined as

𝐶
𝑂𝐴𝑖 = ∑𝐶−1
𝑐=1 ∑𝑘=𝑐+1 𝜑𝑖 (𝑐, 𝑘), (22)

where
 ACCEPTED MANUSCRIPT

𝑟 + − 𝑟𝑖𝑘
−
, if 𝑟𝑖𝑐+ > 𝑟𝑖𝑘
−
,
𝜑𝑖 (𝑐, 𝑘) = { 𝑖𝑐 (23)
0, otherwise.

The features’ weights can then be calculated by scaling and normalizing the 𝑂𝐴𝑖 . This is used to
initialize the numerical features due to its simplicity and rapid computation speed. Section 3 discusses
the process in more detail.

3. Proposed Algorithm: FSMSD

T
This section discusses the proposed algorithm in detail, based on the foundation presented in

IP
Section 2, and presents the concept of feature space decomposition for rough set models, which is the
key idea behind the proposed feature selection algorithm. We also work through an example to

CR
demonstrate the algorithm in operation.

3.1. Procedure

Figure 1 illustrates the overall FSMSD procedure. Here, 𝑿 and 𝒁 denote the original numerical

US
and categorical feature sets, respectively, while 𝑈𝑿 and 𝑈𝒁 denote the candidate numerical and
categorical feature sets. In addition, 𝑺𝑿 , 𝑺𝒁 , and 𝑺 (= 𝑺𝑿 ∪ 𝑺𝒁 ) denote the selected numerical and
categorical feature sets and the union of these two sets, respectively.
AN
M
ED
PT
CE
AC

Figure 1. Proposed FSMSD procedure

Step 1 of our method involves using a ranking method to separately put the numerical and
categorical features into descending order. The numerical features are ranked using ERGS (Chandra
 ACCEPTED MANUSCRIPT

and Gupta, 2011), which assigns higher weights to features with smaller overlapping areas, as these
are considered to be more relevant. The categorical features are ranked using an equivalence relation,
which can be directly inferred from their feature values. SFS is used as the (greedy) search procedure,
and one numerical and one categorical feature are selected from the ranked features during each
iteration.

Before explaining Step 2 in detail, we will first explain the concept of feature space decomposition
for clarity. In the feature space defined by a mixed set of both numerical and categorical features,
,𝑿, 𝒁-, the error probability 𝑃𝑒 (𝑿, 𝒁) for class 𝑦𝑗 , 𝑗 = 1, 2, … , 𝐶, can be expressed as
𝑃𝑒 (𝑿, 𝒁) = ∑𝒁 ∫𝑿[1 − max𝑗 𝑝(𝑦𝑗 |𝑿, 𝒁)]𝑝(𝑿, 𝒁) 𝑑𝑿. (24)

T
IP
Rearranging equation (24) yields the following.

𝑃𝑒 (𝑿, 𝒁) = 1 − ∑ ∫ max 𝑝(𝑦𝑗 , 𝑿, 𝒁) 𝑑𝑿

CR
𝑿 𝑗
𝒁

= ∑ 𝑝(𝒁) − ∑ ∫ max 𝑝(𝑦𝑗 , 𝑿|𝒁) 𝑝(𝒁) 𝑑𝑿

𝑿 𝑗
𝑍 𝒁

𝑍 𝑿US
= ∑ 𝑝(𝒁) ∫ [1 − max 𝑝(𝑦𝑗 |𝑿, 𝒁)] 𝑝(𝑿) 𝑑𝑿

= ∑𝒁 𝑝(𝒁)𝑃𝑒 (𝑿|𝒁) = ∑𝑁
𝑗

𝑖=1 𝑝(𝑧𝑖 )𝑃𝑒 (𝑿|𝑧𝑖 ). (25)

AN
Here, the maximum likelihood estimate of 𝑝(𝑧𝑖 ) is the ratio of the number of samples given the
value combination 𝑧𝑖 , denoted as 𝑛𝑖 to the total number of samples 𝑛: 𝑝̂ (𝑧𝑖 ) = 𝑛𝑖 /𝑛 (Tang and
M

Mao, 2007). Once categorical features have been selected, the feature space is decomposed into sets
based on the multi-nominal feature 𝒛, using the error probability defined in equation (25) for a mixed
feature set. Similarly, 𝑃̂𝑒 (𝑿, 𝒁) ≈ 𝑃𝑒 (𝑿, 𝒁) can be calculated as follows:
ED

𝑁 𝑛𝑖
𝑃̂𝑒 (𝑿, 𝒁) = ∑𝑁
𝑖=1 𝑝(𝑧𝑖 )𝑃𝑒 (𝑿|𝑧𝑖 ) = ∑𝑖=1 𝑛 ∑𝑂⊂𝑧𝑖 𝑠𝑝𝑎𝑐𝑒,𝑜∈𝑂 𝐼 .𝑔(𝑟𝐻𝐸𝑂𝑀 (𝑜))/ /|𝑂|, (26)

where 𝑟𝐻𝐸𝑂𝑀 denotes the samples contained in a neighborhood centered at the point 𝑜 with radius
PT

𝛿 (according to the HEOM distance measure), 𝑔(·) is the class with the largest number of samples,
𝐼(𝑔(·)) is an indicator function that takes the value 1 if 𝑔(·) ≠ o’s class and 0 otherwise, 𝑧𝑖 𝑠𝑝𝑎𝑐𝑒
CE

is the part of the feature space decomposition related to 𝑧𝑖 , and |𝑂| denotes the cardinality of 𝑂.

As an example of feature space decomposition, assume we have two categorical features, namely
color = {red, green, blue} and level = {high, low}. Then, 𝒁 = {(red, high), (red, low), (green, high),
AC

(green, low), (blue, high), (blue, low)}. In this case, the number of possible value combinations is
|𝒁| = 6. When applying our proposed algorithm, each of the subspaces at the previous iteration is
decomposed according to the value of the categorical feature selected at the current iteration
according to the following three decomposition rules: 1) Subspaces where all observations have the
same class labels are not decomposed, 2) Subspaces including too few observations are not
decomposed, 3) Other subspaces are decomposed according to all possible values of the selected
feature.

Afterwards, Step 2 begins by selecting the highest-ranking candidate numerical and categorical
features. After calculating the errors for the chosen numerical and categorical features,
 ACCEPTED MANUSCRIPT

𝑃̂𝑒 .(𝑺𝑿 , 𝑈𝑋 ), 𝑺𝒁 / and 𝑃̂𝑒 .(𝑺𝑿 , 𝑈𝑍 ), 𝑺𝒁 / for the class label via (26), we select the candidate feature
with the lowest error. If this error is not smaller error than the error 𝑃̂𝑒 (𝑺𝑿 , 𝑺𝒁 ) from the previous
iteration, then the feature is removed from the candidate feature set. Otherwise, the selected feature is
retained and we proceed to the next step. The other (unselected) candidate feature is retained for
selection at the next iteration.

Assuming that the candidate selected at Step 2 is valid, Step 3 will differ depending on whether the
candidate feature is numerical or categorical. If it is numerical, then it is assigned to the set of selected
numerical features, 𝑺𝑿 . If it is categorical, then it is assigned to the set of selected categorical features,
𝑺𝒁 , and the feature space is decomposed according to the selected feature’s values, as shown in Figure

T
2, to improve the categorical features’ interpretability. Because the HEOM sets the distance between

IP
two objects to 0 if they have the same value for the given categorical feature and 1 otherwise, it is a
very strict metric and distorts the feature’s properties. Decomposing the feature space alleviates this

CR
problem.

US
AN
M
ED

Figure 2. Example of Step 3 of the feature selection procedure

PT
CE

3.2. Example

To demonstrate the proposed algorithm, we now work through an example using the Credit
Approval dataset, one of the mixed-type datasets used in our experiments below. Initially, prior to
AC

Step 1, there are six candidate numerical features, 𝑈𝑋 ={A2, A3, A8, A11, A14, A15}, and nine
candidate categorical features, 𝑈𝑍 ={A1, A4, A5, A6, A7, A9, A10, A12, A13}, and the set 𝑺 is empty.
At Step 1, the two sets of candidate features are placed in descending order of error, resulting in
𝑈𝑋 ={A15, A14, A2, A3, A11, A8} and 𝑈𝑍 ={A13, A9, A4, A5, A10, A1, A7, A12, A6}.

At Step 2, during the first iteration, we compare the errors for the highest-ranked numerical (𝑈𝑋 )
and categorical (𝑈𝑍 ) feature candidates from the complete feature sets because 𝑺 is empty. In this
example, these are A15 and A13, respectively, with errors of 𝑃̂𝑒 .(𝑺𝑿 , 𝑈𝑋 ), 𝑺𝒁 / = 𝑃̂𝑒 (𝐴15) = 0.3828
and 𝑃̂𝑒 .(𝑺𝑿 , 𝑈𝑍 ), 𝑺𝒁 / = 𝑃̂𝑒 (𝐴13) = 0.4533. Thus, A15 is selected as the best candidate, while A13
is retained for the next iteration as the highest-ranked feature in 𝑈𝑍 . Since it is the first feature to be
 ACCEPTED MANUSCRIPT

selected, A15 is automatically considered to be valid and is assigned to 𝑺𝑿 .

Moving forward to the fourth iteration, the best candidates are A3 and A13, with errors of
𝑃̂𝑒 .(𝑺𝑿 , 𝑈𝑋 ), 𝑺𝒁 / = 𝑃̂𝑒 .(𝑺𝑿 , 𝐴3), 𝑺𝒁 / = 0.3645 and 𝑃̂𝑒 .(𝑺𝑿 , 𝑈𝑍 ), 𝑺𝒁 / = 𝑃̂𝑒 .(𝑺𝑿 , 𝐴13), 𝑺𝒁 / =
0.3178, while 𝑺𝑿 = *𝐴15, 𝐴14, 𝐴2+ and 𝑺𝒁 = ∅. In this case, although the categorical feature A13
is selected as the best candidate, its error (0.3178) is higher than the previous error 𝑃̂𝑒 (𝑺𝑿 , 𝑺𝒁 ) =
0.2984. Consequently, it is considered to be invalid and removed from the candidate feature set.

At the fifth iteration, the best candidates are A3 and A9, with errors of 𝑃̂𝑒 .(𝑺𝑿 , 𝐴3), 𝑺𝒁 / = 0.3645

T
and 𝑃̂𝑒 .(𝑺𝑿 , 𝐴9), 𝑺𝒁 / = 0.1271. The categorical feature A9 is thus selected as the best candidate and,

IP
unlike at the fourth iteration, its error (0.1271) is lower than the previous error (0.2984), so it is
considered valid and is handled differently than a numerical feature at Step 3. At Step 3 of the fifth
iteration, the valid categorical feature A9 is assigned to 𝑺𝒁 and feature space decomposition is

CR
applied, according to the decomposition rules given above. To clarify the effect of feature space
decomposition, Figure 3 presents a visualization of the results when the numerical features {A15, A14}
and the categorical features {A9, A10} are selected.

US
AN
M
ED
PT
CE
AC

Figure 3. Visualization of feature space decomposition results for features {A15, A14} and {A9, A10}
(white dots: class 1, black dots: class 2)
 ACCEPTED MANUSCRIPT

Figure 3 shows that decomposing the feature space according to the categorical features A9 and
A10 clearly distinguishes data objects in different classes. For categorical features with multiple
possible values, feature space decomposition addresses spaces that the HEOM cannot handle. Thus,
FSMSD improves the interpretability of categorical features.

4. Experiments and Discussion

T
4.1. Datasets

IP
For the experiments, we used 12 benchmark mixed-type datasets from the UCI Machine Learning
Repository ([Link] and StatLib ([Link] These

CR
datasets incorporate both numerical and categorical features and have various numbers of classes. For
instance, the Credit Approval dataset contains 690 observations of a binary class, and has six
numerical features and nine categorical features. Table 1 summarizes the datasets used.

Table 1. Summary of the datasets US

AN
Dataset Observations Numerical Categorical Class
Credit Approval 690 6 9 2
German 1,000 7 13 2
Heart Disease 303 6 7 2
Heart Statlog 270 6 7 2
M

Hepatitis 155 6 13 2
Horse-colic 368 7 15 2
Housing 506 13 1 2
ED

Contraceptive 1,473 5 4 3
PBC 276 11 7 3
Autos 193 15 10 6
Zoo 101 1 15 7
PT

Arrhythmia 452 206 73 13

DLBCL-C 58 3,745 50 4
Novartis_tissue 103 931 69 4
Brain_Tumor1 49 5,833 87 5
CE

Three additional biomedical datasets were also used, namely DLBCL-C (Shipp et al., 2002),
AC

Novartis_tissue (Su et al., 2002), and Brain_Tumor1 (Pomeroy et al., 2002). These have extremely
high dimensions (𝑛 ≪ 𝑝) and include only numerical features. To create mixed-type data, they were
therefore discretized as follows: 1) convert all the numerical features into binary categorical features
via equal width binning, 2) group the categorical features according to error using the equivalence
relation, 3) randomly choose two features from each group, or one for groups with only one feature 4)
convert the remaining features back into the original numerical features.

Here, we used equal width binning as it is a very simple and widely-used discretization technique and
used binary splitting because of the small number of samples. For example, DLBCL-C involves 25
groups, resulting in 50 categorical features.
 ACCEPTED MANUSCRIPT

4.2. Experimental Setting

The proposed algorithm was compared with three widely-used feature selection methods: Relief-F,
IG, and CFS. In addition, NDEM (Hu et al., 2010) was chosen as the benchmark filter feature
selection method in order to compare the effects of feature space decomposition. For both NDEM and
the proposed algorithm, the values used for the radius 𝛿 were 0.005, 0.006, 0.008, 0.01, 0.02, 0.04,
0.06, 0.08, 0.1, 0.12, 0.14, 0.16, 0.18, 0.20, 0.25, 0.30, and 0.40. In addition, for FSMSD, the feature
space was only decomposed when there were at least 20 samples. Datasets with missing values were

T
processed in two ways: 1) when the proportion of missing values was less than 10%, the observation
containing the missing value was removed and 2) when the proportion of missing values was greater

IP
than 10%, an imputation technique (mean for numerical features and mode for categorical features)
was applied.

CR
In this study, a support vector machine with a radial basis function kernel (RBF-SVM) and CART
were used as classifiers to measure classification performance. For the RBF-SVM, we optimized the
hyper-parameters, namely the cost 𝐶 and gamma 𝛾, in the RBF kernel using grid search for

US
𝐶 = ,2−7 , 2−6 , … , 22 , 23 - and 𝛾 = ,2−15 , 2−14 , … , 214 , 215 -. The RBF-SVM was originally designed
for binary classification problems, so we employed the one-against-one method (Krebel, 1999), the
most common method for multi-class problems. With this method, if there are 𝑘 classes, 𝑘(𝑘 −
AN
1)/2 classifiers are constructed and each classifier is trained on the data from two classes. Then, each
sample’s class is predicted by using a voting strategy (Friedman, 1996) and selecting the class that
receives the most votes. For CART, we required a minimum of 20 samples in each node and the set
the complexity parameter 𝛼 for pruning to 0.05.
M

We used 5-fold cross-validation (CV) for all datasets, dividing each dataset randomly into 5 equal
subsets. Each subset was utilized as the test dataset in turn, with the other four subsets used for
ED

training. This procedure was repeated 10 times, generating a total of 50 results for each classifier and
dataset. All the experiments were implemented using MATLAB R2017a, except for IG and CFS
which were implemented using Weka 3.8.1. The experiments were carried out on an Intel® Core™ i7-
PT

6700 CPU operating at 3.40 GHz, with 32.0GB of RAM and running Windows 10 (64-bit).
CE

4.3 Experimental Results and Discussion

In this section, we present the results of our experiments and discuss our findings. Figure 4 shows
the numbers of numerical and categorical features selected for each dataset and method. In the cases
AC

of Relief-F and IG, which are used for ranking and not subset selection, the number of features
selected was determined from the average numbers of features selected by the subset selection
methods, namely CFS, NDEM, and FSMSD.
 ACCEPTED MANUSCRIPT

T
IP
CR
US
AN
M
ED
PT
CE

Figure 4. Numbers of features selected by each method

(white bars: numerical features, black bars: categorical features, gray bars: total selected features)
AC

In general, all the feature selection methods selected an even and fair mix of numerical and
categorical features for most of the datasets. However, for the Housing, PBC, Zoo, and Arrhythmia
datasets, the methods that did not take mixed-type data into account showed either biased selections
or focused only on either numerical or categorical features, ignoring some important features. This
was also an issue for the extremely high-dimensional datasets (DLBCL-C, Novartis_tissue, and
Brain_Tumor1). Compared with FSMSD, NDEM was often biased toward numerical features,
possibly because equation (13) caused categorical features to lose some of their properties. Unlike the
other methods, FSMSD’s feature selections were not biased toward any particular data-type,
 ACCEPTED MANUSCRIPT

regardless of whether it was being applied to binary-class or multi-class datasets, as the use of feature
space decomposition allowed the categorical features’ properties to be preserved more accurately.
Hence, our experiments show that FSMSD can be fairer and more accurate than the other benchmark
methods.

Tables 2 and 3 summarize the classification accuracies and their standard deviations for seven
binary-class mixed-type datasets and eight multi-class mixed-type datasets, respectively, with the best
performance for each classifier indicated in bold.

T
Table 2. Classification performance for seven binary-class mixed-type datasets

IP
Classifier Dataset Raw Data Relief-F IG CFS NDEM FSMSD
Credit 86.37 86.42 86.37 86.39 86.37 87.35

CR
Approval (0.4782) (0.4873) (0.4131) (0.0486) (0.5405) (0.4291)
74.67 73.34 75.25 75.19 75.99 75.53
German
(0.4547) (0.5211) (0.2677) (0.3281) (0.7025) (0.4332)
Heart 83.63 82.93 83.48 83.74 83.63 85.07

RBF-SVM
Disease
Heart
Statlog
Hepatitis
(0.8868)
83.29
(0.4632)
83.75
83.00
US
(1.2391)

(0.5974)
83.75
(0.3717)
83.18
(0.4479)
88.38
(0.9151)
83.47
(0.6591)
92.88
(0.1152)
83.40
(0.4519)
91.13
(0.1789)
83.87
(0.6053)
97.38
AN
(0.8733) (0.0355) (1.6719) (1.3242) (1.9049) (0.3953)
84.96 86.44 85.82 83.16 85.57 85.61
Horse-colic
(0.1119) (0.6162) (0.4074) (0.7228) (0.1555) (0.6708)
90.06 93.30 92.82 94.57 98.08 97.79
Housing
(0.4295) (0.6407) (0.4022) (0.2522) (0.4272) (0.2909)
M

Average 83.82 84.17 85.04 85.63 86.31 87.51

Credit 85.35 85.80 85.71 85.35 86.35 86.17
ED

Approval (0.6516) (0.4946) (0.5217) (0.6501) (0.6341) (0.7095)

71.84 72.21 73.38 73.17 72.47 73.09
German
(1.0102) (0.6607) (0.9396) (0.8189) (1.0985) (0.6684)
Heart 75.67 77.33 76.59 74.00 75.44 77.63
PT

Disease (1.8901) (1.3836) (2.6294) (2.3863) (2.2566) (1.3205)

Heart 75.74 76.58 76.76 75.39 75.79 77.18
CART
Statlog (2.1163) (1.3058) (1.3264) (1.4718) (2.5740) (1.1028)
80.50 83.75 82.38 81.50 82.75 83.13
Hepatitis
CE

(1.8305) (0.0000) (3.2516) (1.9930) (2.7513) (1.0623)

84.83 84.83 85.06 80.12 85.14 85.87
Horse-colic
(0.4177) (0.4398) (0.4654) (1.2225) (0.5084) (0.0190)
99.92 99.96 99.96 98.82 99.88 100.00
AC

Housing
(0.1670) (0.1240) (0.1252) (0.1913) (0.1913) (0.0000)
Average 81.98 82.92 82.83 81.19 82.55 83.30

Table 3. Classification performance for eight multi-class mixed-type datasets

Classifier Dataset Raw Data Relief-F IG CFS NDEM FSMSD

53.42 54.89 54.89 56.90 53.42 53.22
Contraceptive
(0.5414) (0.5289) (0.5289) (0.4026) (0.6141) (0.6106)
RBF-SVM
73.00 68.12 75.33 75.48 72.75 74.15
PBC
(0.7485) (1.1711) (0.6697) (0.7387) (0.7766) (1.2103)
 ACCEPTED MANUSCRIPT

70.44 79.08 78.42 78.02 74.23 75.17

Autos
(0.9211) (1.7269) (1.7813) (2.1155) (1.3564) (1.5540)
94.94 89.24 90.00 96.45 94.63 96.93
Zoo
(1.6116) (1.5897) (1.7000) (1.8177) (0.8575) (0.3286)
70.42 65.34 67.53 65.54 64.65 68.69
Arrhythmia
(0.8427) (0.8253) (0.7547) (0.6452) (0.4703) (1.0044)
82.73 77.29 85.55 92.03 94.77 88.94
DLBCL-C
(6.9908) (3.3740) (1.8185) (1.5635) (2.3895) (2.4585)
96.65 96.20 97.94 95.23 95.90 98.23
Novartis_tissue
(0.5005) (0.6813) (0.3149) (0.6567) (0.7482) (0.9586)
83.78 84.11 85.78 86.56 86.78 87.22
Brain_Tumor1
(1.1476) (1.8182) (0.7027) (1.2001) (2.3100) (1.0210)

T
Average 78.17 76.78 79.43 80.78 79.64 80.32

IP
54.10 54.13 54.13 54.70 54.10 53.58
Contraceptive
(0.8180) (0.4561) (0.4561) (0.7438) (0.8040) (0.9984)
70.33 68.07 68.04 69.28 70.07 71.81

CR
PBC
(1.3059) (1.5043) (2.7593) (2.5795) (1.3082) (2.0518)
56.01 56.63 50.83 56.81 55.09 57.91
Autos
(2.7518) (2.3825) (2.2843) (3.4088) (2.4036) (0.8766)
67.54 60.37 60.61 60.59 66.99 67.98
CART
Zoo

Arrhythmia
(0.8294)
64.20
(0.9656)
42.09
US
(0.0920)
64.85
(1.3671)
41.03
(0.4675)
65.24
(0.9153)
46.71
(0.4183)
64.71
(0.9831)
42.00
(2.6624)
65.64
(0.8436)
51.79
(0.7741)
62.88
(0.6816)
53.14
AN
DLBCL-C
(6.9908) (5.4876) (5.2453) (5.4296) (4.4455) (2.1922)
70.17 72.18 74.34 67.77 75.56 73.48
Novartis_tissue
(8.4861) (8.0244) (4.4409) (8.7613) (6.8459) (5.2185)
62.89 63.33 64.78 65.00 63.22 68.00
Brain_Tumor1
M

(2.4679) (3.7037) (2.4031) (3.7497) (2.4818) (3.8060)

Average 60.91 60.08 60.58 60.11 62.81 63.60
ED

Overall, the proposed FSMSD algorithm gave the best performance for most datasets. For the
Hepatitis, Housing, and Zoo datasets in particular, it outperformed the other benchmark methods.
PT

Although other methods showed slightly better results for the other datasets, the differences were
marginal. Because the accuracy varied significantly between datasets, Table 4 ranks the methods by
accuracy for each dataset to provide a clearer comparison.
CE

Table 4. Rank results of classification performance

AC

Classifier Dataset Raw Data Relief-F IG CFS NDEM FSMSD

# of the first
1 2 0 2 3 7
RBF-SVM rank
Average 4.29 4.33 3.47 2.93 3.40 2.13
# of the first
0 1 1 1 3 9
CART rank
Average 4.40 3.53 3.13 4.33 3.20 2.07
Total # of the first rank 1 3 1 3 6 16
Total Average 4.34 3.93 3.30 3.63 3.30 2.10
 ACCEPTED MANUSCRIPT

Table 4 shows that FSMSD achieved the highest average rank (2.10), followed by NDEM (3.30)
and IG (3.30). It ranked first 7 times with the RBF-SVM and 9 times with CART. Taken together,
Tables 2–4 show that it was suitable for both binary and multi-class mixed-type datasets and
outperformed the other benchmark methods in various respects. In these experiments, the value of the
radius 𝛿 was chosen so as to achieve the highest average accuracy via grid search. The classification
performance was generally better when using the RBF-SVM than with CART.

4.4. Discussion of Feature Selection Methods

T
Here, we discuss three issues: feature interpretability, computational complexity, and filter method

IP
type. Improving feature interpretability is one of the main focuses of this paper, and the proposed
algorithm was able to maintain the unique properties of the numerical and categorical features. In

CR
contrast, Relief-F regards categorical features as numerical features, while IG and CFS apply specific
discretization techniques to the numerical features. As we have previously discussed, transformations
cause both information and physical meaning to be lost. While NDEM considers mixed-type datasets,
FSMSD mitigates the strict assumptions made by HEOM through feature space decomposition. As

US
shown in Figure 3, this decomposition allows data objects with different class labels to be clearly
distinguished, so FSMSD uses the categorical features’ information more fully than NDEM.
AN
Next, we consider the computational complexity, setting aside the complexity of feature space
decomposition for now. FSMSD first finds the neighborhood of each sample and only considers the
samples in this window. The computational complexity of this step is 𝑛 log 𝑛 + 𝑘𝑛, where 𝑛 is the
number of samples and 𝑘 is a constant value that is used when searching the sample neighborhoods.
M

Next, it determines the class probabilities in the neighborhoods, with a computational complexity of
𝑂(𝑛). Then, it evaluates the remaining features and selects the significant ones one by one. Let 𝑃
denote the number of features selected, consisting of 𝑋 numerical features and 𝑍 categorical ones.
ED

The overall computational complexity of FSMSD is then 𝑀𝑋𝑣(𝑢 log 𝑢 + 𝑘𝑢 + 𝑢), where 𝑀 is the
number of candidate features, 𝑣 is the number of feature spaces produced by decomposition (based
on 𝑍), and 𝑢 (< 𝑛) is the number of samples in each of these spaces. For comparison, NDEM is
PT

similar to FSMSD but without feature space decomposition, so its computational complexity is
𝑀𝑃(𝑛 log 𝑛 + 𝑘𝑛 + 𝑛) (Hu et al., 2010). The other benchmark methods are computationally simple:
Relief-F, for example, has a complexity of 𝑀𝑛 𝑙𝑜𝑔 𝑛 (Sikonja and Kononenko, 2003).
CE

Finally, we considered two types of filter method, namely ranking and subset selection methods.
Relief-F and IG are ranking methods, while CFS, NDEM, and FSMSD are subset selection methods
AC

based on SFS. Ranking methods are generally faster than subset selection methods, but have the
disadvantage that users have to set the number of features or feature weight threshold manually to
obtain feature subsets. In contrast, subset selection methods find subsets automatically, without
requiring users to set a parameter.

5. Conclusions

In this paper, we have proposed FSMSD, a feature selection method for mixed-type data with
feature space decomposition. This makes two contributions to the field. First, decomposing the feature
space according to the selected categorical features improves classification performance and preserves
 ACCEPTED MANUSCRIPT

the original properties of the selected features. This makes it possible to make best use of the
properties of both the numerical and categorical features without information distortion, caused by
transforming categorical features into numerical ones, or information loss, caused by discretizing
numerical features into categorical ones. As a result, the features selected by the proposed algorithm
are not biased toward any given data-type.

Second, FSMSD was able to effectively handle both binary and multi-class mixed-type
classification problems by selecting a subset of features, and also performed well on extremely high-
dimensional data. In our experiments, FSMSD showed better performance overall than the benchmark
methods on mixed-type datasets. Even when its performance was only marginally different compared

T
with the other methods, it was more effective because it was able to make full use of the numerical
and categorical features’ properties without requiring any transformations.

IP
Despite its better performance, the proposed method does have some limitations, which will be the

CR
focus of future work. First, although it was able to effectively preserve the selected features in mixed-
type datasets, it was slow and we will need to find ways to improve the algorithm’s speed. Second, the
high-dimensional mixed-type datasets used in this paper, that is the biomedical datasets, originally
only included numerical features, so we will need to collect and analyze truly mixed-type high-

US
dimensional datasets. In addition, we plan to develop a systematic method of tuning the radius
parameter δ and devise a search strategy that improves upon the SFS strategy used here, as we
believe that this could improve our proposed algorithm.
AN

Acknowledgement
M

This work was supported by a grant of the National Research Foundation of Korea (Project
Number: 2017R1A2B4005450).
ED

Reference
PT

Bennasar, M., Hicks, Y., & Setchi, R. (2015). Feature selection using joint mutual information
maximisation. Expert Systems with Applications, 42(22), 8520-8532.
CE

Blum, A. L., & Langley, P. (1997). Selection of relevant features and examples in machine
learning. Artificial intelligence, 97(1-2), 245-271.

Breiman, L., Friedman, J., Stone, C. J., & Olshen, R. A. (1984). Classification and regression trees.
AC

CRC press. Wadsworth and Brooks, Monterrey, CA.

Chen, D., & Yang, Y. (2014). Attribute reduction for heterogeneous data based on the combination of
classical and fuzzy rough set models. IEEE Transactions on Fuzzy Systems, 22(5), 1325-1334.

Dubois, D., & Prade, H. (1990). Rough fuzzy sets and fuzzy rough sets. International Journal of
General System, 17(2-3), 191-209.

Efroymson, M. A. (1960). Multiple regression analysis. Mathematical methods for digital computers,
191-203., volume 1. Wiley, New York, pp 191-203.

Friedman, J. (1996). Another approach to polychotomous classification (Vol. 56). Technical report,
 ACCEPTED MANUSCRIPT

Department of Statistics, Stanford University.

Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection. Journal of
machine learning research, 3(Mar), 1157-1182.

Hall, M. A. (1999). Correlation-based feature selection for machine learning. Dissertation, University
of Waikato.

Hoque, N., Bhattacharyya, D. K., & Kalita, J. K. (2014). MIFS-ND: a mutual information-based
feature selection method. Expert Systems with Applications, 41(14), 6371-6385.

T
Hsu, H. H., Hsieh, C. W., & Lu, M. D. (2011). Hybrid feature selection by combining filters and
wrappers. Expert Systems with Applications, 38(7), 8144-8150.

IP
Hu, Q., Pedrycz, W., Yu, D., & Lang, J. (2010). Selecting discrete and continuous features based on

CR
neighborhood decision error minimization. IEEE Transactions on Systems, Man, and Cybernetics,
Part B (Cybernetics), 40(1), 137-150.

Hu, Q., Yu, D., & Xie, Z. (2008). Neighborhood classifiers. Expert systems with applications, 34(2),
866-876.
US
Hu, Q., Zhang, L., Zhang, D., Pan, W., An, S., & Pedrycz, W. (2011). Measuring relevance between
AN
discrete and continuous features based on neighborhood mutual information. Expert Systems with
Applications, 38(9), 10737-10750.

Kittler, J. (1978). Feature set search algorithms. Pattern recognition and signal processing. Springer,
M

Alphen aan den Rijn, The Netherlands, pp 41-60.

Kononenko, I. (1994, April). Estimating attributes: analysis and extensions of RELIEF. In European
ED

conference on machine learning (pp. 171-182). Springer, Berlin, Heidelberg. pp 171-182.

Krebel, U. G. (1999). Pairwise classification and support vector machines. Advances in Kernel
Methods. Cambridge, MA: MIT Press, 1999, pp. 255–268.
PT

Lin, T. Y. (2001). Granulation and nearest neighborhoods: rough set approach. In Granular
computing (pp. 125-142). Physica, Heidelberg.
CE

Liu, H., Li, J., & Wong, L. (2002). A comparative study on feature selection and classification
methods using gene expression profiles and proteomic patterns. Genome informatics, 13, 51-60.
AC

Pawlak, Z. (1991). Rough Sets: Theoretical Aspects of Reasoning about Data (Vol. 9). Springer
Science & Business [Link] & Boston: Kluwer Academic Publishers.

Peng, H., Long, F., & Ding, C. (2005). Feature selection based on mutual information criteria of max-
dependency, max-relevance, and min-redundancy. IEEE Transactions on pattern analysis and
machine intelligence, 27(8), 1226-1238.

Pomeroy, S. L., Tamayo, P., Gaasenbeek, M., Sturla, L. M., Angelo, M., McLaughlin, M. E., ... &
Allen, J. C. (2002). Prediction of central nervous system embryonal tumour outcome based on gene
expression. Nature, 415(6870), 436.

Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1996). Numerical recipes in C (Vol.
 ACCEPTED MANUSCRIPT

2). Cambridge: Cambridge university press. Cambridge University Press, New York, pp 636-639.

Quinlan, J. R. (1993). C 4.5: Programs for machine learning. Morgan Kaufmann, San Francisco, CA.

Saeys, Y., Inza, I., & Larrañaga, P. (2007). A review of feature selection techniques in
bioinformatics. bioinformatics, 23(19), 2507-2517.

Shipp, M. A., Ross, K. N., Tamayo, P., Weng, A. P., Kutok, J. L., Aguiar, R. C., ... & Ray, T. S. (2002).
Diffuse large B-cell lymphoma outcome prediction by gene-expression profiling and supervised
machine learning. Nature medicine, 8(1), 68.

T
Shishkin, A., Bezzubtseva, A., Drutsa, A., Shishkov, I., Gladkikh, E., Gusev, G., & Serdyukov, P.
(2016). Efficient high-order interaction-aware feature selection based on conditional mutual

IP
information. In Advances in Neural Information Processing Systems (pp. 4637-4645).

CR
Su, A. I., Cooke, M. P., Ching, K. A., Hakak, Y., Walker, J. R., Wiltshire, T., ... & Patapoutian, A.
(2002). Large-scale analysis of the human and mouse transcriptomes. Proceedings of the National
Academy of Sciences, 99(7), 4465-4470.

US
Tang, W., & Mao, K. Z. (2007). Feature selection algorithm for mixed data with both nominal and
continuous features. Pattern Recognition Letters, 28(5), 563-571.
AN
Wang, H. (2006). Nearest neighbors by neighborhood counting. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 28(6), 942-953.

Wilson, D. R., & Martinez, T. R. (1997). Improved heterogeneous distance functions. Journal of
M

artificial intelligence research.

Zhao, H., & Qin, K. (2014). Mixed feature selection in incomplete decision table. Knowledge-Based
ED

Systems, 57, 181-190.

PT
CE
AC